Making Your Cukes More Awesomer With Screenshots

In my last post about Cucumber, I alluded to using watir-webdriver instead of using native watir. There was one very surprising advantage I discovered with watir-webdriver's use of selenium as a backend: Awesome Screenshot API.

Here's how we setup our Cucumber suite to take screenshots (and html/txt "screenshots") using watir-webdriver and celerity.  First, set your tests up to write an error when a test fails:

	After do |scenario|
	write_errors([ENV['TEST_SERVER'] || 'devl'], scenario, @browser) if scenario.failed?
	end

view raw hooks.rb hosted with ❤ by GitHub

In this hook, we're passing in an environment, the scenario and the browser object we setup in the prior post.  The environment is really handy since QA folks might be running tests against different regions and would want to pass that information back to the developers to recreate problems.  The TEST_SERVER environment variable is already used to pick this during test startup.

The ErrorWriter module is next.

	module ErrorWriter
	def scenario_name(scenario)
	if scenario.instance_of?(Cucumber::Ast::OutlineTable::ExampleRow)
	scenario_name = scenario.scenario_outline.name.gsub(/[^\w\-]/, ' ')
	scenario_name += "-Example#{scenario.name.gsub(/\s*\|\s*/, '-')}".chop
	else
	scenario_name = scenario.name.gsub(/[^\w\-]/, ' ')
	end
	scenario_name
	end
	def file_name(region, scenario_name)
	time = Time.now.strftime("%Y-%m-%d-%H%M%S")
	# Windows has file name limits
	"#{region}-#{time}-#{scenario_name}".slice(0, 250).gsub(/[\,\/]/, '.')
	end
	def write_errors(region, scenario, browser)
	Dir.mkdir("error_pages") unless File.directory?("error_pages")
	scenario_name = scenario_name(scenario)
	file_name = file_name(region, scenario_name)
	File.open("error_pages/#{file_name}.txt", 'wb') { |fh|
	fh.write browser.text.to_s
	}
	File.open("error_pages/#{file_name}.html", 'wb') { |fh|
	fh.write browser.html.to_s
	}
	capture_screenshot(browser, file_name)
	end
	end
	World(ErrorWriter)

view raw error_writer.rb hosted with ❤ by GitHub

Several things are going on here.  The write errors message is simple enough.  It creates a directory for the errors to be written, and dumps the context of the browser's text and html representations to file.  What's more interesting are the scenario_name and file_name methods.

The scenario_name method does some interesting things.  We want to name our files by the name of the scenario that fails.  In the else clause, you see how dashes and white space are replaced by a single white space character.  But, in the if clause, we're testing to see if we have an ExampleRow.  These are the Examples in a Cucumber Scenario Outline.  If the scenario is an ExampleRow, we lose the name of the Scenario itself.  Fortunately, the ExampleRow knows what ScenarioOutline it belongs to.  We can get the name of the ScenarioOutline, perform the same transform on the name, and then append the example that is actually running to it's name.

The file_name method simply concatenates the region, time of failure and the scenario name together for the filename to be written to.

This can result in some very long file names, however, their descriptiveness wins overall.  You don't have to open a file to find out what failed.  A simple reading of the directory containing the files lets anyone find the test that they need to examine quickly.

The write_errors method did one other thing. Call capture_screenshot to grab an image of the browser.  When using Watir, you had to do some fancy things with the Win32 API to make the capture work.  We had a lot of issues using it with our QA folks who had SnagIT installed.  So how refreshing it was to have watir-webdriver just work with so little code:

	module Screenshots
	if WEBDRIVER
	def capture_screenshot(browser, file_name)
	Dir.mkdir("screenshots") unless File.directory?("screenshots")
	@browser.driver.save_screenshot("screenshots/#{file_name}.png")
	end
	else
	def capture_screenshot(browser, file_name)
	#puts "Sorry - no screenshots on your platform yet."
	end
	end
	end
	World(Screenshots)

view raw screenshots.rb hosted with ❤ by GitHub

We have another little trick, where in our prior example, we setup a WEBDRIVER variable to determine if we could take a screenshot.  If we're running on Celerity, no harm, no foul, but no screenshot.

Granted, it's really selenium that we're using here, but we get really nice and small png files, which only contain the contents of the browser.  No title bar, no toolbars, nothing. Just the contents of the window.  Our only issue is that IE causes problems on our CI box, but just having Firefox able to run there is an outstanding step forward.

Hopefully this is helpful, please let me know if it is.

Cucumber with Celerity and Watir-Webdriver

Earlier this year, I had the problem of trying to enable our QA staff to run our current suite of Celerity driven Cucumber tests through the actual browsers that we were using.  One obvious solution is to use various flavors of Watir.

The main concern with this solution is that Waitr required native ruby instead of jruby.  Not a huge deal, but still an extra annoyance for folks in a java shop.  Requiring QA to switch between jruby and watir wasn't ideal.  Additionally, taking snapshots of the screen during a failed test was far from trivial.

So instead, I embarked on a quest to use watir-webdriver and Celerity together.  The advantage here is that watir-webdriver, through selenium-webdriver, can talk to browsers and runs on jruby.  This combination was alluded to on several blog sites that I had found, but I never quite found something that put it all together for me.   Eventually, i got it working. Hopefully this will at least get you about 90% of the way there.

The first step would be adding watir-webdriver to your Gemfile:

	source "http://rubygems.org"
	gem "cucumber", "0.10.0"
	gem "gherkin", "2.3.3"
	gem "watir-webdriver", "0.2.0"
	gem "rake"
	gem "rspec"
	platforms :jruby do
	gem "celerity", "0.8.7"
	gem "syntax"
	end

view raw Gemfile hosted with ❤ by GitHub

Following that we create a browser.rb file that determines what sort of browser you are running.  I placed this in the support folder in the Cucumber project.  One thing to remember about this, is that you could also throw this method into your env.rb.

	# Just returns the appropriate browser class to instantiate
	case RUBY_PLATFORM
	when /java/
	if ENV['BROWSER'] =~ /true/
	require "watir-webdriver"
	Browser = Watir::Browser
	WEBDRIVER = true
	puts "Running in a Browser"
	if ENV['CHROMEWATIR']
	BROWSER_TYPE = :chrome
	puts "Using Chrome"
	elsif ENV['FIREWATIR']
	BROWSER_TYPE = :firefox
	puts "Using Firefox"
	if ENV['FIREWATIRPATH']
	# Custom Firefox Path, handy if you don't install in default locations
	# or if you want to test with multiple Firefoxes
	Selenium::WebDriver::Firefox.path = ENV['FIREWATIRPATH']
	puts "at #{ENV['FIREWATIRPATH']}"
	end
	else # ENV['WATIR']
	BROWSER_TYPE = :ie
	puts "Using IE"
	end
	else
	require 'celerity'
	Browser = Celerity::Browser
	Celerity.index_offset = 0
	WEBDRIVER = false
	puts "Running Headless"
	end
	else
	raise "This platform is not supported (#{PLATFORM})"
	end

view raw browser.rb hosted with ❤ by GitHub

What's going on here? We're using the environment variable BROWSER to determine if we want to run something in a browser.  We then use another environment variable to determine which browser to use.  We even have an environment variable to determine what Firefox we want to use, which is a nice feature of selenium-webdriver.  At the end, Browser contains the class of browser we want to instantiate for tests, and BROWSER_TYPE contains the actual type of browser to instantiate for watir-webdriver.

One other thing to note, when we use Celerity, we're setting the offset for index based searches to 0.  Unlike watir-webdriver, Watir uses 1th based arrays.  Celerity, being a wrapper of HtmlUnit, had to translate those arrays from 1 to 0 so that HtmlUnit would run properly.  By setting this value, we can use 0th based arrays, just like watir-webdriver does in Celerity.

Next we setup our Cucumber Before block in hooks.rb:

	Before do
	case RUBY_PLATFORM
	when /java/
	if WEBDRIVER
	@browser = Browser.new BROWSER_TYPE
	else
	@browser = Browser.new
	@browser.webclient.setUseInsecureSSL(true)
	end
	else
	raise "This platform is not supported (#{PLATFORM})"
	end
	@base_url = RegionBaseURL
	end
	After do
	@browser.close
	end

view raw hooks.rb hosted with ❤ by GitHub

In here, we simply instantiate the object.  In my case, we also needed to setup some URLs and turn off SSL for our Celerity testing.

The Rakefile is probably a bit more complex due to our desire to run both headless and browser tests on our CI box, but it is helpful, since it uses the :browser task to setup the BROWSER environment variable.

	require 'rubygems'
	require 'rake/clean'
	require 'cucumber'
	require 'cucumber/rake/task'
	task :browser => ['features:setupenv', 'features:isbrowser', 'features:browser'] do
	end
	task :headless => ['features:setupenv', 'features:headless'] do
	end
	def setup_environment
	ENV['TEST_SERVER'] ||= 'devl'
	puts "running on the #{ENV['TEST_SERVER']} environment..."
	end
	namespace :features do
	task :setupenv do
	setup_environment
	end
	task :isbrowser do
	ENV['BROWSER'] ||= 'true'
	end
	Cucumber::Rake::Task.new(:browser, 'Run Cucumber Features using a Browser') do |t|
	t.profile = ENV['CI'] ? 'browser_ci' : 'browser'
	end
	Cucumber::Rake::Task.new(:headless, 'Run Cucumber Features using a Headless Browser') do |t|
	t.profile = ENV['CI'] ? 'ci' : 'default'
	end
	end
	task :default => :headless

view raw Rakefile hosted with ❤ by GitHub

Finally, to help our QA folks, three batch files to execute the tests.

	@echo off
	set TEST_SERVER=
	set FIREWATIR=
	set CHROMEWATIR=
	set WATIR=
	if "" == "%1%" goto NO_SERVER
	set TEST_SERVER=%1
	shift
	echo testing on %TEST_SERVER% environment
	:NO_SERVER
	echo jruby -S rake --trace headless
	jruby -S rake --trace headless

view raw test-headless.bat hosted with ❤ by GitHub

	@echo off
	set TEST_SERVER=
	set FIREWATIR=
	set CHROMEWATIR=
	set WATIR=true
	if "" == "%1%" goto NO_SERVER
	set TEST_SERVER=%1
	shift
	echo testing on %TEST_SERVER% environment
	:NO_SERVER
	echo jruby -S rake --trace browser
	jruby -S rake --trace browser

view raw test-ie.bat hosted with ❤ by GitHub

	@echo off
	set TEST_SERVER=
	set FIREWATIR=true
	set CHROMEWATIR=
	set WATIR=
	@REM set FIREWATIRPATH=C:\path\to\your\other\version\of\firefox.exe
	if "" == "%1%" goto NO_SERVER
	set TEST_SERVER=%1
	shift
	echo testing on %TEST_SERVER% environment
	:NO_SERVER
	echo jruby -S rake --trace browser
	jruby -S rake --trace browser

view raw test-ff.bat hosted with ❤ by GitHub

Hopefully you'll find this useful. Please let me know if you do.