Monkey Patching Firefox Race Conditions For Watir-WebDriver/Cucumber

If your team is like my team, you're trying to find ways to automate everything. Perhaps you're also using watir-webdriver to run browser based tests with Cucumber. This works great locally, but when you try to run your tests on a CI environment you start to run into some very strange problems with Firefox testing.  You start to get errors around not being able to find the h1 tag on your page.  You look at the screen shot you've generated using a method similar to this, and you find that your screenshot shows the text that contains the h1.  In fact, your error dumped html file also shows the h1 tag:

You've run into selenium issue 2099.  It appears what happens with this error is that the test is executing before the page has fully loaded, causing it to fail.  Old school race conditions.  Nice.

So how do we get around this?  There's been a few postings on how to handle this with Capybara, but what about tools that use selenium-webdriver more directly?  Here's a monkey patch that hacks over the Ruby bindings to enable tools like watir-webdriver to retry.  This patch will retry the find until selenium-webdriver receives an error that is not related to the strange InvalidSelectorError that it gets when the Firefox isn't done loading the page. I just stuff this file into the support directory in a Cucumber suite.

Since this is hacky and a monkey patch, I'm very keen on ideas on how to improve this implementation, since my Ruby skills are very raw.

Making Your Cukes More Awesomer With Screenshots

In my last post about Cucumber, I alluded to using watir-webdriver instead of using native watir. There was one very surprising advantage I discovered with watir-webdriver's use of selenium as a backend: Awesome Screenshot API.

Here's how we setup our Cucumber suite to take screenshots (and html/txt "screenshots") using watir-webdriver and celerity.  First, set your tests up to write an error when a test fails:


In this hook, we're passing in an environment, the scenario and the browser object we setup in the prior post.  The environment is really handy since QA folks might be running tests against different regions and would want to pass that information back to the developers to recreate problems.  The TEST_SERVER environment variable is already used to pick this during test startup.

The ErrorWriter module is next.


Several things are going on here.  The write errors message is simple enough.  It creates a directory for the errors to be written, and dumps the context of the browser's text and html representations to file.  What's more interesting are the scenario_name and file_name methods.

The scenario_name method does some interesting things.  We want to name our files by the name of the scenario that fails.  In the else clause, you see how dashes and white space are replaced by a single white space character.  But, in the if clause, we're testing to see if we have an ExampleRow.  These are the Examples in a Cucumber Scenario Outline.  If the scenario is an ExampleRow, we lose the name of the Scenario itself.  Fortunately, the ExampleRow knows what ScenarioOutline it belongs to.  We can get the name of the ScenarioOutline, perform the same transform on the name, and then append the example that is actually running to it's name.

The file_name method simply concatenates the region, time of failure and the scenario name together for the filename to be written to.

This can result in some very long file names, however, their descriptiveness wins overall.  You don't have to open a file to find out what failed.  A simple reading of the directory containing the files lets anyone find the test that they need to examine quickly.

The write_errors method did one other thing. Call capture_screenshot to grab an image of the browser.  When using Watir, you had to do some fancy things with the Win32 API to make the capture work.  We had a lot of issues using it with our QA folks who had SnagIT installed.  So how refreshing it was to have watir-webdriver just work with so little code:


We have another little trick, where in our prior example, we setup a WEBDRIVER variable to determine if we could take a screenshot.  If we're running on Celerity, no harm, no foul, but no screenshot.

Granted, it's really selenium that we're using here, but we get really nice and small png files, which only contain the contents of the browser.  No title bar, no toolbars, nothing. Just the contents of the window.  Our only issue is that IE causes problems on our CI box, but just having Firefox able to run there is an outstanding step forward.

Hopefully this is helpful, please let me know if it is.

Cucumber with Celerity and Watir-Webdriver

Earlier this year, I had the problem of trying to enable our QA staff to run our current suite of Celerity driven Cucumber tests through the actual browsers that we were using.  One obvious solution is to use various flavors of Watir.

The main concern with this solution is that Waitr required native ruby instead of jruby.  Not a huge deal, but still an extra annoyance for folks in a java shop.  Requiring QA to switch between jruby and watir wasn't ideal.  Additionally, taking snapshots of the screen during a failed test was far from trivial.

So instead, I embarked on a quest to use watir-webdriver and Celerity together.  The advantage here is that watir-webdriver, through selenium-webdriver, can talk to browsers and runs on jruby.  This combination was alluded to on several blog sites that I had found, but I never quite found something that put it all together for me.   Eventually, i got it working. Hopefully this will at least get you about 90% of the way there.

The first step would be adding watir-webdriver to your Gemfile:


Following that we create a browser.rb file that determines what sort of browser you are running.  I placed this in the support folder in the Cucumber project.  One thing to remember about this, is that you could also throw this method into your env.rb.


What's going on here? We're using the environment variable BROWSER to determine if we want to run something in a browser.  We then use another environment variable to determine which browser to use.  We even have an environment variable to determine what Firefox we want to use, which is a nice feature of selenium-webdriver.  At the end, Browser contains the class of browser we want to instantiate for tests, and BROWSER_TYPE contains the actual type of browser to instantiate for watir-webdriver.

One other thing to note, when we use Celerity, we're setting the offset for index based searches to 0.  Unlike watir-webdriver, Watir uses 1th based arrays.  Celerity, being a wrapper of HtmlUnit, had to translate those arrays from 1 to 0 so that HtmlUnit would run properly.  By setting this value, we can use 0th based arrays, just like watir-webdriver does in Celerity.

Next we setup our Cucumber Before block in hooks.rb:


In here, we simply instantiate the object.  In my case, we also needed to setup some URLs and turn off SSL for our Celerity testing.

The Rakefile is probably a bit more complex due to our desire to run both headless and browser tests on our CI box, but it is helpful, since it uses the :browser task to setup the BROWSER environment variable.


Finally, to help our QA folks, three batch files to execute the tests.






Hopefully you'll find this useful. Please let me know if you do.

Random Cucumber/Ruby Things I Learned

Some random stuff I learned as a "seasoned" Java developer, working in Ruby and more specifically Cucumber for the first time this year, and not having a clue as to what I was doing. I realize some of this stuff will be mind numbingly obvious to "seasoned" Ruby devs, but thought it would be interesting to share regardless.

Some background. On my project some "legacy" Cucumber tests existed. Legacy, you ask? How can Cucumber tests be legacy? Simple, they were overly fragile, and violated DRY over and over again. Therefore, since my overall group has chosen Cucumber as the ATDD tool of choice (and by extension Gherkin as the Story card language of choice), it was a good idea to get in and refactor the tests. I had a good example in how to do this in the blog posts by Jeff Morgan, and I fortunately had someone who I could tap on the shoulder on a sister team in Joel Helbling. So with the goal of learning some Ruby in mind, and a copy of RubyMine I began tearing up the soil and planting cukes.

Firstly, when you're working with multiple versions of Ruby, in my case, JRuby and Ruby for Windows, you might need to modify your Gemfile by the platform. In my case, I needed JRuby to run the tests the same way as the legacy tests were running. JRuby drove Celerity (yay HtmlUnit!) tests on Hudson, while native Ruby drives Waitr/FireWatir browser driven versions of the same tests.

So, although it's very obvious after the fact, here's a clue...there is no Celerity for Ruby. HtmlUnit is a Java library, and Celerity is just a Waitr API based wrapper of it. So it only works on JRuby. I thought the Gemfile would just handle that, but I was very very wrong. Therefore, you have to do something like this:

platforms :jruby do
 gem "celerity", "0.8.2"
 gem "syntax"
end

if RUBY_PLATFORM =~ /mswin|mingw/ #Jruby no likey mingw
 platforms :mswin, :mingw do
   gem "watir", "1.6.7"
   gem "watir-webdriver"
   gem "win32console"
   gem "win32-clipboard"
 end
end

This works out pretty well, just don't check in your Gemfile.lock. Ideally your QA testers will be running the Watir versions, while CI will maintain sanity when they can't. If you don't have QA testers, then I suggest having two versions of the project open, with separate Gemfile.lock that represent each platform.

I'm certain there's a better way to do this...but I've not discovered it yet.

Another random thing. If you want to pass a parameter (like a table) from one Cucumber step definition to another, you use a comma at the end of the line and pass the parameter in.

In my case, I was developing tests that triggered various UI based errors on each page during a process to say retrieve a password.

So I had a step definition that said:

When /^I forget my password and enter the credentials$/ do |table|
  # table is a |username|account_number|
  #blah blah blah with username and account_number
end

But what do I do when I have passed this step and need to enter security questions? I want to go through the successful parts of this step to enter the next step. How do I do this?

When /^I forget my password and enter the credentials and answer the security questions$/ do |table|
  # table is a |username|account_number|answer1|answer2|answer3|
  When 'I forget my password and enter the credentials', table
  # Do stuff to answer the security questions

See that comma at the end of the step definition call? It passes table to the next step and lets it grab username and account_number out of the hash. I couldn't find this anywhere in documentation, which means it's painfully obvious to Ruby folks, but it's not if you don't know it. I'm sure it's documented somewhere, but I've failed at finding it. But boy discovering that was quite useful.

One last thing that I can think of right now. For some reason the version of JRuby I have to use doesn't like to use rspec. So similarly, to the first problem, we need to do strange things to make JRuby happy in env.rb:

require 'rubygems'
if RUBY_PLATFORM =~ /java/ #JRuby is strange
  require 'spec'
else
  require 'rspec'
end

Hopefully these tips will be of some use to folks like myself who don't know enough Ruby to be dangerous...yet.