Another update in our team’s journey to functional testing nirvana via Specs and Selenium.

We discovered an earth-shattering revelation (not): Internet Explorer Sucks. Unfortunately, it’s the “target” browser for a certain project we’re working on, therefore we’re stuck with it. Being stuck with it means finding a way to make it suck sufficiently less that it will actually run our lovely tests, which Firefox (and several other browsers) digest quite happily.

One way in which IE sucks is that it’s painfully slow – test that take a few seconds on Firefox timeout after several minutes in IE. If you keep cranking out the timeout they might eventually pass, but we want test results in our lifetimes, so we set about finding a way to get our feedback out of IE a bit faster.

It turns out one of the very slow things about IE is asking for an element via an xpath, and we were doing this a lot. We tried reducing the number of places we did this, and indeed that helped a bit, but there were some things on our existing app that were really hard to refer to without xpath. To ease the pain further, we took a slight detour. Instead of asking Selenium for a specific xpath, which then caused Selenium to ask the browser, we fetched the entire output of the browser (via the getHtmlSource method in Selenium).

What we’d like to do instead, where possible, is simply make assertions on the XML of the entire page – so we get it just one time, and so that we don’t depend on the Xpath implementation of the browser to find bits of our page for us.

Unfortunately, we still can’t ask anything xpath-ish of it, as although it bears a close resemblance to XML, it’s not really html, so any parser we could find choked on it.

Enter HtmlCleaner a handy little jar that can digest even the most disguisting HTML and deposit clean shiny XML in it’s place. Now we could take our XML version of what the browser spat out and finally make Xpath calls to it.

Now Scala itself has excellent support for XML (see the whole book here), but it’s support for Xpath is… interesting. Xpath is actually done via a series of methods and functions, meaning although it is very powerful, it’s not the same xpath as you might be used to, or, more importantly in our case, not the same as you could expect a browser to understand. This means we could either re-write all our Xpath queries (and make it harder to use Selenium IDE or Firebox to help us write new ones), or find another way…

Enter Jaxen, a handy Java lib with full Xpath support, without all the weight of other solutions. Of course, as Specs is written in Scala and Scala happily uses any existing Java lib, we could just drop a jar into our “lib” dir and work some magic.

We needed to convert the HTML produced by Selenium into XML, then turn it into a DOM document that Jaxen can process, so we can ask questions in xpath of the resulting Jaxen document.

This little incantation:

 val cleanerAndProps = initCleaner

  private def initCleaner = {
    var cleaner : HtmlCleaner = new HtmlCleaner();
    var props : CleanerProperties = cleaner.getProperties();

    props.setTranslateSpecialEntities(true)
    props.setRecognizeUnicodeChars(true)
    props.setOmitComments(true)
    (cleaner, props)
  }

Returns a tuple of the HtmlCleaner instance and it’s properties, which we can then use for our conversion somewhat like so:

def getHtmlText(element: String) = {
      var node : TagNode = cleanerAndProps._1.clean(selenium.getHtmlSource())
      val domSerializer: DomSerializer = new DomSerializer(cleanerAndProps._2)
      val document: org.w3c.dom.Document = domSerializer.createDOM(node)
      val xpath = new DOMXPath(element)
      xpath.stringValueOf(document)
  }

This method then takes a string (actually an xpath query), “cleans” the HTML from Selenium, then serializes it into a DOM (a regular org.w3c.dom.Document), allowing us to use Jaxen’s DOMXpath object to fish out the string value of whatever our Xpath query returns, allowing us to say things like:

manufacturerSelected = getHtmlText("//table[@class='DataList']//a")

In our tests.

Of course, we can dress this with other methods that get attributes and other things from the DOM tree, but you get the point.

The nice part about all this is that it doesn’t require a new round trip to the browser – we can assert as many things as we want and fish out as many values as we need directly from the resulting page in milliseconds now, instead of several seconds (or worse for IE) before.

So now we can run our tests much faster than before – but we’re writing a lot of tests, so it’s going to get slow again if we can’t scale better, even with the optimization with Jaxen and HtmlCleaner.

So we looked into Selenium Grid. Before, we had our Ant script that ran the tests (whether locally or on TeamCity), fire up a Selenium RC Server before the run, and shut it down again afterwards. This was time consuming, and problematic as we added more suites of tests. We wanted our tests to be able to run all at the same time, but each Selenium server (which does the communicating with the browser), needs to be on a different port for this to happen. Gah – this was getting complicated.

Selenium Grid, however, address this problem and several more at once.

It allows you to start a single “Hub” server, on one port (4444 by default). This hub then redirects load from tests to one of a number of actual Selenium RC servers that you fire up and leave running long-term.

In our case, we started 3 RC servers to talk to FireFox, and one for IE (it’s not a good idea to run more than one IE RC server on a single machine, as IE doesn’t play nice with other IE’s).

Then on other VM’s we can fire up even more RC servers – both IE and FireFox flavors. Selenium Grid gives you a nice control panel to see the servers and which ones are available, and you can add new servers (and remove them) while the grid is up.

The max number of test runs you can have going at once is now the total number of Selenium RC servers in your grid, allowing many short sets of tests to all run at once, giving much faster actual elapsed time before you get feedback when something breaks.

Of course, one of the benefits of this new approach with Selenium Grid is that I can now test if something works via IE on Windows – from my Mac! For that matter, any combination of browsers and operating systems that we’ve got an RC server for can be tested, all at once. Good stuff.

There are still plenty of frustrations in this process – if you kill a test suite before it finishes, for instance, it appears that the Selenium RC server it was talking to remains unavailable forever, and when you restart the hub, you have to restart all of the RC servers… but it’s a step up from where we were.

I project I happened across a while ago is starting to intrigue me more and more: Specs

It’s a framework written in Scala designed to support Behavior-Driven design and testing. This style of testing encourages tests to describe the be “behavior” of the class or component under test, ideally in way that is easy to understand by both developers and non-developers alike.

How does this differ from other forms of testing? Mostly in the way functionality is described – a feature is expressed in terms of a series of interactions with the system under test, describing functionally in a narrative form. We don’t describe a functional technically, as in “passing 43 should result in a response of ‘ABC’”, but in terms of valuable feature interactions, such as “when a valid login is entered, the foo link becomes available to be clicked”, for example.

Behaviour-driven design and development make sure we’re developing code that implements the right features, as opposed to test-driven development, which focuses more on developing the code correctly and accurately. BDD and BDT is more about the “what” as opposed to the “how”, in other words.

If the subject-area expert for the system under development can understand the language of the test, then it becomes more of an executable specification than just a test – it may even form the acceptance criteria for a story or feature, especially when multiple such specifications are chained into a narrative.

Flowing from the oft-described “As a… I want… So that…” way of describing a story, a narrative describes a sequence of flow through an application, describing and verifying the behavior of the system at each step in the narrative.

Specs supports this very nicely, allowing you to write in a DSL (Domain-Specific Language) which is nonetheless a fully capable programming language at the same time (in this case, Scala).

The basic unit of a Specs test is the “specification”, implying, appropriately, that you literally write executable specifications – documentation that actually verifies that it properly describes the system being described!

Scala is, by design, an extremely extensible language, so it’s support for DSL’s such as specs is very good – it doesn’t burden you with a lot of unnecessary syntax or verbosity, so the resulting specification is highly readable and relatively english-like. So much so that it’s even possible for a domain expert non-developer to write, much less read, the executable specifications – especially if they have examples to go from.

Scala fits seamlessly into any JVM-based environment, plays nicely with IDEs, Ant, Junit, Maven, and all the tools you might already be using to build your systems, so it’s adoption doesn’t cause much of a ripple effect. Scala files simply live in a directory structure inside an otherwise all-Java project, for instance, and are compiled to .class files, just like Java.

Specs also leverages other powerful test frameworks, such as Mockito, JMock and Scalacheck, among several others.

My team is using Specs in two modes, so far: One is as a part of the primary build process, just like you’d run your JUnit or SpecsTest tests. The Scala tests in this case are used as either actual unit tests (although expressed in a behavior-driven way) or in-process functional tests (if they include several classes or a whole subsystem under test).

The other way we use Specs is as a set of stand-alone acceptance tests. In this mode we run Specs externally from the system under test. This is especially useful for our systems that rely heavily on REST web services. In these cases we write our narratives to assert various responses from the REST services found at various URLs, while launching Specs from Ant in it’s own VM on another box entirely. These are true “black box” tests, where the tests have no knowledge of or access to the code under test – they must interact with the running system in a very “real world” way. This provides an ideal environment for acceptance tests, especially given the readability of the Specs specificiations. Our user-proxies also like the fact that the test itself can’t be influencing the code “inside the box”, as it’s easy for programmers to write “happy path” tests

Of course, once you’ve used specs and Scala for your testing, you may be tempted to infiltrate it into your other development, but that’s another post