In a project I was tinkering with lately I found the complexity getting to the point where I could use component dependency injection. There are in fact a number of techniques to do this in “plain” Scala, but they generally seem more painful to me than just dropping in a lightweight DI framework, so I went with Guice.

As it happens, Guice works with Scala very nicely, and is very unobtrusive. There’s a few minor things I haven’t quite figured out yet, but I’d like to share what I’ve discovered so far.

After adding this section to my project POM:

 
            com.google.inject
            guice
            2.0
        

I had all I needed of Guice embedded in my app. Now I can scratch up a series of tests that exercise all of the different ways use Guice from Scala. Let’s start with the simplest, and work our way up:

 @Test
  def basicDependencyInjection {
    class ScalaModule extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).to(classOf[SomeService])
      }
    }

    class ScalaModule2 extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).to(classOf[SomeOtherService])
      }
    }

    val injector = Guice.createInjector(new ScalaModule)
    val component = injector.getInstance(classOf[MyComponent])
    assertEquals("someService", component.callTheService)

    val injector2 = Guice.createInjector(new ScalaModule2)
    val component2 = injector2.getInstance(classOf[MyComponent])
    assertEquals("someOtherService", component2.callTheService)
  }

So let’s explore the test a bit. I’m using Scala to build a JUnit test, as shown by the annotation on line 1. One of the advantages of this is that it’s easy to use Scala’s namespace capabilities to build test classes that are only visible within the test itself, which is what I’m doing starting line 3. I create a class call ScalaModule with extends Guice’s base class AbstractModule. I override one method called “configure”, which sets up the bindings of interfaces to implementations for this Guice module. (This is all covered in detail in Guice’s doc).

On line 6 I bind the interface “AService” to an implementation of AService called “SomeService”. Here’s what they look like (these classes are defined later in the file – I don’t embed them inside the test class as Guice didn’t let me).

Here’s the trait for the service:

trait AService {
  def service: String
}

And a simple implementation:

class SomeService extends AService {
  def service(): String = "someService"
}

Now I have a component that uses that service, called MyComponent. It looks like this:

class MyComponent @Inject()(val service: AService) {
  def callTheService(): String = service.service
}

Note the annotation on the first line. This is how you put an annotation on the constructor in Scala, Annoyingly, the () is apparently required. This Inject annotation indicates that when this component is requested from Guice, it should have the appropriate implementation of Service injected.

So now to return to our test, on line 10 you’ll see we are building a second module, this time binding the AService interface (trait, actually) to the SomeOtherService implementation, which looks like this:

class SomeOtherService extends AService {
  def service(): String = "someOtherService"
}

Then we get into the meat of the test in line 17. First, we ask Guice for our injector, the “god object” for Guice, from which you get all the other object you’ll need. In the first case, we use our ScalaModule class. Then we ask this injector for our component, and it constructs the component for us, injecting the dependency as necessary. In this situation, we have not told Guice that MyComponent is a singleton (we’ll deal with that later – in several different ways), so it builds a new one for us every time.

On line 18, we make the assertion that proves that our MyComponent instance has in fact been built with the “SomeService” instance, as opposed to any other instance, by verifying our string that SomeService responds with.

Then we do the whole thing again, except this time asking for the ScalaModule2. ScalaModule2 is where we have the SomeOtherService implementation of AService wired up, so when we assert on line 23, we now see the “someOtherService” string returned instead, which means that we’re properly wired, as expected.

Now let’s give Guice a little harder work, exploring some scenarios we’ll see in more sophisticated applications.

In many situations, we’ll want to avoid creating a new instance of our service object every time we use it. There are a couple of ways to do this with Guice.

@Test
  def instanceBindingExample {
    // if you want to bind to a specific instance of a class...
    class InstanceExampleModule extends AbstractModule {
      @Override
      protected def configure() {
        val instance1 = new InstanceService("instance1")
        bind(classOf[AService]).toInstance(instance1)
      }
    }
    val injector = Guice.createInjector(new InstanceExampleModule)
    val service = injector.getInstance(classOf[AService])
    assertEquals("instance1", service.service)
  }

In the code above, we’re configuring our module, called InstanceExampleModule, with a binding that specifies our AService trait uses a specific instance of the class called InstanceService.

Here’s InstanceService:

class InstanceService(val value: String) extends AService {
  def service(): String = value
}

Back in our test, line 13 ensures that we’re actually talking to the right service, as we fed the service it’s return string (“instance1″) when it was created in the module.

Ok, so now we can inject services, and specify an instance of a service instead of a newly-created one every time. That’s a good start, but we want to do more.

What if we have to do some more work to instantiate our singleton instance? E.g. what happens if it takes a whole method to create the service?

Guice has that covered as well, like so:

Here you see a new module being built, but this time we don’t actually say anything in the “configure” method. Instead, we annotate another method in the module with the @Provides annotation – this means to Guice “whenever you need a thing of the type this method returns, call this method”. Then we verify that that’s in fact what’s happening by creating the injector and asking for the service.
Ok, so now we can create instances of services however we want, but we’ve still only created a single instance of a given trait. What if we’ve got a whole whack of services that implement our interface? Ok, that’s also easy:
@Test
  def bindingWithNameAnnotationExample {
    // what if I have two implementations of the same interface, how do I say which one I want?
    class ScalaModule extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).annotatedWith(Names.named("foo")).to(classOf[SomeServiceNamedFoo])
        bind(classOf[AService]).annotatedWith(Names.named("bar")).to(classOf[SomeServiceNamedBar])
      }
    }

    val injector = Guice.createInjector(new ScalaModule)
    val barComponent = injector.getInstance(classOf[MyBarComponent])
    val fooComponent = injector.getInstance(classOf[MyFooComponent])
    assertEquals("bar", barComponent.callTheService)
    assertEquals("foo", fooComponent.callTheService)
  }

Now we’ve got a couple of services to declare:
class SomeServiceNamedFoo extends AService {
  def service(): String = "foo"
}
class SomeServiceNamedBar extends AService {
  def service(): String = "bar"
}

The services themselves both implement our trait, of course, but now we have two different components, one which needs the Foo service, other other one needs the Bar:
class SomeServiceNamedFoo extends AService {
  def service(): String = "foo"
}
class SomeServiceNamedBar extends AService {
  def service(): String = "bar"
}

In our test, you can see we ask the injector for the two components. What’s very interesting to me is that the component themselves are not listed in the module. Why that’s important is that unlike Spring (to pick an example many people know), you don’t need any configuration provided at all for stuff that Guice can work out on it’s own. This is a good thing.
Anyway, you can see by our asserts that in this situation the Foo service gets the Foo implementation of the AService (SomeServiceNamedFoo), and the Bar service gets SomeServiceNamedBar instead.
Now that we can handle multiple implementations, let’s go back and consider the singleton vs. new instance issue from another angle: Scala has no concept of the idea of “static” classes, but it does have companion objects, which are, in all cases I’ve seen, better.
If we declare a service like so:
object SingletonService extends AService {
  def service(): String = "singleton"
}

and another (non-singleton) implementation, like so:
class NonSingletonService extends AService {
  def service(): String = "nonsingleton"
}

Now we can write a test that tries this out like so:
  @Test
  def singletonOrPrototype {
    class ScalaModuleSingleton extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).toInstance(SingletonService)
      }
    }

    class ScalaModuleNonSingleton extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).to(classOf[NonSingletonService])
      }
    }

    val singletonInjector = Guice.createInjector(new ScalaModuleSingleton)
    val singleton = singletonInjector.getInstance(classOf[AService])
    assertEquals("singleton", singleton.service)
    val secondInstance = singletonInjector.getInstance(classOf[AService])
    assertSame(singleton, secondInstance)

    val nonSingletonInjector = Guice.createInjector(new ScalaModuleNonSingleton)
    val nonsingleton = nonSingletonInjector.getInstance(classOf[AService])
    assertEquals("nonsingleton", nonsingleton.service)
    val secondNonSingletonInstance = nonSingletonInjector.getInstance(classOf[AService])
    assertNotSame(nonsingleton, secondNonSingletonInstance)
  }

The money here comes at the line that binds the classOf[AService] to a singleton – in this case, the SingletonService. This isn’t a class, so we don’t use to(classOf[SingletonService]), we say toInstance(SingletonService) instead.
As you can see from the test, this works very nicely, and gives us a much more Scala-canonical way to do singleton.
However, there are times when we want to declare a regular Scala class, and have Guice only instantiate one of em for us, and handle it as a singleton that way (some examples might be when we’re using a constructor to inject other dependencies – although we can use field injection for this, which we’ll look at later).
We can annotate our class explicitly to tell Guice it’s a singleton like so:
@com.google.inject.Singleton class SingletonClassService extends AService {
  def service(): String = "singletonClass"
}

Now we can write a test like so to prove that we get the same instance of our service, even if we ask the injector for it multiples times:
@Test
  def singletonClassExample {
    class ScalaModuleSingleton extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).to(classOf[SingletonClassService])
      }
    }

    val singletonInjector = Guice.createInjector(new ScalaModuleSingleton)
    val singleton = singletonInjector.getInstance(classOf[AService])
    assertEquals("singletonClass", singleton.service)
    val secondInstance = singletonInjector.getInstance(classOf[AService])
    assertSame(singleton, secondInstance)
  }

The assertSame in JUnit asserts that we have the exact same instance of an object, not that they’re only equal.
One last scenario: What is the service isn’t the singleton, but the component is? E.g. what if we have an object instead of class that needs a service injected into it? Turns out Guice can handle that nicely as well. The only extra work we need is to define a trait for the binding of our component, like so, with the service injected into the object:
trait SingletonComponentInterface {
  def callTheService:String
}
object SingletonComponent extends SingletonComponentInterface {
  @Inject val service:AService = null
  def callTheService(): String = service.service
}

Now we can write a test to make sure this is doing what we want:
@Test
  def injectIntoSingletonExample {
    // if you want to inject into an object instead of a class
    class InstanceExampleModule extends AbstractModule {
      @Override
      protected def configure() {
        val instance1 = new InstanceService("instance1")
        bind(classOf[AService]).toInstance(instance1)
        bind(classOf[SingletonComponentInterface]).toInstance(SingletonComponent)
      }
    }
    val injector = Guice.createInjector(new InstanceExampleModule)
    val component = injector.getInstance(classOf[SingletonComponentInterface])
    assertEquals("instance1", component.callTheService)
  }

That’s it for my experiments so far. Here’s my conclusions, please comment with your experiences:
The Good, The Bad, and the Ugly
The Good

There was a lot I liked about Guice in Scala. First, no XML was injured in the making of these examples – everything was code, and fully type-safe (I mentioned it was Scala, right?) I always find myself writing a “smoke test” for my Spring apps to verify I haven’t made a typo in my “beans.xml” file – here I didn’t need to. Yes, I know you can do XML-free Spring as well, but it’s the norm with Guice, and it feels a lot lighter weight.
I don’t need to configure anything for classes that just get injected. This means that the configuration I do need to write is very short, and because it’s a class, it’s extensible. In my “real” project, for instance, I was able to write a module that created services that use static lists for their data (instead of reading and writing to a real database, for instance), then extend that class with an OSGi-aware version that used the “real” services from other modules. No repeated work, no wiring, no config – it just worked.
Auto-handling of singletons, both with companion objects and classes, as I prefer: It was nice to be able to write Scala in a very Scala-like way, and just have the DI framework slot right in whichever way I needed it to, as described above.
The Bad

I have to annotate my classes with Guice-specific annotations, although there is a JSR on the way to standardize these, which makes it a bit less objectionable. What happens if I need to inject to a class I don’t have source for (e.g. something in an existing Java library that I’m calling? I also haven’t tried this in a Servlet environment, although I understand there’s a Guice extension for that, so I don’t expect any trouble there.
The Ugly

I have to (it seems) use the @Inject() format, not just @Inject, and I have to insert it in what looks like an odd place in the declaration. Not a big deal, and I suspect I’ll get used to it.
            

I project I happened across a while ago is starting to intrigue me more and more: Specs

It’s a framework written in Scala designed to support Behavior-Driven design and testing. This style of testing encourages tests to describe the be “behavior” of the class or component under test, ideally in way that is easy to understand by both developers and non-developers alike.

How does this differ from other forms of testing? Mostly in the way functionality is described – a feature is expressed in terms of a series of interactions with the system under test, describing functionally in a narrative form. We don’t describe a functional technically, as in “passing 43 should result in a response of ‘ABC’”, but in terms of valuable feature interactions, such as “when a valid login is entered, the foo link becomes available to be clicked”, for example.

Behaviour-driven design and development make sure we’re developing code that implements the right features, as opposed to test-driven development, which focuses more on developing the code correctly and accurately. BDD and BDT is more about the “what” as opposed to the “how”, in other words.

If the subject-area expert for the system under development can understand the language of the test, then it becomes more of an executable specification than just a test – it may even form the acceptance criteria for a story or feature, especially when multiple such specifications are chained into a narrative.

Flowing from the oft-described “As a… I want… So that…” way of describing a story, a narrative describes a sequence of flow through an application, describing and verifying the behavior of the system at each step in the narrative.

Specs supports this very nicely, allowing you to write in a DSL (Domain-Specific Language) which is nonetheless a fully capable programming language at the same time (in this case, Scala).

The basic unit of a Specs test is the “specification”, implying, appropriately, that you literally write executable specifications – documentation that actually verifies that it properly describes the system being described!

Scala is, by design, an extremely extensible language, so it’s support for DSL’s such as specs is very good – it doesn’t burden you with a lot of unnecessary syntax or verbosity, so the resulting specification is highly readable and relatively english-like. So much so that it’s even possible for a domain expert non-developer to write, much less read, the executable specifications – especially if they have examples to go from.

Scala fits seamlessly into any JVM-based environment, plays nicely with IDEs, Ant, Junit, Maven, and all the tools you might already be using to build your systems, so it’s adoption doesn’t cause much of a ripple effect. Scala files simply live in a directory structure inside an otherwise all-Java project, for instance, and are compiled to .class files, just like Java.

Specs also leverages other powerful test frameworks, such as Mockito, JMock and Scalacheck, among several others.

My team is using Specs in two modes, so far: One is as a part of the primary build process, just like you’d run your JUnit or SpecsTest tests. The Scala tests in this case are used as either actual unit tests (although expressed in a behavior-driven way) or in-process functional tests (if they include several classes or a whole subsystem under test).

The other way we use Specs is as a set of stand-alone acceptance tests. In this mode we run Specs externally from the system under test. This is especially useful for our systems that rely heavily on REST web services. In these cases we write our narratives to assert various responses from the REST services found at various URLs, while launching Specs from Ant in it’s own VM on another box entirely. These are true “black box” tests, where the tests have no knowledge of or access to the code under test – they must interact with the running system in a very “real world” way. This provides an ideal environment for acceptance tests, especially given the readability of the Specs specificiations. Our user-proxies also like the fact that the test itself can’t be influencing the code “inside the box”, as it’s easy for programmers to write “happy path” tests

Of course, once you’ve used specs and Scala for your testing, you may be tempted to infiltrate it into your other development, but that’s another post

Recently I’ve had the opportunity to consider the right qualities of a tool and/or framework for acceptance testing.

Acceptance tests can be found at a number of different levels, depending on how the story criteria is expressed. (See my previous post on levels of testing) . Often they’re called “executable specifications” if they’re written in such as way as to describe the behavior of a system in a given scenario.

Often they are functional or integration tests, and generally they are best expressed as “black box” tests, that is, tests that have no awareness of the internals of the code or component being tested – all they see is what goes in and what comes out, and they assert their success or failure based on those elements alone.

An acceptance test must be comprehensible to the story author, or whatever domain expert is going to actually do the “accepting” that the story has been satisfied. If they simply take the word of a developer that this big ball of code they see in front of them is testing what they said it should, that’s not as good as if they can actually read and understand the executable specification (test) themselves. Ideally, they should even be able to author their own acceptance tests, without a developer involved – perhaps using existing tests as an example.

So, some of the criteria for a good tool might be:

1. High Visibility: Something that’s easy to see by all stakeholders
2. Easily Understood: Something that’s easy to at least read by any non-developer domain expert
3. Easily Authored: Something that’s easy to learn to write
4. “Black Box”: Something that has no knowledge of the internals of the code under test, that only interacts with it from “outside the box”
5. Realistic: Something that tests a version of the application deployed in a realistic environment, as close to production as possible.

Given these criteria, should the tests be part of the build process for the piece of code under test? For acceptance tests that span modules, this is not practical – you can’t actually run the test when you build a module, you can only run it after a certain set of modules has been built (and perhaps even deployed).

I would propose that acceptance tests don’t even belong with the build they’re testing. I think they belong elsewhere, in an independent location where they can be updated as stories are written and verified. Ideally, there should be a shared space for stories for an entire system or suite of applications, as often a piece of criteria spans multiple components – so organizing the tests by component is artificial at best.

Another question that comes up when considering tools for these kinds of tests is whether or not they must be stored and versioned with the code they validate. As a developer, my immediate reaction is “yes” – until I think about it a bit. How does it help me to know what tests a previous version of my code passed or did not pass? All that tells me is where I was in the past, not where I am now. How much inconvenience am I willing to put up with on the part of test authors to get this capability? Dealing with any version control system is extra work, especially for a domain expert/BA who’s not also a developer. I’m now convinced that where I am now is more important than where I was, and being able to easily write and run and make visible tests is more important to me than knowing what happened back in history.

Let’s examine a few different tools and ways of doing acceptance tests and look at the pros and cons:

JUnit
Developers working with Java have a choice of a number of excellent test frameworks, including TestNG and JUnit. They are likely already familiar with them, and the tests fit nicely into the build cycle of Java applications built with any of the more popular project build and management tools, such as Ant, Buildr, Maven, and so forth.

Our acceptance tests are written just like a functional test, in that they fire up whatever context our application should run within, then provide input to it and verify the output with assertions.

This kind of test is not well-suited for acceptance tests for a number of reasons. First, it’s hard to separate the code from the test to ensure we’re truly doing black-box testing. If, for example, we’re within the same VM as the thing we’re testing, it’s very tempting to manipulate objects directly in our test, as opposed to going through, lets say, the publish REST api. This also makes it more difficult to initialize a new testable instance of the application – to be truly independent from it, our test should spawn a whole new JVM from the system under test – which is non-trivial from within Java, although of course re-usable fixtures can be written to take the sting out of it.

This doesn’t help us much when our test spans multiple modules or components, however – then we must either create stubs for each of the components we depend on, or work in a much more complex deployment process to create a “live” version of each of our dependencies.

Finally, a JUnit test is not particularly readable by a non-developer, and not particularly visible, other than as a green or red bar in an IDE or CI server somewhere, so it fails a couple of our most important criteria.

JBehave, easyb, Specs, RSpec
BDD frameworks such as those listed above take our testing power in a different direction. They mostly rely on sophisticated DSL’s, enabling us to write our tests in a much more english-like fashion, often quite readable to the non-developers who have taken the time to learn a bit of the DSL.

They still suffer from some of the other problems described above, however – although some of these tools do offer better output formats to make their results more visible (such as the excellent Forms capability in specs, which can show test results in HTML table formats).

They of course still have a learning curve, as the DSLs in each case are another whole language to be learned.

FitNesse
A different approach can be seen in tools like FitNesse, which allows tests to be created in a Wiki, by editing web pages and inserting special markup to call test “fixtures”. These fixtures still have to be customized to the situation, of course, but with FitNesse we have the potential of being de-coupled from a single module or system under test, and of developing our tests independently of the code altogether.

The table approach of FitNesse still requires the test author to understand the fixtures available to FitNesse – this is essentially FitNesse’s DSL – but of course only a fairly small number of fixtures need be learned to be able to write a wide variety of tests.

Some projects, however, in an attempt to retain the ability to use FitNesse tests to verify previous versions of their application, take the step of checking the FitNesse tests in to their version control system, thereby losing some of that independence.

FitNesse has it’s own ability to track changes to it’s Wiki pages (and thus it’s tests), but this is not tied to the checkins or releases of the software under test.

FitNesse makes it easy for a non-developer domain expert to author tests by using existing tests as templates, then changing the inputs to the fixtures being used to create new tests from the existing building blocks.

These tests and their results are highly visible, especially if the FitNesse wiki is hosted and available to anyone (including developers) to use at any time to verify against a specific deployed instance of the entire suite of applications.

Of course, you’ve still got the issue of deploying a testable instance of your system to deal with in FitNesse. One approach I’ve seen to solve this is to actually have a fixture that can deploy the testable instance as part of the set up for the whole test suite, using versions to indicate the revision of code to be tested – e.g. you have a fixture that says “deploy module X version 123 to test environment 1″, “deploy module Y version 345 to test environment 1″ and so forth, then executes its tests against those deployed instances.

Of course, any database cleanup/reset to get to known state can happen before or just after the deployment, so your tests always start from a known point.

An important part of making this fully automatable is a mutex service: a way to make a call to a known location and say “check out test environment 1″, basically saying that test environment 1 is now busy until it’s released by another call. This ensures that you don’t start another test run on the same environment while it’s still in a transition state.

The mutex can also of course report on who has what environment in use, and since when, to detect failures that might not release a lock.

FitNesse has it’s warts, however, even in a scenario like this, but overall I’ve seen it succeed more often that other approaches for the high-visibility acceptance tests that many projects need, while tools such as easyb, specs, and RSpec are better for describing behaviours within a single module, and executing as part of that module’s build process.

I’ve had reason recently to do some thinking on the various “levels” of software testing. I think there’s a rough hierarchy here, but there’s some debate about the naming and terminology in some cases. The general principals are pretty well accepted, however, and I’d like to list them here and expound on what I think each level is all about.

An important concern in each of these levels is to achieve as high a level of automation as possible, along with some mechanism to report to the developers (or other stakeholders, as required) when tests are failing, in a way that doesn’t require them to go somewhere and look at something. I’m a big fan of flashing red lights and loud sirens, myself

Unit
Unit testing is one of the most common, and yet in many ways, misunderstood levels of test. I’ve got a separate rant/discussion in the works about TDD (and BDD), but suffice it to say that unit-level testing is a fundamental of test-driven development.

A unit test should test one class (at most – perhaps only part of a class). All other dependencies should be either mocked or stubbed out. If you are using Spring to autowire classes into your test, it’s definitely not a unit test – it’s at least an integration test. There should be no databases or external storage involved – all of those are external and superfluous to a single class that you’re trying to verify is doing the right thing.

Another reason to write comprehensive unit tests is that it’s the easiest place to fix a bug: there are fewer moving parts, and when a simple unit tests breaks it should be entirely clear what’s wrong and what needs to be fixed and how to fix it.

As you go up the stack to more and more complex levels of testing, it becomes harder and harder to tell what broke and how to fix it.

Generally unit tests for any given module are executed as part of every build before a developer checks in code – sometimes this will also include some functional tests as well, but it’s generally a bad idea for any higher-level tests to be run before each and every check-in (due to the impact on developer cycle-time). Instead, you let your CI server handle that, often on a scheduled basis.

Functional
Some people suggest that functional and integration are not two separate types, but I’m separating them here. The key differentiation is that a functional test will likely span a number of classes in a single module, but not involve more than one executable unit. It likely will involve a few classes from within a single classpath space (e.g. from within a single jar or such). In the Java world (or other JVM-hosted languages), this means that a functional test is contained within a single VM instance.

This level might include tests that involve a database layer with an in-memory database, such as hypersonic – but they don’t use an *external* service, like MySQL – that would be an integration test, which we explore next.

Generally in a functional test we are not concerned with the low-level sequence of method of function calls, like we might be in a unit test. Instead, we’re doing more “black box” testing at this level, making sure that when we pour in the right inputs we get the right outputs out, and that when we supply invalid input that an appropriate level of error handling occurs, again, all within a single executable chunk.

Integration
As soon as you have a test that requires more than one executable to be running in order to test, it’s an integration test of some sort. This includes all tests that verify API contracts between REST or SOAP services, for instance, or anything that talks to an out-of-process database (as then you’re testing the integration between your app and the database server).

Ideally, this level of test should verify *just* the integration, not repeat the functionality of the unit tests exhaustively, otherwise they are redundant and not DRY.

In other words, you should be checking that the one service connects to the other, makes valid requests and gets valid responses, not comprehensively testing the content of the request or response – that’s what the unit and functional tests are for.

An example of an integration test is one where you fire up a copy of your application with an actual database engine and verify that the operation of your persistence layer is as expected, or where you start the client and server of your REST service and ensure that they exchange messages the way you wanted.

Acceptance
Acceptance tests often take the same form as a functional or integration test, but the author and audience are usually different: in this case an acceptance test should be authored by the story originator (the customer proxy, sometimes a business analyst), and should represent a narrative sequence of exercising various application functionality.

They are again not exhaustive in the way that unit tests attempt to be in that they don’t necessarily need to exercise all of the code, just the code required to support the narrative defined by a series of stories.

Fitnesse, RSpec and Green Pepper are all tools designed to assist with this kind of testing.

Concurrency
If your application or service is designed to be used by more than one client or user, then it should be tested for concurrency. This is a test that simulates simultaneous concurrent load over a short period of time, and ensures that the replies from the service remain successful under that load.

For a concurrency test, we might verify just that the response contains some valid information, and not an error, as opposed to validating every element of the response as being correct (as this would again be an overlap with other layers of testing, and hence be redundant).

Performance
Performance, not to be confused with load and scalability, is a timing-based test. This is where you load your application (either with concurrent or sequential requests, depending on it’s intended purpose) with requests, and ensure that the requests receive a response within a specified time frame (often for interactive apps a rule is the “two second rule”, as it’s thought that users will tolerate a delay up to that level).

It’s important that performance tests be run singly and on an isolated system under a known load, or you will never get consistency from them.

Performance can be measured at various levels, but is most commonly checked at the integration or functional levels.

Load/Scalability
A close relative of, but not identical with concurrency tests are load and/or scalability tests. This is where you (automatically) pound on the app under a simulated user (or client) load, ideally more than it will experience in production, and make sure that it does not break. At this point you’re not concerned with how slow it goes, only that it doesn’t break – e.g. that you *can* scale, not that you can scale linearly or on any other performance curve.

Quality Assurance
Many Agile and Lean teams eschew a formal quality assurance group, and the testing such a group does, in favor of the concept of “built in” QA. Quality assurance, however, goes far beyond determining if the software perform as expected. I have a detailed post in the works that talks about how else we can measure the quality of the software we produce, as it’s a topic unto itself.

Alpha/Beta deployments
Not strictly testing at all, the deployment of alpha or beta versions of an application nonetheless relates to testing, even though it is far less formalized and rigorous than mechanized testing.

This is a good place to collect more subjective measures such as usability and perceived responsiveness.

Manual Tests
The bane of every agile project, manual tests should be avoided like the undying plague, IMO. Even the most obscure user interface has an automated tool for scripting the testing of the actual user experience – if nothing else, you should be recording any manual tests with such a tool, so that when it’s done you can “reply” the test without further manual interaction.

At each level of testing here, remember, have confidence in your tests and keep it DRY. Don’t test the same thing over and over again on purpose, let the natural overlap between the layers catch problems at the appropriate level, and when you find a problem, drive the test for it down as low as possible on this stack – ideally right back to the unit test level.

If all of your unit test are perfect and passing, you’d never see a failing test at any of the other levels, theoretically. I’ve never seen that kind of testing nirvana achieved entirely, but I’ve seen projects come close – and those were projects with a defect rate so low it was considered practically unattainable by other teams, yet it was done at a cost that was entirely reasonable.

A technique I’ve seen a lot in recent years, and I believe for good reason, is using one language (programming language, that is!) to test code written in another.

There are some non-obvious advantages to this that I’d like to explore a bit below, as well as some obvious (but perhaps less important) disadvantages.

On the downside, of course, you need a developer or team conversant in at least two different languages. This is more rare than you might think, although I think more experienced developers have the edge in that they’ve likely been working in more than one language if they’ve been in the industry a while. Also, many developers like to tinker, and will often pick up one of the “low ceremony” languages to do that tinkering with, and end up gaining at least a passing familiarity with a new language as a result.

Another downside (and often the reason this isn’t done more) is a lack of common tool support between the two languages. If you’re working in an IDE that only speaks Java fluently, for instance, you’re less inclined to use another language for your tests, as then you’ve not only got to switch languages, but endure the “context switch” overhead of changing IDE’s midstream as well. If you’re doing real TDD, that’s a lot of context switches.

On the upside, though, there are a number of good reasons you might want to consider using a different language for your tests:

• Intentional context switch: The same reason some people avoid switching languages for testing can actually be an advantage. When you’re tests are in another language, you’ve got a built-in mental gear-change between the code and the tests
• Decoupling: If you’re changing language, you’re more inclined (maybe even required) to limit the touch-points between the tests and the actual code – e.g. you might isolate the interface to a single class that’s made visible in the test. This can be a good thing, as a too-tightly coupled test is a brittle test (and often not a unit test, but that’s a whole ‘nother topic)
• Better test framework: While xUnit-like frameworks abound in just about every language, some of the higher-level and BDD-type frameworks are best in one specific language. RSpec is an excellent example: It’s originally in Ruby (although I understand some attempts have been made to port it), so it might be that you’d consider using Ruby as a test language in order to get the goodness of RSpec. Ditto for easyb, which is written in Groovy (a very close relative of Java, in many ways).

  The reverse could be true: You could write JUnit unit tests for your Scala application, for instance, taking advantage of JUnit 4 and friends being on the Java platform.

• Better language suitability: Some languages are much “lower ceremony” than other – e.g. compare Ruby and Java. You need a lot of scaffolding and fuss around a large Java application, but Ruby is a bit more nimble and in some cases, more expressive. Expressiveness being a key quality of a good test, this might mean testing a Java app with Ruby (or perhaps JRuby, to be specific) is a good plan, especially with the existence of Test::Unit and RSpec and such
• Multi-Lingual project: It’s becoming more and more common to see a project use more than one development language in any case (e.g. for the ‘real’ code, not just the tests). In this case, there’s even less of a barrier to doing your tests in one of those languages – you’re only helping to maintain your fluency in both language as well, which can’t be a bad thing.

So, how about it? Anyone out there have good or bad stories about multi-lingual testing to share?