I recently had occasion to use a trick with test execution that is simple and very helpful, but that I forget about frequently, so I thought I’d pass it along here.

We use Maven to execute our unit (and sometimes functional, but that’s another story) tests for a suite of applications my team works on. Some of our apps have quite a few tests, and even though each unit test is very fast (as good unit tests ought to be), running that many of them still takes a bit of time. We’re also burdened with a very slow CI server, unfortunately, which aggravates that situation.

Unit tests, by definition, do not rely on the operation of any other test – they are self-contained, stateless and repeatable. This means they’re ideally suited to running in parallel, and fortunately, that’s very easy to do with Maven.

Maven uses the Surefire Plugin to do it’s test-running, and it’s configurable to run tests in parallel. Our project uses a parent pom (different from an aggregator pom) to specify common configuration, so our pom files stay lean. In our parent pom, we just add the following:

  org.apache.maven.plugins
  maven-surefire-plugin
  2.5
  
methods
20
  

And we’ve told Maven to run our tests in a maximum of 20 threads, and to make each method run in parallel. (There are other options here for only running classes in parallel, etc, which might be helpful for functional tests).

Just doing that took a minute off the time of a couple of our builds, and a bit off the time of even some of the smaller ones. Of course, it’s even more effective if you’re running on a multicore machine, but even the feeble VM running our CI was faster with it there.

Enjoy!

One of the ways to ensure quality in a software project is to find a set of coding standards that your team can agree on then put automated checks in place to ensure they are adhered to.

In this post I’d like to take a very small example of such a stanard, and show how you can use several different tools to help ensure and measure compliance.

The Example
The simple rule we wil use is the use of curl braces on if statements and similar blocks. Our team reached a concensus that we would always prefer this form

if (condition) {
   block
}

over this

if (condition)
    block

even when there is only one line in the block, where the braces are optional. I’m in the habit of not doing this, so I kept doing it the wrong way, making code just that little bit harder for the rest of the team to read and maintain.

We have an existing codebase of hundreds of thousands of lines, some of which adhere to this rule, some of which doesn’t.

First, I wanted to measure adherence to the rule, so I chose the Sonar quality assurance tool.

Sonar
Sonar installation is straightforward: download the distribution, unzip it and run one script. Sonar runs as a web application, so you can immediate go to your browser and see the initial page, like so:

I’m using Sonar (for now) in it’s default mode with a bundled Derby database. If I were going to use it more heavily, I could install a separate MySQL database, and run Sonar itself inside a container of my choice (it comes bundled with a basic web container so you can get going right away).

For my machine, I ran

bin/macosx-universal-64/sonar.sh console

“Console” runs Sonar without backgrounding the task. Normally you’d use “start”, which works just like a Unix service, and is suitable for running from an init.d-type script immediately. The first time around, Sonar takes a few moments to fire up, as it creates all it’s required database tables and populates them with an initial set of data to get your started.

Sonar can measure many hundreds of different metrics, but for the purposes of this simple experiment I only want the one, so I create a custom profile, called “team”.

I set this profile to be the default, then click to edit it.

Now I can search for the braces rule in checkstyle by setting filter criteria:

You can click the rule name to see a description, and to set any optional parameters on the rule.

I activate the rule, and leave it at the default “Minor” priority.

If I now return to my profile, I have a link to access the generated checkstyle.xml file, the configuration file that is used by checkstyle to configure its rules. This will be important later on.

Now I can go to my project and analyze it by running “mvn clean install sonar:sonar”. With the default configuration, this works only when the build is taking place on the same box that Sonar is running, but it’s simple to set up a settings.xml file to tell the Sonar plugin where to find the Sonar instance, so you can have a single shared Sonar instance for the whole team. This is covered nicely in the Sonar documentation.

One minor hiccup: The released version of the Sonar plugin for Maven doesn’t work with Maven version 3 (still in Beta), which I was using. You can build the source version if you need this support, or I’m guessing it’ll be released about the time Maven 3 might be.

Now that I’ve analyzed my project, I can go back into Sonar and see how well I adhered to my rule:

As I see I have a violation, I can drill down and see the violation in context right in the code:

This is handy for checking the state of a rule against an existing codebase, but I’d like to have more immediate feedback. No problem, I can integrate Sonar into both my IDE and my build process very easily.

IntelliJ IDEA: Immediate Feedback

There’s also a new plugin for IntelliJ IDEA for Sonar. To install, just go to the “Available” tab in your plugin configuration within IntelliJ, right-click and select Download and Install. You’ll have to restart IntelliJ to activate the plugin.

Now you can find the Sonar entry in configuration, and set up a URL to point to your Sonar installation, along with a username and password to log in to Sonar. The default is “admin/admin”.

Having done that, I can now see violations directly in my code, as I open each file, like so:

In the above, I’m hovering over the little “price tag” annotation in the left gutter to see the message. You can also see the usual IntelliJ “yellow bar” on the right side of the screen, which shows the same message.

This lets me know of a rule violation before I even attempt to build, so when I forget my braces, I don’t have to wait to be chastized, IntelliJ can do it for me immediately

Maven: Ensuring Conformance

There is a Maven Checkstyle Plugin that allows us to make a run of checkstyle part of the build process of our project. I configured this plugin in my POM like so:

    4.0.0
    com.jglobal
    experiment
jar
    Experiment
    1.0.0-SNAPSHOT

UTF-8
    

    

        
            junit
            junit
            4.5
            test
        

    

    


                maven-compiler-plugin
                
                    1.6
                    1.6
                
            

                org.apache.maven.plugins
                maven-checkstyle-plugin
                2.5
                
                    
                        org.codehaus.plexus
                        plexus-resources
                        1.0-alpha-7
                    
                
                
                    http://localhost:9000/rules_configuration/export/java/team/checkstyle.xml
                    
                    true
                    true
                
            

                org.codehaus.sonar
                sonar-maven-plugin
                1.8M1
            
        
    


            codehaus
            Maven Plugin Repository
            http://repository.codehaus.org/
            default
            
                false
            
            
                never
            
        
    

As you can see, instead of providing a file system based location for the configuration file for Checkstyle, I just pointed the Checkstyle plugin at my Sonar-generated checkstyle.xml, this way my IDE, Sonar and my build stay in perfect sync, and I have a nice GUI for adjusting my checkstyle rules, instead of having to hand-edit XML.

Now when I run my build, Maven will analyze my project against the checkstyle rules, and report any violations. With code as we’ve seen above, it of course reports a violation, but the build still succeeds.

This is fine, but now that I’ve verified we are adhering to this rule, I’d like to actually fail the build if we slip back. Also easily done – in Sonar, I just change the level of violation for my one and only rule to “Critical” in my profile. Now when I run my Maven build, sure enough, the build fails, noting the fact that I’m not adhering to a Critical rule.

As my codebase gets cleaner and cleaner, I can up the priority of the rules I really care about until there are no more violations – then Maven and Sonar can guarantee it stays that way. Of course, in a multimodule build (e.g. with an aggregator POM), I might want to create a parent POM, and specify all this configuration there. (It’s generally not a good idea to have the aggregator and the parent in one pom, use two).

This is of course a trivial example of Sonar, IntelliJ and Maven working together. Sonar can apply over 700 rules right out of the box, and more can be added via plugins easily.

What I find good is that I can synchronize all my tools off one common and agreed-upon rulebase, without having to tweak XML files all over the place to gradually increase my quality standards over time.

I’ve been sold for some time on the benefits of highly modular software development (and deployment, for that matter, almost equally importantly).

However, I’m aware that not everyone is on the same page, or has the same definition of “modularity” in the first place, so I thought I’d collect and enumerate the things that have convinced me, as best I can.

I’ve been in the software business for a very long time, and sometimes I have a preference or leaning towards a certain technique or technology. Sometimes that preference has clear causes behind it, but sometimes not. Modularity is one of the latter, where I know it’s not only good, but critically important, but it’s hard for me to immediately say why or how I know that in a succinct way.

So, I apologize for the length of this explanation, as I’ve not yet taken the time to make it shorter

Modularity Defined
First things first: What do I mean by modularity?

In short, I mean a system comprised of relatively small self-contained units. The idea of modularity has caught on much more completely in many other disciplines, but far less so in software. In manufacturing, especially mechanical and electronic, in business, especially finance, the idea of modularity is well established. Modularity is used to increase and maintain quality, and reduce cost while increasing output. In the automotive industry, for example, extensive use of standardization and modularity allows us all to afford a car, as opposed to just the wealthy or the mechanically inclined.

In the software world, the use of modularity is still woefully small, however. When it is employed, though, some spectacular successes have arisen. One of the oldest examples is Unix, arguably the most successful operating system ever. Since 1969, the “do one thing, do it well, and play well with others” modular design of Unix has been the basis of it’s success and flexibility, allowing it to span everything from embedded devices to mainframes.

The Java Virtual Machine (JVM) is another realm where modularity is on the rise, and starting to have an impact. Many different approaches are available, and some of them work with any of the large number of languages available on this platform.

Increase overall success rate of projects
In the many software projects I’ve been involved with over the years, I’ve noticed a few trends. Smaller, well-contained projects have a higher success rate than larger and more poorly contained projects. (By contained, I mean the number of places and ways the project interacts with other projects is small). I’m sure there are a host of reasons for this, but part of it is simple comprehension – small projects are generally easier to understand than big one, so the number of times things get done wrong is reduced.

A system comprised of well-isolated modules tends to be more like a set of small projects that happen to work together, so I find this reason alone a plus for modularity.

But this hides much of the detailed reasoning behind why modularization works, so let’s dig a bit deeper…

Makes the API between components explicit, not hidden
Sometimes I heard the argument that modularity is complex, that it actually increases the difficulty of working with a system. I find that doesn’t hold up under examination: when modularity seems complex, it’s probably because it’s being done wrong, or being retrofitted to a system that was not designed with modularity in mind. What you’re really seeing in this situation is that the attempt to modularize is exposing the complexity that was already there.

When a system is properly designed in a modular fashion, the APIs between modules are small and well-defined, with a limited number of cross-module dependencies.

Reduce unnecessary complexity
By reducing a large complex system into a series of small and better understood modules, we’ve reduced the complexity we have to deal with at any one time. This means that the overall large system only has to deal with how the exposed APIs of our modules will be fitted together, treating the modules themselves as “black boxes” that perform specific functions, without worrying about how they perform it.

In both cases, we’ve reduced the complexity we need to deal with.

I specify here “unnecessary” complexity, as of course the complexity of the issue itself doesn’t get any simpler just because we modularize it – but if we don’t have to deal with a lot of tangled plumbing, we can concentrate on the real issue.

Increase testability
A module that performs a single function is much easier to test exhaustively than an entire monolithic system. The number of interdependence is limited, meaning we can do more of our testing at the unit and orchestration level than at the functional or integration level, meaning our feedback time is improved and our coverage likely increased.

This means our development velocity can increase, as the impact of changes can be evaluated more easily and more quickly.

Support Composition
When you have a number of smaller building blocks, it’s relatively easy to reshuffle those blocks to change the behaviour and functionality of your overall system. This is making use of composition – connecting blocks together to do more than they can on their own, and its a pattern that is supported well in a modular environment.

Although composition is often thought of in the context of re-usable modules, when building a new system, it’s also valuable in modules that are only used within one system, as it allows the large-grained behavior of the system to change much more rapidly than without modules.

Composition makes large complex systems easy to understand and work on, and vastly increase the maintainability of the overall system.

Impact of changes isolated
The isolation of changes to an effected module is a major benefit of modularity, especially as it supports maintainability. If you’re constantly concerned with your changes affecting parts of the system you’re not actually working on, you’re not able to go as fast as you could if you knew that your changes would only apply to a small area of the whole system – the current module.

This is related to and similar to the support for refactoring that unit tests give – if you’re tests are all green after your refactor, you know you haven’t changed the functionality of your system. In modularity, if you’re isolated from other modules in the system, this goes a step further – you can change the functionality of the module freely without concern about other bits of your system breaking. This is true if you’re doing anything that’s not in the one package you’ve “exported” as the service definition.

Allows physical design to reflect logical design
Modularity also allows developers to build systems where the logical design actually corresponds as needed to the physical design, an often overlooked abstraction. Kirk’s blog post on the topic describes this issue better than I could, so I refer you there.

Supports Refactoring Better
Refactoring is the process of changing the implementation of a system without changing it’s behavior. One of the primary reasons to write unit tests, for instance, is to support and allow refactoring, so we know the system does the same thing when we’re done as when we started.

Modularization supports this ability by making clear the portions of a system that affect other modules, and the portions which do not. When you’re refactoring in a monolithic system, even with IDE support, the scope of your changes is the entire project. When you’re refactoring in a modularized system, if you’re “inside” the module, you have no need to consider scope on the outside, as there isn’t any. And when you’re refactoring the exported interfaces, you know ahead of time your scope is cross module, and can take it into account.

In short, modularization lets you refactor more freely within a module, and lets you be intentional and organized when refactoring between modules.

A colleague recently suggested that modularization would interfere with the ability to see a system as a whole. It’s been my experience that the exact opposite is true – a well structured set of modules can still be considered as a whole when necessary (an aggregator POM is any easy way to set this up in the Maven universe), but it also allows modules to be worked on “safely” in isolation, which a monolithic system does not. This ability to only look at one piece at a time is all the more critical the larger the system becomes.

The need to refactor “across” modules is a warning sign, in my opinion – it generally indicates that the API between modules is changing, and this is a change you want to be more aware of then any amount of changes within modules. If it happens often, it probably indicates that the module boundaries are not yet stabilized, and you have more design work to do.

Scalable Development
A modular system lends itself well to many hands being involved in it’s development and maintenance. It’s not necessary for a team or a pair to understand the whole system well, or even at all – they can still work on a small well-defined and decoupled module, and can do it in parallel with development on other modules.

Helps prevent Design Rot
As discussed in this blog post, design tends to degrade over time, especially in large monolithic systems. Modularity helps to prevent this rot, as pieces that become superseded by better ways of doing things can be replaced individually, and the impact to the overall system managed and isolated. This is akin to removing a brick from the wall and replacing it with a better brick, rather than tearing down the wall.

Solves the classpath hell problem entirely
Classpath hell is the name for the condition where a large number of dependencies are “in scope” at once. It’s not unique to Java, or even the JVM. It is apparently called “DLL hell” on Windows environments, but I’m happy to report I know very little about that. I do know it can happen on platforms other than the JVM, however.

By isolating the influence of dependencies, modularization, especially the way OSGi does it, makes it an explicit process to import and export exactly what packages you wish to and from your modules. What the module uses in it’s private implementation under the hood is no longer relevant to the system at large.

What happens in the bundle stays in the bundle, in other words, with appropriate apologies to Las Vegas.

Management of dependencies is the cure for classpath hell, and modularization is key to this, as described in this excellent article.

Allows smaller pieces of the system to be versioned
A key element in continuous releasability is the careful versioning of both individual modules and their dependency requirements. Modules give you the ability to release pieces of your system, not the whole system, better supporting the idea of continuous releasability and enhancing maintainability.

An article about software maintenance that a colleague recently sent me describes the problem perfectly – and independently versioned modules is a large part of the answer to this problem.

Re-Use
Although re-use it touted often as a major benefit of modularity, I’d categorize it as the least important benefit, not the most. Although the idea of re-using well-designed modules in new projects is indeed very powerful, it’s beyond the technical and design capabilities of many teams, often causing them to reject the need for modularity because they see no reason to build for re-use.

In a way, they’re right: you don’t build for re-use. You build for just the specific case at hand, but when you separate concerns properly and adhere to modularity best practices, you end up with a module that may lend itself to later re-use much more easily. If such a re-use case emerges, well and good, then you can reap the benefit. It is a lot like a framework – you can’t design a good one in isolation from it’s use cases – a framework is an emergent artifact from the re-use of code on many different (although similar) projects. The same is true of re-usable modules: you don’t set out to build a module explicitly to be re-usable.

Allows dependencies to be isolated: e.g. all Weblogic-specific code in one module, or all XMLMill-specific code in one module, increasing ability to refactor safely

Suggested Reading:
Agile Architecture Requires Modularity
Modularity Patterns
Runtime and Development time Modularity

In a later post we’ll look at some of the counter-arguments against modularity, and why I think they’re not especially valid or convincing in my experience.

In Part 1 of our OSGi adventures, we described how to build a nice Ajax-aware Vaadin UI app, and couple it to a generic back-end OSGi service we called the service dispatcher.

Now we’ll take the adventure to the next step, show how to deploy that webapp into our OSGi container and get it up and running, then go through the functionality of the Dispatch Service, and show how it routes requires through to the first of our domain-specific application services.

In our UI, we built a form that allowed the user to enter and save a “Dealer”, with some fields like name, phone number, etc., so the first service we’ll build for the service dispatcher to talk to will be our “dealer” service.

First, though, let’s see how to get our webapp into our OSGi container.

As we’re building our Vaadin app with Maven, we can easily add the small bits of additional configuration to turn our project into an OSGi-friendly WAR file.

OSGi deploys “bundles”, but a bundle is just a jar file (or a war, which is after all just a special kind of jar) with a bit of extra meta-data. The META-INF/MANIFEST.MF file is where the magic happens. We add the following to our POM:

              <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.0</version>
                <configuration>
                  <archive>

                      org.springframework.osgi.web.context.support,\
                       org.springframework.web.servlet,\
                       org.springframework.web.servlet.handler,\
                       org.springframework.web.servlet.mvc,\
                       org.springframework.web.servlet.view,\
                       dwmj.domain,\
                       org.springframework.web.servlet.mvc.annotation,\
                       org.springframework.web.context

                    <manifestEntries>
                      <Bundle-ManifestVersion>2</Bundle-ManifestVersion>
                        <Bundle-SymbolicName>com.point2.Admin</Bundle-SymbolicName>
                        <Bundle-Name>Admin</Bundle-Name>
                        <Bundle-Version>1.0.0</Bundle-Version>
                        <Bundle-Activator>com.point2.ServiceClient</Bundle-Activator>
                        <Import-Package>org.osgi.framework,com.point2.services.dispatch,javax.servlet;version="2.4.0",
javax.servlet.http;version="2.4.0",org.osgi.service.http;version="1.2.0",
org.osgi.util.tracker;version="1.3.2"</Import-Package>
                        <Webapp-Context>admin</Webapp-Context>
                        <Bundle-ClassPath>WEB-INF/classes, WEB-INF/lib/service-dispatch-api-1.0.0.jar, WEB-INF/lib/vaadin-6.1.3.jar</Bundle-ClassPath>
                    </manifestEntries>
                  </archive>
                </configuration>
              </plugin>
     

This bit of magic allows Maven to build us a MANIFEST.MF like this:

Manifest-Version: 1.0
Archiver-Version: Plexus Archiver
Created-By: Apache Maven
Built-By: mnash
Build-Jdk: 1.6.0_15
Extension-Name: admin
Implementation-Title: admin
Implementation-Version: 1.0-SNAPSHOT
Bundle-Activator: com.point2.ServiceClient
Bundle-ClassPath: WEB-INF/classes, WEB-INF/lib/service-dispatch-api-1.
 0.0.jar, WEB-INF/lib/vaadin-6.1.3.jar
Bundle-ManifestVersion: 2
Bundle-Name: Admin
Bundle-SymbolicName: com.point2.Admin
Bundle-Version: 1.0.0
Import-Package: org.osgi.framework,com.point2.services.dispatch,javax.
 servlet;version="2.4.0",javax.servlet.http;version="2.4.0",org.osgi.s
 ervice.http;version="1.2.0",org.osgi.util.tracker;version="1.3.2"
Webapp-Context: cadmin

Now we have our OSGi-friendly .war file, sitting in our target directory. We can then connect to the console of our OSGi container (in this case, ServiceMix), and say:

install war:file:/Users/mnash/experiments/admin/target/admin-1.0.0-SNAPSHOT.war

Our container honors the “Webapp-Context” tag in our manifest, so we can then surf to http://localhost:8181/admin/ to see our application. 8181 is the default port for the Jetty HTTP service in ServiceMix – it can easily be changed to another number as required, of course.

Dispatch Service
Now that we can see how to get the wepapp into ServiceMix, let’s look at the Dispatch Service in detail.

Our Dispatch Service is going to be a regular OSGi bundle, so we have a separate Maven project that produces a .jar file in this case, not a WAR file.

Our MANIFEST.MF needs the following magic for this project:

Bundle-ManifestVersion: 2
Bundle-SymbolicName: com.point2.services.dispatch.DispatchService
Bundle-Name: DispatchService
Bundle-Version: 1.0.0
Bundle-Classpath: .,lib/commons-beanutils-1.8.0.jar,lib/commons-collections-3.2.1.jar
Bundle-Activator: com.point2.services.dispatch.impl.DispatchServicePublisher
Import-Package: org.osgi.framework
Export-Package: com.point2.services.dispatch

You can see we’re again specifying a “Bundle-Activator” class, which, much like the ServiceClient class in our webapp, is called by the OSGi framework when the bundle containing this service is started.

One slight oddity: The dispatch service needs the two jars listed under “Bundle-Classpath:” to do it’s business – because we are building an OSGi bundle, we “embed” these jars in our bundle (which is, yes, another jar) by putting them in the src/main/resources/lib directory. We refer to them in that location in the POM, and Maven automatically includes them in the finished jar, where they’re available when our bundle needs them. The other alternative is to install the dependencies as their own bundles, but that’s a whole ‘nother post

In the case of DispatchService, we’ve got an activator like this:

public class DispatchServicePublisher implements BundleActivator {
    private ServiceRegistration registration;

    public void start(BundleContext context) throws Exception {
        registration = context.registerService(DispatchService.class.getName(),
                new DispatchServiceImpl(), null);
        DispatchServiceImpl.setBundleContext(context);
        System.out.println("Dispatch Service registered");
    }

    public void stop(BundleContext context) throws Exception {
        registration.unregister();
        System.out.println("Dispatch Service unregistered");
    }
}

Which simply takes a reference to the BundleContext on startup and passes it to the DispatchServiceImpl, which implements the DispatchService interface, like so:

public interface DispatchService {
   List<Map<String, Object>> call(String serviceName, String serviceOperation, Map<String, Object> parameters,
                                  String versionPattern, String securityToken) throws Exception;
}

The big JuJu with OSGi is that only this interface is “exposed” from the bundle. No other service can see the innards of our service, unlike Jars on a classpath.

The implementation of the DispatchService is equally straightforward:

public class DispatchServiceImpl implements DispatchService {

    private static BundleContext context;

    public List<Map<String, Object>> call(String serviceName, String serviceOperation, Map<String, Object> parameters, String versionPattern, String securityToken) throws Exception {

        ServiceReference reference = getServiceNamed(serviceName);
        if (reference == null)
            throw new RuntimeException("There is no service with service-name " + serviceName);
        Object service = context.getService(reference);
        Adapter adapter = new Adapter(service);
        List<Map<String, Object>> response = adapter.callList(serviceName, serviceOperation, parameters);
        context.ungetService(reference);
        return response;
    }

    private ServiceReference getServiceNamed(String serviceName) throws Exception {
        ServiceReference[] references = context.getAllServiceReferences(null, null);
        System.out.println("There are " + references.length + " services available");
        for (int i = 0; i < references.length; i++) {
            ServiceReference reference = references[i];
            Object serviceNameValue = reference.getProperty("service-name");
            if (serviceNameValue != null) {
                System.out.println("I see service with name " + serviceNameValue);
                if (serviceNameValue.toString().equalsIgnoreCase(serviceName))
                    return reference;
            }
        }
        throw new RuntimeException("There is no service available with service-name " + serviceName);
    }

    public static void setBundleContext(BundleContext newContext) {
        context = newContext;
    }
}

The dispatch service implementation simply looks through the “published” services in the OSGi context, and looks at the “service-name” property of each service to find the one specified in the call. Assuming it finds an appropriate service, it then uses another class, Adapter, to do the type conversion of the generic Map of parameters to the specific types needed for the service being called, then uses Java reflection to actually make the call and return the response as a List of Maps, again converting the service-specific types to a generic Map in order to pass the response back to the user interface.

Why do we go through those gyrations, instead of just having access to the domain-specific beans in our UI? If we want a full decoupling, and the advantages that come with it, the type-independence of this approach gives us that. It also allows parallel development of the UI and the back-end services without an ever-increasing number of binary dependencies as well – for that matter, with the static data adapter in the Vaadin app allows us to develop on our UI entirely without the back-end services. That’s pretty decoupled.

The Dispatch Service by itself, though, doesn’t give us any functionality. It acts much like a router on a network, simply moving the request from the client to the proper service on the back-end, so let’s build such a service – in this case, a service for handling Dealers.

Dealer Service
The bulk of our application-specific code will be prepared as independent OSGi services, just like this one. In later postings, I’ll describe how to set up dependencies between services (and how to use Spring DM to make doing that kind of thing far simpler).

First, let’s look at our MANIFEST.MF for our new service (again, we can use Maven to produce this for us, but the result is the same):

Bundle-ManifestVersion: 2
Bundle-SymbolicName: com.point2.services.dealer.DealerService
Bundle-Name: DealerService
Bundle-Version: 1.0.0
Bundle-Activator: com.point2.services.dealer.impl.DealerServicePublisher
Import-Package: org.osgi.framework
Export-Package: com.point2.services.dealer

The two key elements above are the Bundle-Activator, a class called DealerServicePublisher, and the Export-Package.

OSGi best practices dictate that the package that is exported should only contain the interface for the class (or classes) you wish to make available from your service. In our case, that comprises a bean class, called “Dealer”, and the interface to our actual service, DealerService.

The Dealer bean is very simple, just your basic property-holding JavaBean:

public class Dealer {
    private String name;
    private String phone;
    private String contactLast;
    private String contactFirst;
    private String address1;
    private String address2;

    public String getContactLast() {
        return contactLast;
    }

    public void setContactLast(String contactLast) {
        this.contactLast = contactLast;
    }

    public String getContactFirst() {
        return contactFirst;
    }
   .....

I’ve ommitted the rest of the getters and setters here for brevity (in case you’re wondering, yes, this is much easier to express in Scala, and yes, OSGi works just fine with Scala…)

The interface for our service is even simpler:

public interface DealerService {
    public void save(Dealer dealer);
    public Dealer findById(int id);
}

Of course, a fully fleshed-out service might in fact have more methods, but again, you get the idea.

I won’t bore you with the actual implementation of the DealerServiceImpl class, but it’s important to note that it’s not in the same package as the Dealer bean and the DealerService implementation. They are the only two classes (well, one class, one interface) in that package, as the entire package is “exported” by OSGi, and therefore visible to other services and clients.

The DealerServicePublisher is the last piece to examine, and it’s pretty straightforward as well:

package com.point2.services.dealer.impl;

import com.point2.services.dealer.DealerService;
import org.osgi.framework.BundleActivator;
import org.osgi.framework.BundleContext;
import org.osgi.framework.ServiceRegistration;

import java.util.Dictionary;
import java.util.Hashtable;

public class DealerServicePublisher implements BundleActivator {
    private ServiceRegistration registration;

    public void start(BundleContext context) throws Exception {
        System.out.println("Registering dealer service");
        Dictionary dictionary = new Hashtable();
        dictionary.put("service-name", "dealer");
        registration = context.registerService(DealerService.class.getName(),
                new DealerServiceImpl(), dictionary);
    }

    public void stop(BundleContext context) throws Exception {
        System.out.println("Unregistering dealer service");
        registration.unregister();
    }
}

Essentially all we do in this activator is “register” our service, making it visible to the OSGi container and other services or clients that need it. We add an extra property via the “Dictionary” object, which allows us to specify arbitrary properties to be associated with our service. Because we want to look up our services from the dispatcher by name, rather than by class or interface name, we use the string “dealer” and associate it with the key “service-name”. If you examine the code for our DispatchService, you’ll see that it uses this property to find services.

Now we can build our new service with a simple “mvn package” command, and install the resulting jar into our OSGi runtime with “install file:/Users/mnash/experiments/dealer/target/dealer-1.0.0.jar” (again, you can do an install directly from the Maven repository, or from a URL, as opposed to a file).

That’s it – our “dealer” service is now available, and our “dispatch” service is fired up and ready to locate it.

If we deploy our Vaadin application in our OSGi container (as described in the previous post), you’ll find that calls to the “call” method of the ServiceClient now return the “real” data from our service.

In the next few posts we’ll examine an even easier way to define new services, and explore the power of OSGi for updating and working with our decoupled services. Then we’ll look at how to hook services together, allowing services to call other services to do their jobs as required.

We’re probably all familiar with the term “technical debt”, meaning the cost of doing things in a non-optimal or non-quality way. While I can go on at length (as my colleagues can attest!) about how this is avoidable by baking in quality, and thus saving time and money at every turn, the fact is that many existing projects have considerable technical debt.

Setting aside for the moment the discussion of telling “good” technical debt from “bad” technical debt, let’s just focus on a projects “bad” technical debt.

If we describe this kind of debt as factors that slow down the ability to change and improve the system, then we see that we are paying the “interest” on this debt every time we touch the codebase or it’s deployed instances.

What I’d like to focus on in this posting is the point at which this technical debt is evaluated to be sufficient that it makes sense to do what we’d normally call a “re-write” of the offending system or subsystem. Basically, when the -ah- mud gets so deep that the hip-waders aren’t helping, it might be time to throw in the towel and start a do-over.

I call this point “technical debt bankruptcy”. Much like a real bankruptcy, it’s an admission that chipping away at the debt isn’t going to work and isn’t worth it – that it’s time to re-group, and in a chapter-11-ish way, fold our tent in a responsible fashion.

Of course, determining if you’re at this point is critical. If you’re not, then you might be throwing out the baby with the bathwater and losing valuable work and former effort for reasons that are not sufficient. Often, political reasons can get us into that kind of situation, where the pressure to do a re-write is not justified. If there are no or few changes to a system, and it’s working sufficiently well as is, then there may be no reason to declare bankruptcy.

If, however, the bill collectors of technical debt are knocking down the virtual door, it’s important to know when to make the right move. As the song says “know when to fold ‘em”.

Part of knowing this is to be able to measure the pace and cost of change, and to be able to estimate, or better, measure, the cost of change if a re-write were done. Let’s say you’ve got a legacy project and a new project. The legacy project is using some old technology and techniques that are painful to work with, and you know they’re causing you to burn more time than they should be. If you also have a newer project (maybe something nice and greenfield) that’s being done with the latest new and shiny tools, Agile techniques, and so forth, you can get a rough idea of what each feature point in the new project is costing you. Now you can compare this to the cost of a feature point in the old project and make a comparison.

If you can look at your backlog of epics and get an idea as to what the future cost of maintenance on the legacy project is going to cost you, then you can take this cost and estimate it instead using the ruler of the new technologies and techniques. The delta is the amount you’ve got available to “spend” on a re-write, essentially.

If the math is right, then declare your technical debt bankruptcy and begin anew!

In some further posts, I’ll explore how to do a well-organized and structured “chapter 11″ on a project, rather than just dropping the ball, including the part that functional tests play in this process, and look at a re-write as a form of highly aggressive refactoring, rather than a whole new project.

Another update in our team’s journey to functional testing nirvana via Specs and Selenium.

We discovered an earth-shattering revelation (not): Internet Explorer Sucks. Unfortunately, it’s the “target” browser for a certain project we’re working on, therefore we’re stuck with it. Being stuck with it means finding a way to make it suck sufficiently less that it will actually run our lovely tests, which Firefox (and several other browsers) digest quite happily.

One way in which IE sucks is that it’s painfully slow – test that take a few seconds on Firefox timeout after several minutes in IE. If you keep cranking out the timeout they might eventually pass, but we want test results in our lifetimes, so we set about finding a way to get our feedback out of IE a bit faster.

It turns out one of the very slow things about IE is asking for an element via an xpath, and we were doing this a lot. We tried reducing the number of places we did this, and indeed that helped a bit, but there were some things on our existing app that were really hard to refer to without xpath. To ease the pain further, we took a slight detour. Instead of asking Selenium for a specific xpath, which then caused Selenium to ask the browser, we fetched the entire output of the browser (via the getHtmlSource method in Selenium).

What we’d like to do instead, where possible, is simply make assertions on the XML of the entire page – so we get it just one time, and so that we don’t depend on the Xpath implementation of the browser to find bits of our page for us.

Unfortunately, we still can’t ask anything xpath-ish of it, as although it bears a close resemblance to XML, it’s not really html, so any parser we could find choked on it.

Enter HtmlCleaner a handy little jar that can digest even the most disguisting HTML and deposit clean shiny XML in it’s place. Now we could take our XML version of what the browser spat out and finally make Xpath calls to it.

Now Scala itself has excellent support for XML (see the whole book here), but it’s support for Xpath is… interesting. Xpath is actually done via a series of methods and functions, meaning although it is very powerful, it’s not the same xpath as you might be used to, or, more importantly in our case, not the same as you could expect a browser to understand. This means we could either re-write all our Xpath queries (and make it harder to use Selenium IDE or Firebox to help us write new ones), or find another way…

Enter Jaxen, a handy Java lib with full Xpath support, without all the weight of other solutions. Of course, as Specs is written in Scala and Scala happily uses any existing Java lib, we could just drop a jar into our “lib” dir and work some magic.

We needed to convert the HTML produced by Selenium into XML, then turn it into a DOM document that Jaxen can process, so we can ask questions in xpath of the resulting Jaxen document.

This little incantation:

 val cleanerAndProps = initCleaner

  private def initCleaner = {
    var cleaner : HtmlCleaner = new HtmlCleaner();
    var props : CleanerProperties = cleaner.getProperties();

    props.setTranslateSpecialEntities(true)
    props.setRecognizeUnicodeChars(true)
    props.setOmitComments(true)
    (cleaner, props)
  }

Returns a tuple of the HtmlCleaner instance and it’s properties, which we can then use for our conversion somewhat like so:

def getHtmlText(element: String) = {
      var node : TagNode = cleanerAndProps._1.clean(selenium.getHtmlSource())
      val domSerializer: DomSerializer = new DomSerializer(cleanerAndProps._2)
      val document: org.w3c.dom.Document = domSerializer.createDOM(node)
      val xpath = new DOMXPath(element)
      xpath.stringValueOf(document)
  }

This method then takes a string (actually an xpath query), “cleans” the HTML from Selenium, then serializes it into a DOM (a regular org.w3c.dom.Document), allowing us to use Jaxen’s DOMXpath object to fish out the string value of whatever our Xpath query returns, allowing us to say things like:

manufacturerSelected = getHtmlText("//table[@class='DataList']//a")

In our tests.

Of course, we can dress this with other methods that get attributes and other things from the DOM tree, but you get the point.

The nice part about all this is that it doesn’t require a new round trip to the browser – we can assert as many things as we want and fish out as many values as we need directly from the resulting page in milliseconds now, instead of several seconds (or worse for IE) before.

So now we can run our tests much faster than before – but we’re writing a lot of tests, so it’s going to get slow again if we can’t scale better, even with the optimization with Jaxen and HtmlCleaner.

So we looked into Selenium Grid. Before, we had our Ant script that ran the tests (whether locally or on TeamCity), fire up a Selenium RC Server before the run, and shut it down again afterwards. This was time consuming, and problematic as we added more suites of tests. We wanted our tests to be able to run all at the same time, but each Selenium server (which does the communicating with the browser), needs to be on a different port for this to happen. Gah – this was getting complicated.

Selenium Grid, however, address this problem and several more at once.

It allows you to start a single “Hub” server, on one port (4444 by default). This hub then redirects load from tests to one of a number of actual Selenium RC servers that you fire up and leave running long-term.

In our case, we started 3 RC servers to talk to FireFox, and one for IE (it’s not a good idea to run more than one IE RC server on a single machine, as IE doesn’t play nice with other IE’s).

Then on other VM’s we can fire up even more RC servers – both IE and FireFox flavors. Selenium Grid gives you a nice control panel to see the servers and which ones are available, and you can add new servers (and remove them) while the grid is up.

The max number of test runs you can have going at once is now the total number of Selenium RC servers in your grid, allowing many short sets of tests to all run at once, giving much faster actual elapsed time before you get feedback when something breaks.

Of course, one of the benefits of this new approach with Selenium Grid is that I can now test if something works via IE on Windows – from my Mac! For that matter, any combination of browsers and operating systems that we’ve got an RC server for can be tested, all at once. Good stuff.

There are still plenty of frustrations in this process – if you kill a test suite before it finishes, for instance, it appears that the Selenium RC server it was talking to remains unavailable forever, and when you restart the hub, you have to restart all of the RC servers… but it’s a step up from where we were.

I recently had the chance to dive into a new project, this one with a rich web interface. In order to create acceptance test around the (large and mostly untested) existing code, we’ve started writing specs acceptance tests.

Once we have our specs written to express what the existing functionality is, we can refactor and work on the codebase in more safety, our tests acting as a “motion detector” to let us know if we’ve broken something, while we write more detailed low-level tests (unit tests) to allow easier refactoring of smaller pieces of the application.

What’s interesting about our latest batch of specs is that they are written to express behaviours as experienced through a web browser – e.g. “when a user goes to this link and clicks this button on the page, he sees something happen”. In order to make this work we’ve paired up specs with Selenium, a well-known web testing framework.

By abstracting out the connection to Selenium into a parent Scala object, we can build a DSL-ish testing language that lets us say things like this:

object AUserChangesLanguages extends BaseSpecification {

  "a public user who visits the site" should beAbleTo {
    "Change their language to French" in {
      open("/")
      select("languageSelect", "value=fr")
      waitForPage
      location must include("/fr/")
    }
    "Change their language to German" in {
      select("languageSelect", "value=de")
      waitForPage
      location must include("/de/")
    }
    "Change their language to Polish" in {
      select("languageSelect", "value=pl")
      waitForPage
      location must include("/pl/")
    }
  }
}

This code simply expresses that as a user selects a language from a drop-down of languages, the page should refresh (via some Javascript on the page) and redirect them to a new URL. The new URL contains the language code, so we can tell we’ve arrived at the right page by the “location must include…” line.

Simple and expressive, these tests can be run with any of your choice of browsers (e.g. Firefox, Safari, or, if you insist, Internet Explorer).

Of course, there’s lots more to testing web pages, and we’re fleshing out our DSL day by day as it needs to express more sophisticated interactions with the application.

We can get elements of the page (via Xpath), make assertions about their values, click on things, type things into fields and submit forms, basically all the operations a user might want to do with a web application.

There are some frustrations, of course. The Xpath implementation on different browsers works a bit differently – well, ok, to be fair, it works on all browsers except Internet Exploder, where it fails in various frustrating ways. We’re working on ways to overcome this that don’t involve having any “if browser == ” kind of logic.

It’s also necessary to start the Selenium RC server before running the specs, but a bit of Ant magic fixes this.

We’ve got these specs running on our TeamCity continuous integration server, using the TeamCity runner supplied with Specs, where we get nicely formatted reports as to what’s pending (e.g. not finished being written yet), what’s passing, and what’s failing.

The specs written with Selenium this way are also a bit slow, as they must actually wait in some cases for the browser (and the underlying app!) to catch up. When run with IE as the browser, they’re more than just a bit slow, in fact…

They are, however, gratifyingly black-box, as they don’t have any connection to the code of the running application at all. For that matter, the application under test can be written in any language at all, and in this case is a combination of J2EE/JSP and some .NET.

There’s a lot of promise in this type of testing, even with it’s occasional frustrations and limitations, and I suspect we’ll be doing a lot more of it.

A colleague the other day need to write a test that checked a string to ensure it was all digits, and happened to be using Scala for it.

An initial thought was to just iterate through the string character by character – but that was laborious and verbose…

A second thought was to just try to convert the string to an integer with toInt, then watch for the exception when it wasn’t, but even this seemed a bit heavy-handed, now that we’re getting used to Scala’s expressiveness.

After a quick experiment we came up with a cleaner solution: Use the built-in abilities to convert the string to a list, then use a partially applied function to filter the list, then take the length of the resulting all-digits string. If this length is not the same as the length of the original string, then there are non-digit characters in the string – otherwise, it’s all digits.

Here’s a transcript of a Scala command-line session trying this out:

scala> "foo".toList.filter(_.isDigit).length
res5: Int = 0

scala> "123".toList.filter(_.isDigit).length
res6: Int = 3

scala> "abc123".toList.filter(_.isDigit).length
res7: Int = 3
</pre>

assertEquals(originalString.length, originalString.toList.filter(_.isDigit).length)