I’ve recently been doing a lot of my own projects in Scala, and, as many other developers have learned, have been finding it harder and harder to “revert” back to Java when Java is mandated for a specific project.

Don’t get me wrong, I still use a lot of other languages, and languages come and go – I’ve learned not to get too attached. Yesterday (literally) I was coding in Ruby, today in Scala, Java and a lick of Python (well, Jython, technically).

Still, I find myself more and more reaching for Scala as my preferred weapon. So I started asking myself why that is, and thought I’d share the answers I’ve come up with so far. I’d be pleased to hear anyone other experiences as well, as I’m pretty sure this is not a complete list. It’s also not in any particular order, just the order I’ve remembered them.

REPL
The REPL for Scala is a huge plus for me. I can quickly experiment with a few lines of code, either with or without the rest of my project on the classpath, and interactively work my way towards a solution. The closest in Java was by writing a throw-away unit test in my IDE, and that’s clumsy by comparison. The REPL is also an excellent learning tool, as anyone who’s used a language with one can attest.

Actors
Actors are a library, not an in-built feature of Scala per se, but both “regular” Actors and alternatives such as Akka provide much of the power of languages like Erlang, while retaining all the other benefits of Scala. It’s difficult to grasp just how much power this can mean to a development project until you’ve worked with them a while, then they’re very hard to live without.

Speed of Evolution
Scala evolves significantly faster than Java (or many other languages, for that matter). I won’t go into the many reasons why, they’ve been explored elsewhere, but this is a huge plus for me compared to the glacial pace of Java language evolution.

Open Source
Scala is actually open source. Not nearly or mostly or kinda or somewhat. Not it’s libraries, or everything except it’s mobile version, or all the bits someone doesn’t think they can make money off of. It’s released under a very open license, rather like a BSD license, with basically no restrictions. Here’s the license itself, so have a read. It won’t take more than a moment.

By contrast, here’s the page discussing the licenses for Java. Bring lunch. And a lawyer, preferably.

JVM and CLR
I don’t use the CLR, and in general despise everything ever produced by Microsoft, but it’s interesting and valuable in some situations to note that Scala doesn’t only run on the JVM, but also the CLR.

Scripting
Ant is an obsolete build system and a worse cross-platform scripting language. Shell is OK, but showing it’s age a bit. Scala runs on every platform I care about, and is every bit as easy as shell to script with, so I’m finding I use it more and more for both quick-and-dirty scripts where I might have used shell before, and for things that will live longer, such as deployment and installation.

Expressiveness
You can say a lot, quickly, with Scala. It’s not just terse and brief, but highly expressive. Examples abound out there, so I won’t waste space trying to prove it here, but in general I find I’m writing perhaps one fifth as many lines to do any given job when compared with Java. It’s at least on par with Ruby.

Not just speed of writing the code, although that’s good. It’s more about speed of reading the code. I can read good Scala and understand it significantly faster than I can wade through the equivalent code in Java.

Community
I can’t deny Java has a huge and helpful community as well, but Scala’s community compared to that of other “newcomer” languages is very diverse and helpful. It’s not a reason to pick Scala over Java, but it might be enough to pick Scala over, say, Python or Ruby, in some cases.

Functional/OO Blend
Scala brings the capabilities of functional languages like Erlang, Haskell, and others to an approachable platform that lets you incorporate these concepts as you’re ready for them, as opposed to an “all or nothing” approach, as a Java programming moving to, say, OCaml might feel. It encourages immutability, but gently and easily, giving you time to learn the power of functional development, while allowing the many advantages of object-orientation to be retained at the same time. No other languages does this quite as well as Scala, in my view.

Options
Had enough of NullPointerException ruining your day? Have a look at Option in Scala. It’s not a silver bullet, but it’s a heck of a lot better than Null

Productivity
The expressiveness of Scala means I can get a lot done, quickly, yet still have high-quality functional and performant code that I can read and maintain when I’m finished.
Scala is approachable and easy to learn, and you don’t have to learn all the fancy stuff it can do at first – you can just write what looks like extremely expressive Java, then get into the cool stuff as you go.

Not Oracle
I have very little faith in Oracle’s stewardship of Java, to be honest, and I don’t think Oracle can sue me for using Scala. IANAL, however, and they may try

One of my primary reasons for this worry is Oracle’s about-face on the issue of open source Java. Do a google on the “Apache Harmony Project” for the gory details.

Traits
Multiple inheritance without the pain, basically. Interfaces and abstract classes with all the warts of both taken away. This is one area where Scala doesn’t just top Java, it blows it out of the water.

Advanced Type System
Scala’s type system starts with the pure object-orientation principle (everything is an object, there are no primitives), and goes much, much further. Case classes alone are worth the price of admission, and they’re only the tip of the iceberg. See High Wizardry in the Land of Scala for a taste of what the type system in Scala can do.

Java compatability
Conversely, Java itself is a reason for my preference for Scala. Instead of Ruby or Haskell or OCaml, if I choose Scala I basically get the entire JVM ecosystem available to me – and that’s a lot. I get OSGi, JDBC, all the drivers for every SQL and NoSQL database you’ve every heard – the works. Except with a language that’s decades ahead of plain Java in other areas.

Tool support
Not compared to Java, which of course has amazing tool support, but tool support for Scala is now up to the point where it’s quite acceptable, even good in some places. I prefer IntelliJ IDEA with the Scala plugin, but good support also exists for Scala now in most of the major IDEs, and more come along every day, it seems.

Collections
Scala’s collection classes are a long way ahead of Java’s, even when you add libraries like Google Collections to Java, and allow you an easy way to go back and forth to existing Java libraries, especially in Scala 2.8.

DSLs
Scala’s flexible syntax makes internal DSLs almost trivial, a huge boon when working with many frameworks. My own little DSL for Vaadin UI development puts a whole new aspect on creating UI’s, I’ve found, and tools like Squeryl and others put this DSL capability to incredible use, things that just aren’t easily possible in Java.

If you’re interested in learning more about Scala, try my resource page. It’s chock-full of links to many excellent Scala resources on the web.

I am pleased to report that the campaign goes well thus far.

We met the enemy at the border of Weblogic, and battle was joined, SVN commits clashing with intractable lengths of binary spaghetti at every turn.

Our original plan was to crush all remote beans entirely, liberating the entire application from this particular scourge. Much progress has been made towards this goal, although the enemy is powerful and resourceful.

We can confirm that at least five of our opponents fell early in the glorious battle, with minimal losses on our side, despite the heavy resistance. At several points in the battle the WTF/minute meter was rising to dangerous levels. One particularly brave refactor was rolled back in a blaze of glory, still muttering “stay in target directory… stay in target directory”…

Our initial attempt at penetration of the enemy’s lines was at the JMX pass. Despite valiant attempts by our brave tests, we were eventually defeated in this attack by proprietary “extensions” to what should have been well-accepted standards. The enemy knows no shame, and will stoop to perverting even innocent RMI for it’s nefarious purposes. In fact, they dredged up what should have been long-forgotten and discarded protocols to throw us off the track, even resorting to JMX over IIOP, with proprietary client libraries, which was too much for us – we could not withstand that level of firepower. We had to find another way.

We were repelled by the evil machinations of the enemy’s classpath conflicts at the battle of the EAR, but were able to outflank them and retreat to a previous revision number with minimal losses. We then regouped and attacked again in the region of the remote stateless session bean, and only one of the enemy escaped our wrath. We will be watching this one remaining combatant carefully, and at his first slipup he is sure to be doomed as well. We have assigned our best integration test to watch his every move.

Despite attempts to deceive our troops with obscure and fiendish JNDI naming practices, we were able to ensnare many of the enemy in our powerful web of functional tests, where they can do no further harm.

At least four entire modules of the enemy’s forces did not survive the encounter!

We have won this conflict for now, yet many challenges still lie ahead in future battles. The enemy used many clever tactics this time, but we have not yet encountered their ultimate weapon: the greatly-feared container-managed entity bean! Weapons of mass deprecation of this nature must not be allowed to continue to intimidate future generations. Massive as these weapons are, they are slow and no match for our lightweight DDD techniques – we will lure them into shallow memory and watch them go aground, tangled in their own distributed transactions.

For now, though, our troops have withdrawn to the safety of our own home OSGi modules, where order, structure and modularity reign supreme.

Stay tuned for future reports….

Often, in the software development game, we’re asked to add a feature to an existing system. Sometimes this system has been around for a long while, as there’s no notion of “planned obsolescence” in much of software.

I recently encountered a situation that is highly symbolic of the problem, if not an actual software issue, per se.

At an office I’ve worked in, they have very nice washrooms (yes, this is related to software – bear with me here). They’re modern, with nice clean stalls and such, and feature toilets with the “auto-flush” feature you’ve no doubt seen elsewhere.

Once you’ve finished your transaction, you just finish up and leave, and the sensor detects you leaving and auto-flushes the, ah, buffer, for you. Excellent, now you can be free of having to remember to do that yourself, and assured that it’s already been done when you enter the stall. Kind of like auto-commit on a JDBC connection. Well, kind of.

Anyway, let’s talk about feature conflict.

A new feature was recently added to this fine setup: paper seat liners. You’ve probably also seen these – they’re the nice little rings of paper in a dispenser, usually on a wall, that allow you to isolate your back-end interface from the implementation details of the toilet seat.

Also very fine idea. In the case of this particular washroom the dispenser for these is attached to the wall behind and above the toilet itself.

Taken independently, each of these features (auto flush and seat liner) seem like perfectly good features, and their implementations are quite reasonable. In functional testing, each performs exactly as it should.

The difficulty comes in integration testing, when you try to use the system as a whole.

Let’s take a use-case and walk through the scenario: The – ah – user enters the system. Standing in front of the toilet, he (definitely he in this particular case) decides to avail himself of the seat liner, and removes an instance from the dispenser. No problem. At this stage, the auto-flush sensor has already detected the presence in it’s scope, and does nothing, as it doesn’t trigger until you leave the area.

Now an implementation detail: the way the paper liner works is that the spot in the middle of the paper is cut out, and drops down to touch the water, while remaining attached to the remainder of the liner. The idea here is that when the water leaves during the flush, the liner will be drawn down into the bowl with it, and appropriately de-allocated. As we’ve said, previous functional tests verify this works just fine.

Now let’s assume the user has correctly deployed the liner, and wishes to transact his business. Well, if you were facing the toilet when you came in, you probably don’t want to be anymore at this point, so you turn around. Here’s where the feature conflict rears it’s ugly head: the flush sensor, noting the movement, often picks this moment to decide it’s time to flush.

The liner, having deployed correctly, departs the scene at this point, just as the user completes their interface. Unfortunately, that is the moment at which the liner is supposed to be there. You see (or, more correctly in the case of the user, feel) the problem.

So, which feature is broken? Taken one at a time, neither is broken – they both do exactly what they’re supposed to do. The failure is at the point of the wholistic view of the overall system, and neatly demonstrates the fact that it’s impossible to consider features or potential features in isolation, in any system. You must consider scenarios of users actual interactions with the system, and how the new feature will affect those scenarios.

Nobody said it’s easy, but it’s important.

I recently had occasion to use a trick with test execution that is simple and very helpful, but that I forget about frequently, so I thought I’d pass it along here.

We use Maven to execute our unit (and sometimes functional, but that’s another story) tests for a suite of applications my team works on. Some of our apps have quite a few tests, and even though each unit test is very fast (as good unit tests ought to be), running that many of them still takes a bit of time. We’re also burdened with a very slow CI server, unfortunately, which aggravates that situation.

Unit tests, by definition, do not rely on the operation of any other test – they are self-contained, stateless and repeatable. This means they’re ideally suited to running in parallel, and fortunately, that’s very easy to do with Maven.

Maven uses the Surefire Plugin to do it’s test-running, and it’s configurable to run tests in parallel. Our project uses a parent pom (different from an aggregator pom) to specify common configuration, so our pom files stay lean. In our parent pom, we just add the following:

  org.apache.maven.plugins
  maven-surefire-plugin
  2.5
  
methods
20
  

And we’ve told Maven to run our tests in a maximum of 20 threads, and to make each method run in parallel. (There are other options here for only running classes in parallel, etc, which might be helpful for functional tests).

Just doing that took a minute off the time of a couple of our builds, and a bit off the time of even some of the smaller ones. Of course, it’s even more effective if you’re running on a multicore machine, but even the feeble VM running our CI was faster with it there.

Enjoy!

One of the ways to ensure quality in a software project is to find a set of coding standards that your team can agree on then put automated checks in place to ensure they are adhered to.

In this post I’d like to take a very small example of such a stanard, and show how you can use several different tools to help ensure and measure compliance.

The Example
The simple rule we wil use is the use of curl braces on if statements and similar blocks. Our team reached a concensus that we would always prefer this form

if (condition) {
   block
}

over this

if (condition)
    block

even when there is only one line in the block, where the braces are optional. I’m in the habit of not doing this, so I kept doing it the wrong way, making code just that little bit harder for the rest of the team to read and maintain.

We have an existing codebase of hundreds of thousands of lines, some of which adhere to this rule, some of which doesn’t.

First, I wanted to measure adherence to the rule, so I chose the Sonar quality assurance tool.

Sonar
Sonar installation is straightforward: download the distribution, unzip it and run one script. Sonar runs as a web application, so you can immediate go to your browser and see the initial page, like so:

I’m using Sonar (for now) in it’s default mode with a bundled Derby database. If I were going to use it more heavily, I could install a separate MySQL database, and run Sonar itself inside a container of my choice (it comes bundled with a basic web container so you can get going right away).

For my machine, I ran

bin/macosx-universal-64/sonar.sh console

“Console” runs Sonar without backgrounding the task. Normally you’d use “start”, which works just like a Unix service, and is suitable for running from an init.d-type script immediately. The first time around, Sonar takes a few moments to fire up, as it creates all it’s required database tables and populates them with an initial set of data to get your started.

Sonar can measure many hundreds of different metrics, but for the purposes of this simple experiment I only want the one, so I create a custom profile, called “team”.

I set this profile to be the default, then click to edit it.

Now I can search for the braces rule in checkstyle by setting filter criteria:

You can click the rule name to see a description, and to set any optional parameters on the rule.

I activate the rule, and leave it at the default “Minor” priority.

If I now return to my profile, I have a link to access the generated checkstyle.xml file, the configuration file that is used by checkstyle to configure its rules. This will be important later on.

Now I can go to my project and analyze it by running “mvn clean install sonar:sonar”. With the default configuration, this works only when the build is taking place on the same box that Sonar is running, but it’s simple to set up a settings.xml file to tell the Sonar plugin where to find the Sonar instance, so you can have a single shared Sonar instance for the whole team. This is covered nicely in the Sonar documentation.

One minor hiccup: The released version of the Sonar plugin for Maven doesn’t work with Maven version 3 (still in Beta), which I was using. You can build the source version if you need this support, or I’m guessing it’ll be released about the time Maven 3 might be.

Now that I’ve analyzed my project, I can go back into Sonar and see how well I adhered to my rule:

As I see I have a violation, I can drill down and see the violation in context right in the code:

This is handy for checking the state of a rule against an existing codebase, but I’d like to have more immediate feedback. No problem, I can integrate Sonar into both my IDE and my build process very easily.

IntelliJ IDEA: Immediate Feedback

There’s also a new plugin for IntelliJ IDEA for Sonar. To install, just go to the “Available” tab in your plugin configuration within IntelliJ, right-click and select Download and Install. You’ll have to restart IntelliJ to activate the plugin.

Now you can find the Sonar entry in configuration, and set up a URL to point to your Sonar installation, along with a username and password to log in to Sonar. The default is “admin/admin”.

Having done that, I can now see violations directly in my code, as I open each file, like so:

In the above, I’m hovering over the little “price tag” annotation in the left gutter to see the message. You can also see the usual IntelliJ “yellow bar” on the right side of the screen, which shows the same message.

This lets me know of a rule violation before I even attempt to build, so when I forget my braces, I don’t have to wait to be chastized, IntelliJ can do it for me immediately

Maven: Ensuring Conformance

There is a Maven Checkstyle Plugin that allows us to make a run of checkstyle part of the build process of our project. I configured this plugin in my POM like so:

    4.0.0
    com.jglobal
    experiment
jar
    Experiment
    1.0.0-SNAPSHOT

UTF-8
    

    

        
            junit
            junit
            4.5
            test
        

    

    


                maven-compiler-plugin
                
                    1.6
                    1.6
                
            

                org.apache.maven.plugins
                maven-checkstyle-plugin
                2.5
                
                    
                        org.codehaus.plexus
                        plexus-resources
                        1.0-alpha-7
                    
                
                
                    http://localhost:9000/rules_configuration/export/java/team/checkstyle.xml
                    
                    true
                    true
                
            

                org.codehaus.sonar
                sonar-maven-plugin
                1.8M1
            
        
    


            codehaus
            Maven Plugin Repository
            http://repository.codehaus.org/
            default
            
                false
            
            
                never
            
        
    

As you can see, instead of providing a file system based location for the configuration file for Checkstyle, I just pointed the Checkstyle plugin at my Sonar-generated checkstyle.xml, this way my IDE, Sonar and my build stay in perfect sync, and I have a nice GUI for adjusting my checkstyle rules, instead of having to hand-edit XML.

Now when I run my build, Maven will analyze my project against the checkstyle rules, and report any violations. With code as we’ve seen above, it of course reports a violation, but the build still succeeds.

This is fine, but now that I’ve verified we are adhering to this rule, I’d like to actually fail the build if we slip back. Also easily done – in Sonar, I just change the level of violation for my one and only rule to “Critical” in my profile. Now when I run my Maven build, sure enough, the build fails, noting the fact that I’m not adhering to a Critical rule.

As my codebase gets cleaner and cleaner, I can up the priority of the rules I really care about until there are no more violations – then Maven and Sonar can guarantee it stays that way. Of course, in a multimodule build (e.g. with an aggregator POM), I might want to create a parent POM, and specify all this configuration there. (It’s generally not a good idea to have the aggregator and the parent in one pom, use two).

This is of course a trivial example of Sonar, IntelliJ and Maven working together. Sonar can apply over 700 rules right out of the box, and more can be added via plugins easily.

What I find good is that I can synchronize all my tools off one common and agreed-upon rulebase, without having to tweak XML files all over the place to gradually increase my quality standards over time.

In a project I was tinkering with lately I found the complexity getting to the point where I could use component dependency injection. There are in fact a number of techniques to do this in “plain” Scala, but they generally seem more painful to me than just dropping in a lightweight DI framework, so I went with Guice.

As it happens, Guice works with Scala very nicely, and is very unobtrusive. There’s a few minor things I haven’t quite figured out yet, but I’d like to share what I’ve discovered so far.

After adding this section to my project POM:

 
            com.google.inject
            guice
            2.0
        

I had all I needed of Guice embedded in my app. Now I can scratch up a series of tests that exercise all of the different ways use Guice from Scala. Let’s start with the simplest, and work our way up:

 @Test
  def basicDependencyInjection {
    class ScalaModule extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).to(classOf[SomeService])
      }
    }

    class ScalaModule2 extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).to(classOf[SomeOtherService])
      }
    }

    val injector = Guice.createInjector(new ScalaModule)
    val component = injector.getInstance(classOf[MyComponent])
    assertEquals("someService", component.callTheService)

    val injector2 = Guice.createInjector(new ScalaModule2)
    val component2 = injector2.getInstance(classOf[MyComponent])
    assertEquals("someOtherService", component2.callTheService)
  }

So let’s explore the test a bit. I’m using Scala to build a JUnit test, as shown by the annotation on line 1. One of the advantages of this is that it’s easy to use Scala’s namespace capabilities to build test classes that are only visible within the test itself, which is what I’m doing starting line 3. I create a class call ScalaModule with extends Guice’s base class AbstractModule. I override one method called “configure”, which sets up the bindings of interfaces to implementations for this Guice module. (This is all covered in detail in Guice’s doc).

On line 6 I bind the interface “AService” to an implementation of AService called “SomeService”. Here’s what they look like (these classes are defined later in the file – I don’t embed them inside the test class as Guice didn’t let me).

Here’s the trait for the service:

trait AService {
  def service: String
}

And a simple implementation:

class SomeService extends AService {
  def service(): String = "someService"
}

Now I have a component that uses that service, called MyComponent. It looks like this:

class MyComponent @Inject()(val service: AService) {
  def callTheService(): String = service.service
}

Note the annotation on the first line. This is how you put an annotation on the constructor in Scala, Annoyingly, the () is apparently required. This Inject annotation indicates that when this component is requested from Guice, it should have the appropriate implementation of Service injected.

So now to return to our test, on line 10 you’ll see we are building a second module, this time binding the AService interface (trait, actually) to the SomeOtherService implementation, which looks like this:

class SomeOtherService extends AService {
  def service(): String = "someOtherService"
}

Then we get into the meat of the test in line 17. First, we ask Guice for our injector, the “god object” for Guice, from which you get all the other object you’ll need. In the first case, we use our ScalaModule class. Then we ask this injector for our component, and it constructs the component for us, injecting the dependency as necessary. In this situation, we have not told Guice that MyComponent is a singleton (we’ll deal with that later – in several different ways), so it builds a new one for us every time.

On line 18, we make the assertion that proves that our MyComponent instance has in fact been built with the “SomeService” instance, as opposed to any other instance, by verifying our string that SomeService responds with.

Then we do the whole thing again, except this time asking for the ScalaModule2. ScalaModule2 is where we have the SomeOtherService implementation of AService wired up, so when we assert on line 23, we now see the “someOtherService” string returned instead, which means that we’re properly wired, as expected.

Now let’s give Guice a little harder work, exploring some scenarios we’ll see in more sophisticated applications.

In many situations, we’ll want to avoid creating a new instance of our service object every time we use it. There are a couple of ways to do this with Guice.

@Test
  def instanceBindingExample {
    // if you want to bind to a specific instance of a class...
    class InstanceExampleModule extends AbstractModule {
      @Override
      protected def configure() {
        val instance1 = new InstanceService("instance1")
        bind(classOf[AService]).toInstance(instance1)
      }
    }
    val injector = Guice.createInjector(new InstanceExampleModule)
    val service = injector.getInstance(classOf[AService])
    assertEquals("instance1", service.service)
  }

In the code above, we’re configuring our module, called InstanceExampleModule, with a binding that specifies our AService trait uses a specific instance of the class called InstanceService.

Here’s InstanceService:

class InstanceService(val value: String) extends AService {
  def service(): String = value
}

Back in our test, line 13 ensures that we’re actually talking to the right service, as we fed the service it’s return string (“instance1″) when it was created in the module.

Ok, so now we can inject services, and specify an instance of a service instead of a newly-created one every time. That’s a good start, but we want to do more.

What if we have to do some more work to instantiate our singleton instance? E.g. what happens if it takes a whole method to create the service?

Guice has that covered as well, like so:

Here you see a new module being built, but this time we don’t actually say anything in the “configure” method. Instead, we annotate another method in the module with the @Provides annotation – this means to Guice “whenever you need a thing of the type this method returns, call this method”. Then we verify that that’s in fact what’s happening by creating the injector and asking for the service.
Ok, so now we can create instances of services however we want, but we’ve still only created a single instance of a given trait. What if we’ve got a whole whack of services that implement our interface? Ok, that’s also easy:
@Test
  def bindingWithNameAnnotationExample {
    // what if I have two implementations of the same interface, how do I say which one I want?
    class ScalaModule extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).annotatedWith(Names.named("foo")).to(classOf[SomeServiceNamedFoo])
        bind(classOf[AService]).annotatedWith(Names.named("bar")).to(classOf[SomeServiceNamedBar])
      }
    }

    val injector = Guice.createInjector(new ScalaModule)
    val barComponent = injector.getInstance(classOf[MyBarComponent])
    val fooComponent = injector.getInstance(classOf[MyFooComponent])
    assertEquals("bar", barComponent.callTheService)
    assertEquals("foo", fooComponent.callTheService)
  }

Now we’ve got a couple of services to declare:
class SomeServiceNamedFoo extends AService {
  def service(): String = "foo"
}
class SomeServiceNamedBar extends AService {
  def service(): String = "bar"
}

The services themselves both implement our trait, of course, but now we have two different components, one which needs the Foo service, other other one needs the Bar:
class SomeServiceNamedFoo extends AService {
  def service(): String = "foo"
}
class SomeServiceNamedBar extends AService {
  def service(): String = "bar"
}

In our test, you can see we ask the injector for the two components. What’s very interesting to me is that the component themselves are not listed in the module. Why that’s important is that unlike Spring (to pick an example many people know), you don’t need any configuration provided at all for stuff that Guice can work out on it’s own. This is a good thing.
Anyway, you can see by our asserts that in this situation the Foo service gets the Foo implementation of the AService (SomeServiceNamedFoo), and the Bar service gets SomeServiceNamedBar instead.
Now that we can handle multiple implementations, let’s go back and consider the singleton vs. new instance issue from another angle: Scala has no concept of the idea of “static” classes, but it does have companion objects, which are, in all cases I’ve seen, better.
If we declare a service like so:
object SingletonService extends AService {
  def service(): String = "singleton"
}

and another (non-singleton) implementation, like so:
class NonSingletonService extends AService {
  def service(): String = "nonsingleton"
}

Now we can write a test that tries this out like so:
  @Test
  def singletonOrPrototype {
    class ScalaModuleSingleton extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).toInstance(SingletonService)
      }
    }

    class ScalaModuleNonSingleton extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).to(classOf[NonSingletonService])
      }
    }

    val singletonInjector = Guice.createInjector(new ScalaModuleSingleton)
    val singleton = singletonInjector.getInstance(classOf[AService])
    assertEquals("singleton", singleton.service)
    val secondInstance = singletonInjector.getInstance(classOf[AService])
    assertSame(singleton, secondInstance)

    val nonSingletonInjector = Guice.createInjector(new ScalaModuleNonSingleton)
    val nonsingleton = nonSingletonInjector.getInstance(classOf[AService])
    assertEquals("nonsingleton", nonsingleton.service)
    val secondNonSingletonInstance = nonSingletonInjector.getInstance(classOf[AService])
    assertNotSame(nonsingleton, secondNonSingletonInstance)
  }

The money here comes at the line that binds the classOf[AService] to a singleton – in this case, the SingletonService. This isn’t a class, so we don’t use to(classOf[SingletonService]), we say toInstance(SingletonService) instead.
As you can see from the test, this works very nicely, and gives us a much more Scala-canonical way to do singleton.
However, there are times when we want to declare a regular Scala class, and have Guice only instantiate one of em for us, and handle it as a singleton that way (some examples might be when we’re using a constructor to inject other dependencies – although we can use field injection for this, which we’ll look at later).
We can annotate our class explicitly to tell Guice it’s a singleton like so:
@com.google.inject.Singleton class SingletonClassService extends AService {
  def service(): String = "singletonClass"
}

Now we can write a test like so to prove that we get the same instance of our service, even if we ask the injector for it multiples times:
@Test
  def singletonClassExample {
    class ScalaModuleSingleton extends AbstractModule {
      @Override
      protected def configure() {
        bind(classOf[AService]).to(classOf[SingletonClassService])
      }
    }

    val singletonInjector = Guice.createInjector(new ScalaModuleSingleton)
    val singleton = singletonInjector.getInstance(classOf[AService])
    assertEquals("singletonClass", singleton.service)
    val secondInstance = singletonInjector.getInstance(classOf[AService])
    assertSame(singleton, secondInstance)
  }

The assertSame in JUnit asserts that we have the exact same instance of an object, not that they’re only equal.
One last scenario: What is the service isn’t the singleton, but the component is? E.g. what if we have an object instead of class that needs a service injected into it? Turns out Guice can handle that nicely as well. The only extra work we need is to define a trait for the binding of our component, like so, with the service injected into the object:
trait SingletonComponentInterface {
  def callTheService:String
}
object SingletonComponent extends SingletonComponentInterface {
  @Inject val service:AService = null
  def callTheService(): String = service.service
}

Now we can write a test to make sure this is doing what we want:
@Test
  def injectIntoSingletonExample {
    // if you want to inject into an object instead of a class
    class InstanceExampleModule extends AbstractModule {
      @Override
      protected def configure() {
        val instance1 = new InstanceService("instance1")
        bind(classOf[AService]).toInstance(instance1)
        bind(classOf[SingletonComponentInterface]).toInstance(SingletonComponent)
      }
    }
    val injector = Guice.createInjector(new InstanceExampleModule)
    val component = injector.getInstance(classOf[SingletonComponentInterface])
    assertEquals("instance1", component.callTheService)
  }

That’s it for my experiments so far. Here’s my conclusions, please comment with your experiences:
The Good, The Bad, and the Ugly
The Good

There was a lot I liked about Guice in Scala. First, no XML was injured in the making of these examples – everything was code, and fully type-safe (I mentioned it was Scala, right?) I always find myself writing a “smoke test” for my Spring apps to verify I haven’t made a typo in my “beans.xml” file – here I didn’t need to. Yes, I know you can do XML-free Spring as well, but it’s the norm with Guice, and it feels a lot lighter weight.
I don’t need to configure anything for classes that just get injected. This means that the configuration I do need to write is very short, and because it’s a class, it’s extensible. In my “real” project, for instance, I was able to write a module that created services that use static lists for their data (instead of reading and writing to a real database, for instance), then extend that class with an OSGi-aware version that used the “real” services from other modules. No repeated work, no wiring, no config – it just worked.
Auto-handling of singletons, both with companion objects and classes, as I prefer: It was nice to be able to write Scala in a very Scala-like way, and just have the DI framework slot right in whichever way I needed it to, as described above.
The Bad

I have to annotate my classes with Guice-specific annotations, although there is a JSR on the way to standardize these, which makes it a bit less objectionable. What happens if I need to inject to a class I don’t have source for (e.g. something in an existing Java library that I’m calling? I also haven’t tried this in a Servlet environment, although I understand there’s a Guice extension for that, so I don’t expect any trouble there.
The Ugly

I have to (it seems) use the @Inject() format, not just @Inject, and I have to insert it in what looks like an odd place in the declaration. Not a big deal, and I suspect I’ll get used to it.
            

I’ve had an opportunity recently to work on a product that needed an RIA web interface, and I chose my recent favorite tool for this, Vaadin. The services for this project needed to be highly scalable, and lent themselves well to functional techniques, so I selected Scala as my language of choice. I build my projects with Maven, for reasons I won’t go into right now, and I do much of my JVM-language work in Intellij’s excellent IDEA IDE.

Given these tools, I found a way to facilitate very rapid development of web UI’s, and I thought I’d pass it along.

Another technique I use, which I’ll expound on later, is creating “dummy” implementations of all of my backing services for my application. The “real” implementations are written as OSGi services, in separate modules from my UI. The UI is packaged as a war, but is also OSGi aware, with a bundle activator. This activator only gets called if the war is deployed into an OSGi container, and not otherwise. This allows the app to select which implementation of the services it uses – the “dummy” ones when it’s deployed outside of OSGi, and the “real” ones when they’re available.

This means I can use the handy Maven jetty plugin to quickly spin up my application and test it on my local workstation, without needing all of the dependencies (like a data store and such) of my real services. That’s good, in that I can get my “cycle time” down to a few seconds, where “cycle time” is the time between making a change and actually being able to test it in my browser.

We can do better, though.

I’m using Scala as my language of choice for building the UI as well, as it works just fine with Vaadin (and with everything else in the JVM ecosystem, for that matter, which is why I didn’t choose a non-JVM language – but that’s yet another rant).

I compile my Scala with the Maven scala plugin – here’s where the next handy bit comes into play. Turns out the Scala plugin has a goal called “cc”, for continuous compilation. Using this, I can fire up Maven with a “mvn scala:cc” command, and leave it running. Then I also use the “mvn jetty:run” command in another window to fire up the web application, and leave it running as well.

Here’s my configuration for the Scala plugin:

                org.scala-tools
                maven-scala-plugin
                2.9.1
                
                    
                        
                            compile
                            testCompile
                        
                    
                
                
                    ${scala.version}
                    
                        -target:jvm-1.5
                    
                
            

And for Jetty:

                org.mortbay.jetty
                maven-jetty-plugin
                6.1.9
                
                    10
                    src/main/webapp
                    jetty.xml
                    
                        
                            file
                            realm.properties
                        
                    
                    stop
                    8889
                
            

Now I go back to my IntelliJ and start coding. Every time IntelliJ saves (which it does automatically unless you tell it not to), the Scala plugin compiles the files. This generates a new .class file, which the Jetty plugin (well, technically, Jetty itself) detects, and in response, reloads the running classes for the web application.

Net effect is that I can make my change and by the time I switch back to the browser, my new code is running. I test my change, emit the appropriate profanity, and go back to editing, all within a second or two.

This has profound effects on how you develop a UI, which every dynamic language aficionado knows (e.g. like Ruby/Rails or Python/Django). You don’t hesitate to experiment, and you get to see the visual effect of your changes right away. The good news is that I get to do this with my language of choice, and with all the power of the JVM ecosystem to support it.

The technique is not perfect – I’ve found that if you edit some resources or webapp files (images and such), it’s possible the Jetty plugin doesn’t “see” the change. Of course, two things help with that considerably: first, it’s lightning-fast to just Control-C the jetty plugin task and re-launch it, and second a Vaadin app generally doesn’t use many resources, unlike JSP or many other frameworks that make extensive use of templates.

Once in a while I’ve found the scala:cc task will report that it’s lost it’s connection to the “fsc” (fast scala compiler) background process – again, quickly control-c-ing the task and starting it again solves the problem every time.

Overall, I can crank UI out pretty darn quick with this method, and given that I can TDD even my UI code using Vaadin, I find the overall combination very effective and efficient.

I’ve been sold for some time on the benefits of highly modular software development (and deployment, for that matter, almost equally importantly).

However, I’m aware that not everyone is on the same page, or has the same definition of “modularity” in the first place, so I thought I’d collect and enumerate the things that have convinced me, as best I can.

I’ve been in the software business for a very long time, and sometimes I have a preference or leaning towards a certain technique or technology. Sometimes that preference has clear causes behind it, but sometimes not. Modularity is one of the latter, where I know it’s not only good, but critically important, but it’s hard for me to immediately say why or how I know that in a succinct way.

So, I apologize for the length of this explanation, as I’ve not yet taken the time to make it shorter

Modularity Defined
First things first: What do I mean by modularity?

In short, I mean a system comprised of relatively small self-contained units. The idea of modularity has caught on much more completely in many other disciplines, but far less so in software. In manufacturing, especially mechanical and electronic, in business, especially finance, the idea of modularity is well established. Modularity is used to increase and maintain quality, and reduce cost while increasing output. In the automotive industry, for example, extensive use of standardization and modularity allows us all to afford a car, as opposed to just the wealthy or the mechanically inclined.

In the software world, the use of modularity is still woefully small, however. When it is employed, though, some spectacular successes have arisen. One of the oldest examples is Unix, arguably the most successful operating system ever. Since 1969, the “do one thing, do it well, and play well with others” modular design of Unix has been the basis of it’s success and flexibility, allowing it to span everything from embedded devices to mainframes.

The Java Virtual Machine (JVM) is another realm where modularity is on the rise, and starting to have an impact. Many different approaches are available, and some of them work with any of the large number of languages available on this platform.

Increase overall success rate of projects
In the many software projects I’ve been involved with over the years, I’ve noticed a few trends. Smaller, well-contained projects have a higher success rate than larger and more poorly contained projects. (By contained, I mean the number of places and ways the project interacts with other projects is small). I’m sure there are a host of reasons for this, but part of it is simple comprehension – small projects are generally easier to understand than big one, so the number of times things get done wrong is reduced.

A system comprised of well-isolated modules tends to be more like a set of small projects that happen to work together, so I find this reason alone a plus for modularity.

But this hides much of the detailed reasoning behind why modularization works, so let’s dig a bit deeper…

Makes the API between components explicit, not hidden
Sometimes I heard the argument that modularity is complex, that it actually increases the difficulty of working with a system. I find that doesn’t hold up under examination: when modularity seems complex, it’s probably because it’s being done wrong, or being retrofitted to a system that was not designed with modularity in mind. What you’re really seeing in this situation is that the attempt to modularize is exposing the complexity that was already there.

When a system is properly designed in a modular fashion, the APIs between modules are small and well-defined, with a limited number of cross-module dependencies.

Reduce unnecessary complexity
By reducing a large complex system into a series of small and better understood modules, we’ve reduced the complexity we have to deal with at any one time. This means that the overall large system only has to deal with how the exposed APIs of our modules will be fitted together, treating the modules themselves as “black boxes” that perform specific functions, without worrying about how they perform it.

In both cases, we’ve reduced the complexity we need to deal with.

I specify here “unnecessary” complexity, as of course the complexity of the issue itself doesn’t get any simpler just because we modularize it – but if we don’t have to deal with a lot of tangled plumbing, we can concentrate on the real issue.

Increase testability
A module that performs a single function is much easier to test exhaustively than an entire monolithic system. The number of interdependence is limited, meaning we can do more of our testing at the unit and orchestration level than at the functional or integration level, meaning our feedback time is improved and our coverage likely increased.

This means our development velocity can increase, as the impact of changes can be evaluated more easily and more quickly.

Support Composition
When you have a number of smaller building blocks, it’s relatively easy to reshuffle those blocks to change the behaviour and functionality of your overall system. This is making use of composition – connecting blocks together to do more than they can on their own, and its a pattern that is supported well in a modular environment.

Although composition is often thought of in the context of re-usable modules, when building a new system, it’s also valuable in modules that are only used within one system, as it allows the large-grained behavior of the system to change much more rapidly than without modules.

Composition makes large complex systems easy to understand and work on, and vastly increase the maintainability of the overall system.

Impact of changes isolated
The isolation of changes to an effected module is a major benefit of modularity, especially as it supports maintainability. If you’re constantly concerned with your changes affecting parts of the system you’re not actually working on, you’re not able to go as fast as you could if you knew that your changes would only apply to a small area of the whole system – the current module.

This is related to and similar to the support for refactoring that unit tests give – if you’re tests are all green after your refactor, you know you haven’t changed the functionality of your system. In modularity, if you’re isolated from other modules in the system, this goes a step further – you can change the functionality of the module freely without concern about other bits of your system breaking. This is true if you’re doing anything that’s not in the one package you’ve “exported” as the service definition.

Allows physical design to reflect logical design
Modularity also allows developers to build systems where the logical design actually corresponds as needed to the physical design, an often overlooked abstraction. Kirk’s blog post on the topic describes this issue better than I could, so I refer you there.

Supports Refactoring Better
Refactoring is the process of changing the implementation of a system without changing it’s behavior. One of the primary reasons to write unit tests, for instance, is to support and allow refactoring, so we know the system does the same thing when we’re done as when we started.

Modularization supports this ability by making clear the portions of a system that affect other modules, and the portions which do not. When you’re refactoring in a monolithic system, even with IDE support, the scope of your changes is the entire project. When you’re refactoring in a modularized system, if you’re “inside” the module, you have no need to consider scope on the outside, as there isn’t any. And when you’re refactoring the exported interfaces, you know ahead of time your scope is cross module, and can take it into account.

In short, modularization lets you refactor more freely within a module, and lets you be intentional and organized when refactoring between modules.

A colleague recently suggested that modularization would interfere with the ability to see a system as a whole. It’s been my experience that the exact opposite is true – a well structured set of modules can still be considered as a whole when necessary (an aggregator POM is any easy way to set this up in the Maven universe), but it also allows modules to be worked on “safely” in isolation, which a monolithic system does not. This ability to only look at one piece at a time is all the more critical the larger the system becomes.

The need to refactor “across” modules is a warning sign, in my opinion – it generally indicates that the API between modules is changing, and this is a change you want to be more aware of then any amount of changes within modules. If it happens often, it probably indicates that the module boundaries are not yet stabilized, and you have more design work to do.

Scalable Development
A modular system lends itself well to many hands being involved in it’s development and maintenance. It’s not necessary for a team or a pair to understand the whole system well, or even at all – they can still work on a small well-defined and decoupled module, and can do it in parallel with development on other modules.

Helps prevent Design Rot
As discussed in this blog post, design tends to degrade over time, especially in large monolithic systems. Modularity helps to prevent this rot, as pieces that become superseded by better ways of doing things can be replaced individually, and the impact to the overall system managed and isolated. This is akin to removing a brick from the wall and replacing it with a better brick, rather than tearing down the wall.

Solves the classpath hell problem entirely
Classpath hell is the name for the condition where a large number of dependencies are “in scope” at once. It’s not unique to Java, or even the JVM. It is apparently called “DLL hell” on Windows environments, but I’m happy to report I know very little about that. I do know it can happen on platforms other than the JVM, however.

By isolating the influence of dependencies, modularization, especially the way OSGi does it, makes it an explicit process to import and export exactly what packages you wish to and from your modules. What the module uses in it’s private implementation under the hood is no longer relevant to the system at large.

What happens in the bundle stays in the bundle, in other words, with appropriate apologies to Las Vegas.

Management of dependencies is the cure for classpath hell, and modularization is key to this, as described in this excellent article.

Allows smaller pieces of the system to be versioned
A key element in continuous releasability is the careful versioning of both individual modules and their dependency requirements. Modules give you the ability to release pieces of your system, not the whole system, better supporting the idea of continuous releasability and enhancing maintainability.

An article about software maintenance that a colleague recently sent me describes the problem perfectly – and independently versioned modules is a large part of the answer to this problem.

Re-Use
Although re-use it touted often as a major benefit of modularity, I’d categorize it as the least important benefit, not the most. Although the idea of re-using well-designed modules in new projects is indeed very powerful, it’s beyond the technical and design capabilities of many teams, often causing them to reject the need for modularity because they see no reason to build for re-use.

In a way, they’re right: you don’t build for re-use. You build for just the specific case at hand, but when you separate concerns properly and adhere to modularity best practices, you end up with a module that may lend itself to later re-use much more easily. If such a re-use case emerges, well and good, then you can reap the benefit. It is a lot like a framework – you can’t design a good one in isolation from it’s use cases – a framework is an emergent artifact from the re-use of code on many different (although similar) projects. The same is true of re-usable modules: you don’t set out to build a module explicitly to be re-usable.

Allows dependencies to be isolated: e.g. all Weblogic-specific code in one module, or all XMLMill-specific code in one module, increasing ability to refactor safely

Suggested Reading:
Agile Architecture Requires Modularity
Modularity Patterns
Runtime and Development time Modularity

In a later post we’ll look at some of the counter-arguments against modularity, and why I think they’re not especially valid or convincing in my experience.

A few weeks ago I had an opportunity to go to The Server-Side Java Symposium, an annual event for Java aficionados held in Las Vegas, Nevada. What happened there will not stay there, however.

I found a few interesting things out, and I’m going to try to share them over the course of a few blog posts, in case anyone out there is interested, and so I can find them later for myself as well

TSSJS is not about vendors, it’s about technical info, so there were only a few “booths” from the handful of commercial sponsors of the event, one of which was, inevitably, Oracle. Given the Sun acquisition, a question on the mind of many developers has been “what’s Oracle going to do for/with/to Java?”.

It’s a legitimate concern, and indeed a complex question with a number of complex answers – but one of my reasons for being in Vegas was to get some of the answers.

The keynote presentation at TSSJS was by James Gosling, the “father” of Java. As everyone no doubt reading this already knows, he’s subsequently left Oracle, for undisclosed reasons. That, to me, lessens the impact of his remarks somewhat, as I suspect he knew by the time we heard him speak in Vegas that he’d be on the way out the door, but he still had a lot of interesting things to say.

About 800 people were in the room, I estimated, to hear the keynote. Interestingly enough, Gosling’s position at Oracle was “Client Software Group” VP. I generally don’t think of Java as having much to do with “client” software, personally, so that in itself was informative.

He had an image of the Java mascot (the little delta-shaped guy) wearing a snorkle in a fish tank, a play on the “Soracle” name mash-up.

In any case, the keynote touched on what’s happening in the Java world. He emphasized the wide spectrum of devices on which Java is found, and how networking ties all of these devices more and more into a cohesive whole.

Some stats were mentioned: there are about 15 million downloads per week of Java, and something like 10 billion (with a “B”) Java-enabled devices of one sort or another out there in the wild, perhaps 1 billion of which are Java-enabled desktops. By any metric, Java is a huge success, and is in many ways more popular in the rest of the world, especially Europe, than it is in North America.

100 million of those devices are are TV devices, and on top of all it are about 2.6 billion Java-enabled cell phones.

Conclusion: It’s not going away, soon or quickly

A site like the Apple App Store existed for Java apps 7 years before Apple’s app store – but it was in Japan.

There are 5.5 billion smart card devices Java-enabled out there – the V3 standard actually has a servlet container in it – U.S. military dog-tags apparently now use this standard.

The Amazon Kindle is a Java device, under the skin.

Gosling proffered a definition of success: being completely invisible, people just get stuff done.

There are something like 6.5 million professional Java developers out there, 800 thousand of which are in India. Most colleges have a Java course requirement, modifying the old saw of “write once, run anywhere” to include “learn once, work anywhere”.

After this deluge of interesting stats, Gosling moved on to discuss the next big thing in Java, in his opinion: Java EE 6, the foundation for next-generation enterprise software. The spec was approved November 30th, 2009.

There are two major flavors of JEE6 – the “full” profile, with everything, and the “web” profile, with EJB3 “Lite” and just the most frequently used specs, not the more obscure stuff.

A major emphasis in next-generation Java development is modularity: making hard problems easier by breaking them into a number of less-hard problems. Major endorsement here for OSGi.

As far as Gosling’s concerned, the correct amount of XML required is exactly none Many developers in the room agreed with him, and it’s now possible with JEE6 and it’s powerful annotations.

Much of the benefit of dependency injection and resolution that Spring provides has been baking into the core Java EE platform now, and much of the pain of EJB has gone away. In fact, one of the most painful parts of EJBs, CMP Entity Beans, is no longer supported by the spec at all, in favor of JPA.

Glassfish V3 is the JEE6 reference implementation, and it’s OSGi-based.

Then it was demo time: Gosling (with some help from a couple of developer types) fired up Netbeans and Glassfish and did some show and tell of the coolness of JEE6.

We saw a webapp deployed – with no web.xml, no WEB-INF directory at all, everything annotation-based and dependency-injected at the proper times. We saw a stateless EJB auto-deployed to via an OSGi bundle into the container in less than a second, direct from the IDE. No boilerplate, no config, no reams of XML files to bolt it all together, just code and go.

We saw that same app preserve session state even when it was redeployed, which was very cool (a counter was showing the number of hits to the page – code was changed and redeployed, and the counter kept incrementing – and no, it wasn’t stored in a client-side cookie, it was in the session).

I noticed that all the Oracle guys were using Macs, incidentally

Gosling notes that JEE6 shouldn’t just be one version number past JEE5 – it’s more like JEE 60 in comparison

Saw the @Schedule annotation, where a bean can be scheduled for execution with cron-like ease, but all within a JVM.

Gosling noted that Felix was the web container under the skin of Glassfish.

We saw the @Observes annotation being used – a way to register listeners without any code or configuration. It was noted that Swing could benefit enormously by this pattern.

Then we saw a demonstration of Glassfish’s web console. The “web profile” of JEE6 doesn’t include JMS by default, so the example shown was adding that support by installing a new bundle in the running container. Two clicks and it was done. A GUI admin console can be added for any bundle, including your own bundles.

Gosling then wrapped up with an overview of some of his personal projects, and then took some questions. He also mentioned a new “app store” for Java that’s in the works, to be coming on “store.java.com”. It’s in the final stages of work now, apparently.

That was the end of the first keynote of TSSJS. In the next few posts I’ll talk about some additional sessions that my associate and I attended, and some of the other good stuff we learned.

On a recent project I’ve had the opportunity to help set up a new method of producing our software releases. We had a number of goals in setting up this process, and I think we’ve done pretty well in acheiving them. Some of these were:

• Each product should produce a single artifact as part of it’s continuous integration build cycle.
• Each artifact should be a release candidate.
• The product should be releaseable at all times, just with more and more functionality being added over time.
• Each artifact should be versioned sufficiently that we can exactly reproduce the source code it was built from from our source control system (currently SVN).
• Each release candidate artifact should be self-contained and ready to be installed on everything from local workstations to the production environment without modification or external configuration
• Developers should not be inconvenienced by the build/release process at all – they just check in their work and keep going
• Each release candidate should be automatically deployed to a test environment where automated acceptance tests can be run against it.
• Each release candidate should be tested via a suite of automated acceptance tests, and it should be clear which candidates pass that suite.
• Each release candidate should be deployed automatically to an exploratory test environment so the customer proxy can work with it interactively to further approve it for deployment to the customer, or not
• It should be the customer proxy’s decision whether or not to send a particular release candidate to the customer or not. He should not require developer input to that decision.
• The build and install process we use should be the exact same for all environments.
• Once a release candidate is produced, it should be the exact same artifact all the way to production, no changes, no re-building.

In our environment we produce two applications at the moment, which are both part of the same suite of applications and communicate with each other when deployed. We have an automated integration test that verifies they communicate correctly.

One application is packaged as an EAR (Enterprise Application aRchive) file, and is deployed to a cluster of Weblogic servers. The other is packaged as a group of OSGi bundles, one of which is also a web application (WAR) file. We have automated the build/test/release process in the same way for both products, although their install process is of course quite different.

Let me walk through how our process works, starting from the point where a developer checks in some code that implements a new feature (or some other story given to us by the customer proxy).

1. The developer builds and checks his change locally (using the same build and install process as the rest of the sequence, but I’ll skip those steps for clarity for now, as they’re the same as the steps below).
2. Our TeamCity server notes a change in the Subversion repository for the appropriate product (let’s say it’s the Weblogic-based one for now), and triggers the CI job
3. The CI job checks out the latest revision from Subversion and triggers a Maven “clean install” target. Our projects both consist of multiple modules with an aggregator POM, that is, a POM file that describes a set of subordinate modules, each of which also has it’s own POM file. This is helpful for both development and building, as when developing, you can choose to open the aggregator POM with your IDE (we mostly use JetBrains IntelliJ IDEA) if you’re refactoring accross modules, or you can open just one of the components if that’s all you’re working on.
4. The aggregator POM does a compile and test cycle on each subordinate module, running both it’s unit and functional tests (we use an annotation to separate the two -but that’s another blog post)
5. In the aggregator POM we have two important plugins configured. The first is the “Build Number” plugin, which reads the meta-data from the SVN checkout and determines the global revision number that it just checked out. This number is then used in a couple of places: first, it’s built into the MANIFEST.MF file of the resulting artifact, so that we can tell exactly where this source code came from. Secondly, it’s written into a properties file (via the Maven properties plugin) for use in the installation process later. The second plugin is the “assembly” plugin. This tells Maven to collect all of the produced artifacts from the build of all of the modules into a single ZIP file, which we’ll describe in more detail below.
6. When the “assembly” plugin triggers (at the end of the “install” lifecycle phase, which we tell it to do via configuration in the POM), Maven automatically builds us a zip file. Through an additional option, we tell this zip file to include a set of installation scripts. In the case of our Weblogic app, these are Jython scripts that configure the target Weblogic server properly, then deploy the application to the entire cluster at once.
7. The zip file that assembly produces has two version numbers in it’s name (and inside it). The first is the version number from the aggregator POM, which is the “public” version of the application, e.g. 3.1.18 or such. As developers, we don’t care about this version at all, and don’t set it – our customer proxy determines it. The second number is the SVN revision number, which we care about more, as it tells us exactly what point in the source code our artifact was built from.
8. The TeamCity server now completes the initial CI job, and retains the versioned artifact. It might have a name like “productx-3.1.18-r1234.zip.
9. Like most CI servers, TeamCity can then trigger the next dependant job, which in this case copies the artifact from the initial job and uses Ant to SCP it to our automated test cluster, unzip it, and trigger the embedded install script. Bundling the installer with the product means that when the installation needs to change, we just change the script and it goes out automatically, versioned the same was as the code it’s installing.
10. Now the “deploy to test cluster” job is complete, and an series of automated acceptance tests are triggered against the test cluster by TeamCity. If these pass, the test job also gets a copy of the original artifact, and retains it. This is the point where the customer proxy can do exploratory testing against the test cluster if desired, and make up his mind if the release candidate is of sufficient quality and utility to be shipped to the customer. If so, he “pins” the build in TeamCity to leave a note, and ships the zip file off to the end-user for their installation.
11. At this point the customer proxy might want to bump the “public” version number, and he can simply change the version in the aggregator POM file to do so. The developers don’t care – they are concerned with versions only of the sub-modules, as some of them depend on other sub-modules.

For our OSGi-based product we use the exact same process, with the exception that the installer is Java/JMX based instead of a set of Jython scripts. In both cases, the user of the artifact to be installed just unzips the file and says “install.sh” (or install.bat).

We don’t use the Maven release plugin for this process at all, as it “tags” at the end of the build cycle, not the beginning. We don’t even necessarily need to use SNAPSHOT versions, as the version of the actual internal modules are never visible to the customer in any case, and we rachet to the target release number at the beginning of the release development cycle, not the end – in other words, if we’re heading towards releasing 3.2 of the product, the customer proxy changes (or asks us to change) the version of the aggregator POM to 3.2 immediately. Then we produce 3.2-r1234, 3.2-r1235, 3.2-r1236 and so forth until the customer proxy says “ok, it’s done”, at which point he (not we) sends it off to the customer, and bumps us to 3.3.

This is quite a different sequence than many teams use, although I have used it before (although not with TeamCity).

So far it’s working for us, we’ll report if we find things to optimize about it.

Comments and/or questions are most welcome!