I’ve been sold for some time on the benefits of highly modular software development (and deployment, for that matter, almost equally importantly).
However, I’m aware that not everyone is on the same page, or has the same definition of “modularity” in the first place, so I thought I’d collect and enumerate the things that have convinced me, as best I can.
I’ve been in the software business for a very long time, and sometimes I have a preference or leaning towards a certain technique or technology. Sometimes that preference has clear causes behind it, but sometimes not. Modularity is one of the latter, where I know it’s not only good, but critically important, but it’s hard for me to immediately say why or how I know that in a succinct way.
So, I apologize for the length of this explanation, as I’ve not yet taken the time to make it shorter
Modularity Defined First things first: What do I mean by modularity?
In short, I mean a system comprised of relatively small self-contained units. The idea of modularity has caught on much more completely in many other disciplines, but far less so in software. In manufacturing, especially mechanical and electronic, in business, especially finance, the idea of modularity is well established. Modularity is used to increase and maintain quality, and reduce cost while increasing output. In the automotive industry, for example, extensive use of standardization and modularity allows us all to afford a car, as opposed to just the wealthy or the mechanically inclined.
In the software world, the use of modularity is still woefully small, however. When it is employed, though, some spectacular successes have arisen. One of the oldest examples is Unix, arguably the most successful operating system ever. Since 1969, the “do one thing, do it well, and play well with others” modular design of Unix has been the basis of it’s success and flexibility, allowing it to span everything from embedded devices to mainframes.
The Java Virtual Machine (JVM) is another realm where modularity is on the rise, and starting to have an impact. Many different approaches are available, and some of them work with any of the large number of languages available on this platform.
Increase overall success rate of projects In the many software projects I’ve been involved with over the years, I’ve noticed a few trends. Smaller, well-contained projects have a higher success rate than larger and more poorly contained projects. (By contained, I mean the number of places and ways the project interacts with other projects is small). I’m sure there are a host of reasons for this, but part of it is simple comprehension – small projects are generally easier to understand than big one, so the number of times things get done wrong is reduced.
A system comprised of well-isolated modules tends to be more like a set of small projects that happen to work together, so I find this reason alone a plus for modularity.
But this hides much of the detailed reasoning behind why modularization works, so let’s dig a bit deeper…
Makes the API between components explicit, not hidden Sometimes I heard the argument that modularity is complex, that it actually increases the difficulty of working with a system. I find that doesn’t hold up under examination: when modularity seems complex, it’s probably because it’s being done wrong, or being retrofitted to a system that was not designed with modularity in mind. What you’re really seeing in this situation is that the attempt to modularize is exposing the complexity that was already there.
When a system is properly designed in a modular fashion, the APIs between modules are small and well-defined, with a limited number of cross-module dependencies.
Reduce unnecessary complexity By reducing a large complex system into a series of small and better understood modules, we’ve reduced the complexity we have to deal with at any one time. This means that the overall large system only has to deal with how the exposed APIs of our modules will be fitted together, treating the modules themselves as “black boxes” that perform specific functions, without worrying about how they perform it.
In both cases, we’ve reduced the complexity we need to deal with.
I specify here “unnecessary” complexity, as of course the complexity of the issue itself doesn’t get any simpler just because we modularize it – but if we don’t have to deal with a lot of tangled plumbing, we can concentrate on the real issue.
Increase testability A module that performs a single function is much easier to test exhaustively than an entire monolithic system. The number of interdependence is limited, meaning we can do more of our testing at the unit and orchestration level than at the functional or integration level, meaning our feedback time is improved and our coverage likely increased.
This means our development velocity can increase, as the impact of changes can be evaluated more easily and more quickly.
Support Composition When you have a number of smaller building blocks, it’s relatively easy to reshuffle those blocks to change the behaviour and functionality of your overall system. This is making use of composition – connecting blocks together to do more than they can on their own, and its a pattern that is supported well in a modular environment.
Although composition is often thought of in the context of re-usable modules, when building a new system, it’s also valuable in modules that are only used within one system, as it allows the large-grained behavior of the system to change much more rapidly than without modules.
Composition makes large complex systems easy to understand and work on, and vastly increase the maintainability of the overall system.
Impact of changes isolated The isolation of changes to an effected module is a major benefit of modularity, especially as it supports maintainability. If you’re constantly concerned with your changes affecting parts of the system you’re not actually working on, you’re not able to go as fast as you could if you knew that your changes would only apply to a small area of the whole system – the current module.
This is related to and similar to the support for refactoring that unit tests give – if you’re tests are all green after your refactor, you know you haven’t changed the functionality of your system. In modularity, if you’re isolated from other modules in the system, this goes a step further – you can change the functionality of the module freely without concern about other bits of your system breaking. This is true if you’re doing anything that’s not in the one package you’ve “exported” as the service definition.
Allows physical design to reflect logical design Modularity also allows developers to build systems where the logical design actually corresponds as needed to the physical design, an often overlooked abstraction. Kirk’s blog post on the topic describes this issue better than I could, so I refer you there.
Supports Refactoring Better Refactoring is the process of changing the implementation of a system without changing it’s behavior. One of the primary reasons to write unit tests, for instance, is to support and allow refactoring, so we know the system does the same thing when we’re done as when we started.
Modularization supports this ability by making clear the portions of a system that affect other modules, and the portions which do not. When you’re refactoring in a monolithic system, even with IDE support, the scope of your changes is the entire project. When you’re refactoring in a modularized system, if you’re “inside” the module, you have no need to consider scope on the outside, as there isn’t any. And when you’re refactoring the exported interfaces, you know ahead of time your scope is cross module, and can take it into account.
In short, modularization lets you refactor more freely within a module, and lets you be intentional and organized when refactoring between modules.
A colleague recently suggested that modularization would interfere with the ability to see a system as a whole. It’s been my experience that the exact opposite is true – a well structured set of modules can still be considered as a whole when necessary (an aggregator POM is any easy way to set this up in the Maven universe), but it also allows modules to be worked on “safely” in isolation, which a monolithic system does not. This ability to only look at one piece at a time is all the more critical the larger the system becomes.
The need to refactor “across” modules is a warning sign, in my opinion – it generally indicates that the API between modules is changing, and this is a change you want to be more aware of then any amount of changes within modules. If it happens often, it probably indicates that the module boundaries are not yet stabilized, and you have more design work to do.
Scalable Development A modular system lends itself well to many hands being involved in it’s development and maintenance. It’s not necessary for a team or a pair to understand the whole system well, or even at all – they can still work on a small well-defined and decoupled module, and can do it in parallel with development on other modules.
Helps prevent Design Rot As discussed in this blog post, design tends to degrade over time, especially in large monolithic systems. Modularity helps to prevent this rot, as pieces that become superseded by better ways of doing things can be replaced individually, and the impact to the overall system managed and isolated. This is akin to removing a brick from the wall and replacing it with a better brick, rather than tearing down the wall.
Solves the classpath hell problem entirely Classpath hell is the name for the condition where a large number of dependencies are “in scope” at once. It’s not unique to Java, or even the JVM. It is apparently called “DLL hell” on Windows environments, but I’m happy to report I know very little about that. I do know it can happen on platforms other than the JVM, however.
By isolating the influence of dependencies, modularization, especially the way OSGi does it, makes it an explicit process to import and export exactly what packages you wish to and from your modules. What the module uses in it’s private implementation under the hood is no longer relevant to the system at large.
What happens in the bundle stays in the bundle, in other words, with appropriate apologies to Las Vegas.
Management of dependencies is the cure for classpath hell, and modularization is key to this, as described in this excellent article.
Allows smaller pieces of the system to be versioned A key element in continuous releasability is the careful versioning of both individual modules and their dependency requirements. Modules give you the ability to release pieces of your system, not the whole system, better supporting the idea of continuous releasability and enhancing maintainability.
An article about software maintenance that a colleague recently sent me describes the problem perfectly – and independently versioned modules is a large part of the answer to this problem.
Re-Use Although re-use it touted often as a major benefit of modularity, I’d categorize it as the least important benefit, not the most. Although the idea of re-using well-designed modules in new projects is indeed very powerful, it’s beyond the technical and design capabilities of many teams, often causing them to reject the need for modularity because they see no reason to build for re-use.
In a way, they’re right: you don’t build for re-use. You build for just the specific case at hand, but when you separate concerns properly and adhere to modularity best practices, you end up with a module that may lend itself to later re-use much more easily. If such a re-use case emerges, well and good, then you can reap the benefit. It is a lot like a framework – you can’t design a good one in isolation from it’s use cases – a framework is an emergent artifact from the re-use of code on many different (although similar) projects. The same is true of re-usable modules: you don’t set out to build a module explicitly to be re-usable.
Allows dependencies to be isolated: e.g. all Weblogic-specific code in one module, or all XMLMill-specific code in one module, increasing ability to refactor safely
Suggested Reading: Agile Architecture Requires Modularity Modularity Patterns Runtime and Development time Modularity
In a later post we’ll look at some of the counter-arguments against modularity, and why I think they’re not especially valid or convincing in my experience.
In Part 1 of our OSGi adventures, we described how to build a nice Ajax-aware Vaadin UI app, and couple it to a generic back-end OSGi service we called the service dispatcher.
Now we’ll take the adventure to the next step, show how to deploy that webapp into our OSGi container and get it up and running, then go through the functionality of the Dispatch Service, and show how it routes requires through to the first of our domain-specific application services.
In our UI, we built a form that allowed the user to enter and save a “Dealer”, with some fields like name, phone number, etc., so the first service we’ll build for the service dispatcher to talk to will be our “dealer” service.
First, though, let’s see how to get our webapp into our OSGi container.
As we’re building our Vaadin app with Maven, we can easily add the small bits of additional configuration to turn our project into an OSGi-friendly WAR file.
OSGi deploys “bundles”, but a bundle is just a jar file (or a war, which is after all just a special kind of jar) with a bit of extra meta-data. The META-INF/MANIFEST.MF file is where the magic happens. We add the following to our POM:
<plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-war-plugin</artifactId> <version>2.0</version> <configuration> <archive> org.springframework.osgi.web.context.support,\ org.springframework.web.servlet,\ org.springframework.web.servlet.handler,\ org.springframework.web.servlet.mvc,\ org.springframework.web.servlet.view,\ dwmj.domain,\ org.springframework.web.servlet.mvc.annotation,\ org.springframework.web.context <manifestEntries> <Bundle-ManifestVersion>2</Bundle-ManifestVersion> <Bundle-SymbolicName>com.point2.Admin</Bundle-SymbolicName> <Bundle-Name>Admin</Bundle-Name> <Bundle-Version>1.0.0</Bundle-Version> <Bundle-Activator>com.point2.ServiceClient</Bundle-Activator> <Import-Package>org.osgi.framework,com.point2.services.dispatch,javax.servlet;version="2.4.0", javax.servlet.http;version="2.4.0",org.osgi.service.http;version="1.2.0", org.osgi.util.tracker;version="1.3.2"</Import-Package> <Webapp-Context>admin</Webapp-Context> <Bundle-ClassPath>WEB-INF/classes, WEB-INF/lib/service-dispatch-api-1.0.0.jar, WEB-INF/lib/vaadin-6.1.3.jar</Bundle-ClassPath> </manifestEntries> </archive> </configuration> </plugin>
This bit of magic allows Maven to build us a MANIFEST.MF like this:
Manifest-Version: 1.0 Archiver-Version: Plexus Archiver Created-By: Apache Maven Built-By: mnash Build-Jdk: 1.6.0_15 Extension-Name: admin Implementation-Title: admin Implementation-Version: 1.0-SNAPSHOT Bundle-Activator: com.point2.ServiceClient Bundle-ClassPath: WEB-INF/classes, WEB-INF/lib/service-dispatch-api-1. 0.0.jar, WEB-INF/lib/vaadin-6.1.3.jar Bundle-ManifestVersion: 2 Bundle-Name: Admin Bundle-SymbolicName: com.point2.Admin Bundle-Version: 1.0.0 Import-Package: org.osgi.framework,com.point2.services.dispatch,javax. servlet;version="2.4.0",javax.servlet.http;version="2.4.0",org.osgi.s ervice.http;version="1.2.0",org.osgi.util.tracker;version="1.3.2" Webapp-Context: cadmin
Now we have our OSGi-friendly .war file, sitting in our target directory. We can then connect to the console of our OSGi container (in this case, ServiceMix), and say:
install war:file:/Users/mnash/experiments/admin/target/admin-1.0.0-SNAPSHOT.war
Our container honors the “Webapp-Context” tag in our manifest, so we can then surf to http://localhost:8181/admin/ to see our application. 8181 is the default port for the Jetty HTTP service in ServiceMix – it can easily be changed to another number as required, of course.
Dispatch Service Now that we can see how to get the wepapp into ServiceMix, let’s look at the Dispatch Service in detail.
Our Dispatch Service is going to be a regular OSGi bundle, so we have a separate Maven project that produces a .jar file in this case, not a WAR file.
Our MANIFEST.MF needs the following magic for this project:
Bundle-ManifestVersion: 2 Bundle-SymbolicName: com.point2.services.dispatch.DispatchService Bundle-Name: DispatchService Bundle-Version: 1.0.0 Bundle-Classpath: .,lib/commons-beanutils-1.8.0.jar,lib/commons-collections-3.2.1.jar Bundle-Activator: com.point2.services.dispatch.impl.DispatchServicePublisher Import-Package: org.osgi.framework Export-Package: com.point2.services.dispatch
You can see we’re again specifying a “Bundle-Activator” class, which, much like the ServiceClient class in our webapp, is called by the OSGi framework when the bundle containing this service is started.
One slight oddity: The dispatch service needs the two jars listed under “Bundle-Classpath:” to do it’s business – because we are building an OSGi bundle, we “embed” these jars in our bundle (which is, yes, another jar) by putting them in the src/main/resources/lib directory. We refer to them in that location in the POM, and Maven automatically includes them in the finished jar, where they’re available when our bundle needs them. The other alternative is to install the dependencies as their own bundles, but that’s a whole ‘nother post
In the case of DispatchService, we’ve got an activator like this:
public class DispatchServicePublisher implements BundleActivator { private ServiceRegistration registration; public void start(BundleContext context) throws Exception { registration = context.registerService(DispatchService.class.getName(), new DispatchServiceImpl(), null); DispatchServiceImpl.setBundleContext(context); System.out.println("Dispatch Service registered"); } public void stop(BundleContext context) throws Exception { registration.unregister(); System.out.println("Dispatch Service unregistered"); } }
Which simply takes a reference to the BundleContext on startup and passes it to the DispatchServiceImpl, which implements the DispatchService interface, like so:
public interface DispatchService { List<Map<String, Object>> call(String serviceName, String serviceOperation, Map<String, Object> parameters, String versionPattern, String securityToken) throws Exception; }
The big JuJu with OSGi is that only this interface is “exposed” from the bundle. No other service can see the innards of our service, unlike Jars on a classpath.
The implementation of the DispatchService is equally straightforward:
public class DispatchServiceImpl implements DispatchService { private static BundleContext context; public List<Map<String, Object>> call(String serviceName, String serviceOperation, Map<String, Object> parameters, String versionPattern, String securityToken) throws Exception { ServiceReference reference = getServiceNamed(serviceName); if (reference == null) throw new RuntimeException("There is no service with service-name " + serviceName); Object service = context.getService(reference); Adapter adapter = new Adapter(service); List<Map<String, Object>> response = adapter.callList(serviceName, serviceOperation, parameters); context.ungetService(reference); return response; } private ServiceReference getServiceNamed(String serviceName) throws Exception { ServiceReference[] references = context.getAllServiceReferences(null, null); System.out.println("There are " + references.length + " services available"); for (int i = 0; i < references.length; i++) { ServiceReference reference = references[i]; Object serviceNameValue = reference.getProperty("service-name"); if (serviceNameValue != null) { System.out.println("I see service with name " + serviceNameValue); if (serviceNameValue.toString().equalsIgnoreCase(serviceName)) return reference; } } throw new RuntimeException("There is no service available with service-name " + serviceName); } public static void setBundleContext(BundleContext newContext) { context = newContext; } }
The dispatch service implementation simply looks through the “published” services in the OSGi context, and looks at the “service-name” property of each service to find the one specified in the call. Assuming it finds an appropriate service, it then uses another class, Adapter, to do the type conversion of the generic Map of parameters to the specific types needed for the service being called, then uses Java reflection to actually make the call and return the response as a List of Maps, again converting the service-specific types to a generic Map in order to pass the response back to the user interface.
Why do we go through those gyrations, instead of just having access to the domain-specific beans in our UI? If we want a full decoupling, and the advantages that come with it, the type-independence of this approach gives us that. It also allows parallel development of the UI and the back-end services without an ever-increasing number of binary dependencies as well – for that matter, with the static data adapter in the Vaadin app allows us to develop on our UI entirely without the back-end services. That’s pretty decoupled.
The Dispatch Service by itself, though, doesn’t give us any functionality. It acts much like a router on a network, simply moving the request from the client to the proper service on the back-end, so let’s build such a service – in this case, a service for handling Dealers.
Dealer Service The bulk of our application-specific code will be prepared as independent OSGi services, just like this one. In later postings, I’ll describe how to set up dependencies between services (and how to use Spring DM to make doing that kind of thing far simpler).
First, let’s look at our MANIFEST.MF for our new service (again, we can use Maven to produce this for us, but the result is the same):
Bundle-ManifestVersion: 2 Bundle-SymbolicName: com.point2.services.dealer.DealerService Bundle-Name: DealerService Bundle-Version: 1.0.0 Bundle-Activator: com.point2.services.dealer.impl.DealerServicePublisher Import-Package: org.osgi.framework Export-Package: com.point2.services.dealer
The two key elements above are the Bundle-Activator, a class called DealerServicePublisher, and the Export-Package.
OSGi best practices dictate that the package that is exported should only contain the interface for the class (or classes) you wish to make available from your service. In our case, that comprises a bean class, called “Dealer”, and the interface to our actual service, DealerService.
The Dealer bean is very simple, just your basic property-holding JavaBean:
public class Dealer { private String name; private String phone; private String contactLast; private String contactFirst; private String address1; private String address2; public String getContactLast() { return contactLast; } public void setContactLast(String contactLast) { this.contactLast = contactLast; } public String getContactFirst() { return contactFirst; } .....
I’ve ommitted the rest of the getters and setters here for brevity (in case you’re wondering, yes, this is much easier to express in Scala, and yes, OSGi works just fine with Scala…)
The interface for our service is even simpler:
public interface DealerService { public void save(Dealer dealer); public Dealer findById(int id); }
Of course, a fully fleshed-out service might in fact have more methods, but again, you get the idea.
I won’t bore you with the actual implementation of the DealerServiceImpl class, but it’s important to note that it’s not in the same package as the Dealer bean and the DealerService implementation. They are the only two classes (well, one class, one interface) in that package, as the entire package is “exported” by OSGi, and therefore visible to other services and clients.
The DealerServicePublisher is the last piece to examine, and it’s pretty straightforward as well:
package com.point2.services.dealer.impl; import com.point2.services.dealer.DealerService; import org.osgi.framework.BundleActivator; import org.osgi.framework.BundleContext; import org.osgi.framework.ServiceRegistration; import java.util.Dictionary; import java.util.Hashtable; public class DealerServicePublisher implements BundleActivator { private ServiceRegistration registration; public void start(BundleContext context) throws Exception { System.out.println("Registering dealer service"); Dictionary dictionary = new Hashtable(); dictionary.put("service-name", "dealer"); registration = context.registerService(DealerService.class.getName(), new DealerServiceImpl(), dictionary); } public void stop(BundleContext context) throws Exception { System.out.println("Unregistering dealer service"); registration.unregister(); } }
Essentially all we do in this activator is “register” our service, making it visible to the OSGi container and other services or clients that need it. We add an extra property via the “Dictionary” object, which allows us to specify arbitrary properties to be associated with our service. Because we want to look up our services from the dispatcher by name, rather than by class or interface name, we use the string “dealer” and associate it with the key “service-name”. If you examine the code for our DispatchService, you’ll see that it uses this property to find services.
Now we can build our new service with a simple “mvn package” command, and install the resulting jar into our OSGi runtime with “install file:/Users/mnash/experiments/dealer/target/dealer-1.0.0.jar” (again, you can do an install directly from the Maven repository, or from a URL, as opposed to a file).
That’s it – our “dealer” service is now available, and our “dispatch” service is fired up and ready to locate it.
If we deploy our Vaadin application in our OSGi container (as described in the previous post), you’ll find that calls to the “call” method of the ServiceClient now return the “real” data from our service.
In the next few posts we’ll examine an even easier way to define new services, and explore the power of OSGi for updating and working with our decoupled services. Then we’ll look at how to hook services together, allowing services to call other services to do their jobs as required.
In my experience, the future of modularity on the JVM is OSGi.
Many developers don’t seem to recognize the need for it, but other bloggers have covered this in great detail (see my Resources page for some links), so I’m not going to try to “sell” OSGi here at all – I assume you’re already sold, and looking at how to put OSGi to good use in your development and deployment process.
Extending my recent experiments with the Vaadin framework, I decided I wanted to have a Vaadin front-end talking to a set of OSGi services on the back end. Initially, these will be running within the same OSGi container, which in this case is FUSE 4, the commercially supported variant of Apache ServiceMix.
One of my goals was to acheive a loose coupling between the Vaadin webapp and the backing services, so that new services can readily be added, started, stopped, and updated, all without any impact on the running Vaadin app. I also wanted to maintain the convenience of being able to run and tinker with the UI portion of my app by just doing a “mvn jetty:run”, so the app needed to be able to start even if it wasn’t inside the OSGi container.
Fortunately, doing all this is pretty easy, and in the next series of articles I’ll describe how I went about it, and where the good parts and bad parts of such an approach became obvious.
In this part, we’ll start by describing the Vaadin app, and how it calls the back-end services. In later parts, I’ll describe the evolution of the back-end services themselves, as I experimented with more sophisticated techniques.
Vaadin Dependency I’m building all my apps with Apache Maven, so the first step was to create a POM file suitable for Vaadin. Fortunately, Vaadin is a single jar file, and trivial to add to the classpath. My POM needed this dependency:
<dependency> <groupId>vaadin</groupId> <artifactId>vaadin</artifactId> <version>6.1.3</version> <scope>system</scope> <systemPath>${basedir}/src/main/webapp/WEB-INF/lib/vaadin-6.1.3.jar</systemPath> </dependency>
Here I’m using the trick of specifying a systemPath for my jar, instead of retrieving it on demand from a local Nexus repository or from the internet, but that’s just one way of doing it – the main thing is to get this one Vaadin jar on your path.
web.xml Next I proceeded to tweak my web.xml to have my top-level Vaadin application object available on a URL. The main Vaadin object is an extension of a Servlet, so this is also very easy – here’s my web.xml in it’s entirety:
<?xml version="1.0" encoding="UTF-8"?> <web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5"> <display-name>Admin</display-name> <servlet> <servlet-name>Admin</servlet-name> <servlet-class>com.vaadin.terminal.gwt.server.ApplicationServlet</servlet-class> <init-param> <param-name>application</param-name> <param-value>Admin</param-value> </init-param> </servlet> <servlet-mapping> <servlet-name>Admin</servlet-name> <url-pattern>/*</url-pattern> </servlet-mapping> </web-app>
In this situation my “Admin” class is in the default package, which is not generally a good practice, but you get the idea.
Menu and MenuDispatcher I then used the “Tree” class in Vaadin to build myself a nice tree-style menu, and added that to the layout on my main page. My main page has some chrome regions for a top banner and other assorted visual aids, then a left-side area where the menu lives, and a “main” area, where all the real action in the application will happen.
A class I called MenuDispatcher “listens” for events on the menu (e.g. the user clicking something), and does the appropriate action when a certain menu item is clicked.
Here’s the interesting bits from the MenuDispatcher – as you can see, it’s constructed with a reference to the “mainArea” layout when it’s first initialized.
public class MenuDispatcher implements ItemClickEvent.ItemClickListener { private VerticalLayout mainArea; public MenuDispatcher(VerticalLayout mainArea) { this.mainArea = mainArea; } public void itemClick(ItemClickEvent event) { if (event.getButton() == ItemClickEvent.BUTTON_LEFT) { String selected = event.getItemId().toString(); System.out.println("Selected " + selected + " from menu"); if (selected.equals("create dealer")) { createDealer(); } else if (selected.equals("edit dealers")) { editDealer(); } ... } System.err.println("Selected " + event.getItemId()); } } private void createDealer() { mainArea.removeAllComponents(); Component form = new CreateDealerForm(); mainArea.addComponent(form); mainArea.setComponentAlignment(form, Alignment.MIDDLE_CENTER); mainArea.requestRepaint(); } private void editDealer() { ... } ... }
Again, this code can be made more sophisticated – I’m thinking a little Spring magic could make adding new forms and such very convenient, but this gets us started.
Submitting the Form The “CreateDealerForm” object referred to in the Dispatcher then builds a Vaadin “Form” class, much like the example form built in the “Book of Vaadin”. The only interesting part of my form was that I chose not to back it with a Bean class, which is an option with Vaadin forms. If you back with a bean, then you essentially bind the form to the bean, and the form fields are generated for you from the bean.
If I wanted to then send the corresponding bean to the back-end service, then binding the bean to the form would be a good way to go. Instead, however, I don’t want my back-end services to be sharing beans with my UI application at all. I’ll explain why and how later on in more detail.
The interesting part of my form, then, is how I handle the submission of the form:
Assuming I have a button on my form, defined like so:
Button okbutton = new Button("Submit", dealerForm, "commit");
I can add a listener to this button (again, using Vaadin’s magic ability to route the Ajax events to my Java code) like thus:
okbutton.addListener(new Button.ClickListener() { public void buttonClick(Button.ClickEvent event) { Map<String, Object> parameters = new HashMap<String, Object>(); for (Object id: dealerForm.getItemPropertyIds()) { Field field = dealerForm.getField(id); parameters.put(id.toString(), field.getValue()); } System.out.println("*** Calling dealer service via dispatcher"); ServiceClient.call("dealer", "save", parameters, null, null); getWindow().showNotification("Dealer Saved"); } });
I’m using an anonymous inner class to listen to the event, and the “buttonClick” method gets called when the user says “Submit”.
The next steps are where the form meets the back-end service: First, I iterate over the form and build a map containing all the field values. The map is keyed with a string, for the name of the field or property, and the value in the map is the value the user entered. Note that these values are already typed – e.g. a checkbox can have a boolean, a TextField can have a string, a calendar field can have a java.util.Date. We retain these types, and wrap everything up in a map.
Now (after a quick println so I can see what’s going on), I call a static method on class called ServiceClient, sending along the name of the service I want to call, the operation on that service, and the parameter map I just built from my form.
The last line just shows a nice “fade away” non-modal notification to the user that the dealer he entered has been saved (assuming the call to ServiceClient didn’t throw an exception, which we’re assuming for the moment for simplicity).
So, now we have our “call” method to consider, where the map of values we built in our Vaadin front-end gets handed off to the appropriate back-end service.
Calling the Service
The only job of the ServiceClient object is to route a call from somewhere in our Vaadin app (in this case, the submission of a form) to the proper back-end service, and potentially return a response.
We identify our back-end services by a simple string, the “service name” (our first argument). The second argument to call tells us the “operation” we want from this service, as a single service can normally perform several different operations. For instance, our dealer service might have the ability to save a dealer, get a list of dealers, find a specific dealer, and so forth.
In Java parlance, a service operation might correspond to a method on the service interface, but we’re not coupling that tightly at this point – our service, after all, might not be written in Java for all we know at this point, and I prefer to keep it that way.
This is the “loose joint” in the mechanism between the UI and the back-end. To keep the joint loose, we don’t send a predefined Bean class to the back-end service to define the parameters to the service operation, we send a map, where that map contains a set of key/value pairs. The keys are always Strings, but the values can be any type – possibly even another map, for instance, which would allow us to express quite complex structures if required, in a type-independent fashion.
Let’s have a look at ServiceClient:
public class ServiceClient implements BundleActivator { private static DispatchService dispatchService = null; public void start(BundleContext context) throws Exception { ServiceReference reference = context.getServiceReference(DispatchService.class.getName()); if (reference == null) throw new RuntimeException("Cannot find the dispatch service"); dispatchService = (DispatchService) context.getService(reference); if (dispatchService == null) throw new RuntimeException("Didn't find dispatch service"); } public void stop(BundleContext context) throws Exception { System.out.println("Stopping bundle"); } public static List<Map<String, Object>> call(String serviceName, String operation, Map<String, Object> params, String versionPattern, String securityToken) throws Exception { if (dispatchService == null) { System.out.println("No OSGi dispatch service available - using dummy static data"); return StaticDataService.call(serviceName, operation, params, versionPattern, securityToken); } return dispatchService.call(serviceName, operation, params, versionPattern, securityToken); } }
Let’s go through that class piece by piece. First, you’ll notice that the class implements the BundleActivator interface – this tells the OSGi container that it is possible to call this class when the OSGi bundle containing it is started and stopped. During the start process, you can have the class receive a reference to the BundleContext. This is somewhat analagous to the Spring ApplicationContext, in that it gives our class access to the other services also deployed in OSGi. The Spring DM framework lets you do this kind of thing more declaratively, but it’s good to know how the low-level works before engaging the autopilot, I always find.
In order to allow BundleActivator to be found, we need another couple of things on our classpath, so we add this to our POM:
<dependency> <groupId>org.osgi</groupId> <artifactId>osgi_R4_core</artifactId> <version>1.0</version> </dependency> <dependency> <groupId>org.osgi</groupId> <artifactId>osgi_R4_compendium</artifactId> <version>1.0</version> </dependency>
This defines the BundleActivator interface and other OSGi requirements which we use.
As you can see, we use our reference to the OSGi context to get a ServiceReference to an interface called “DispatchService”. We’ll examine dispatch service in detail in my next posting, but for now you can see we hold on to our reference as an instance variable.
When the call to the “call” method happens, our DispatchService reference is already available if we’re running inside an OSGi container, all wired up and ready to go. To give us flexibility, though, we also allow for the case where dispatch service is null, meaning we’re not running inside and OSGi container.
Instead of crashing and burning, however, we simply redirect our call to a “StaticDataService” class, which does just what you might expect. For every call it understands, it returns a static “canned” response. This allows us to build and test our UI without actually having written any of the back-end services, and to continue to run our Vaadin app with a simple “mvn jetty:run”, when all we’re working on is look and feel, or logic that only affects the UI.
This means my “cycle time” to test a change in the UI code is a matter of seconds – when I do a “mvn jetty:run” my code is compiled and up and running in my browser in about 5 seconds, and that’s on my 5 year-old macbook laptop, so there’s no penalty for not having a dynamic language in the mix here, from my point of view.
If DispatchService is not null, however, then we’re running “for real” inside an OSGi container, so we use our stored reference to the dispatch service to “forward on” our call. The dispatch service works it’s magic, which we’ll examine in a later post, and returns a List of Map objects with our response. This list might only contain one Map, of course, if the service was a simple one.
The response is then returned to the caller elsewhere in the Vaadin application, to do whatever is necessary from a UI point of view – perhaps populate a form or table with the response data.
As we’ll see in detail in my next post, the dispatch service in this case acts as a “barrier” between the UI-oriented code in our Vaadin app and our domain-specific application logic contained in our services. It is responsible for mapping our generic map of parameters into whatever domain beans are used by our back-end services, and for figuring out which of those services should be called, and which operation on that service is required. Those services then return whatever domain-specific objects they return, and the dispatcher grinds them into non-type-bound maps, or lists of maps, if there is a whole series of returns.
This means our Vaadin app only ever has to talk to one thing: the dispatcher. We only change our Vaadin code for one reason: to change the UI, never in response to a change of service code, even if the beans and classes of that service change significantly.
Next time we’ll look at the dispatch service and the first of our application-specific domain-aware services, and see how they come together.
The Apache Group have recently released version 4 of the ServiceMix ESB. (Enterprise Service Bus), and I had a chance to work with it a bit over the weekend.
As we develop more and more services, the plumbing is starting to get complicated. We’re starting to ask questions like what versions of what services do we have? What dependencies do we have between services? How can/should we deploy services? Should we use REST or JMS for this service? How can we easily manage and monitor all these services in a fully clustered, scalable and high-availability environment?
In short, how can we develop powerful scalable services fast and not worry about the plumbing?
We’re not the only people to have asked these questions – and one powerful answer is to use an ESB.
ESB’s have come a long way, and the JBI standard and OSGi finally get together in this version of ServiceMix, and some pretty powerful magic happens as a result.
Just about every organization that writes sophisticated applications, particularly Java applications, have run into some of the problems that an ESB provides solutions for – the same is true of OSGi.
As Anthony Juckel put it on his blog, “Any sufficiently complicated Java program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of OSGi.“. This is quite true, in my experience, and can even be extended to include ESB’s, in that if you’ve got a number of JVM services interacting on a network to solve a problem, you’ve got some kind of an ESB going on, whether you call it that or not.
A common practice recently is to write software that provides “services”, then to combine these services into a complete system – in other words, we write services, but we compose systems (from these services). The awkward part comes (well, one awkward part) when we try to write the service in such a way that it can be used in many different ways – in other words, to maximize the potential for re-use – while at the same time trying to do the simplest thing that can possibly work.
No matter what service mechanism we choose, we lock ourselves in to some degree. If we write our service to use JMS (Java Messaging System), so we can support nice events and queues and other asynchronous goodness, we can’t easily then use our service from, let’s say, a client that looks for a REST service. If we choose SOAP, we can’t easily talk to our service from a client (which could be another service) that wants to post an event, instead of call a SOAP interface.
Normalized Messages The Java JBI (Java Business Integration) standard answers this problem: it provides a single message-oriented interface for services, called the Normalized Message. This Normalized Message contains some header info, some properties, and XML payload, and optionally, binary attachments. It is not specific to any one protocol – in other words, it’s not SOAP, it’s not REST, it’s not JMS, it’s not SMTP, it’s not any of those.
JBI is a JSR for defining Java Business Integration, a way to cut through the complexity of different types of communications mechanisms for component message-passing (e.g. things such as REST, JMS, SOAP, RPC, email, FTP, HTTP, file system, and so on).
ServiceMix is one of several choices – once we have services written to the JBI standard, we can deploy them in any JBI-compliant ESB. Glassfish’s OpenESB is another option, for example.
Instead of worrying about the plumbing of the specific protocol we want to use, we can concentrate on writing our business logic, then rely on the ESB to “wire up” the messages between our service-providing components – whether they’re local (e.g. running in the same ESB, in which case it’s a simple in-memory message), or distributed, on another box in our cluster, or halfway around the world. The ESB provides adapters for each of the different protocols – so we can take our one service and talk to it via REST, JMS, SOAP, Email, carrier pigeon (ok, maybe not the last one – but you could write an adapter!).
This means I can write my service without even knowing what kind of protocol I’m going to be talking to it with – maybe it receives some messages via JMS, some via REST calls, a couple of SOAP services, and one in a while via an email. All that is the problem of the ESB – I just write one simple method in one simple interface to swallow the appropriate NormalizedMessage, do my processing whatever it is, and spit back out another NormalizedMessage.
This can significantly speed up the development of new services, by taking decisions about the plumbing away from the process of writing the correct business logic. We don’t even need to decide up-front if our service will be used by other services or not – we can wire it up as required.
The other important thing about a Normalized Message is that it refers to services by an identifier, but not by a specific location – in other words, a message from a client might say “I need this service to process this message”, but it’s up to the bus to figure out how to get the message to the service – this provides the opportunity to decouple services from each other. Let’s say you need to write a service that takes some kind of business message and sends an email. You don’t have to worry about passing your service information to let it find the email service at all – you just send the message to a service name (maybe “email”), and the ESB figures out the rest.
This means you can swap the email service out for another service that provides the same mechanism without worrying about the details.
JBI and OSGi Together ServiceMix is one example of such an ESB, which marries the modularization advantages of OSGi with the above-described benefits of an ESB. This means each of our components can run in it’s own classpath space, with no interference with other components – even other components that might use different versions of the same jars. How many times have you discovered you’ve got 3 different versions of log4j on your classpath? This can lead to all sorts of weird and wonderful problems that only occur in certain circumstances, and are all-but-impossible to fix. By providing true modularization, OSGi solves this problem properly. Other solutions that I’ve seen applied in the past lead directly to the quote about every complex system having a half-baked OSGi implementation inside them
ServiceMix also provides a whack of existing components in two different JBI flavors: Service Engines and Binding Components. Services Engines are the ones that do the actual business logic, and Binding Components are the adapters, the pieces that route messages to and from various protocols. Many common tasks can be achieved by configuring the existing pieces, without writing a line of code.
When it is necessary to write some code, ServiceMix includes adapters to make that even easier as well – for instance, it provides a Service Engine to allow any POJO (plain old java object) to be used as a service, without that object having to actually implements the interface for handling NormalizedMessages directly. It doesn’t get any easier than that to write a service – just write the bean, plug in a bit of configuration, and you’re done – instance REST (JMS, email, SOAP, etc etc) service!
Spring integration is built into ServiceMix, so SE’s can have their dependencies injected automagically on deployment.
SE’s can be hot-deployed and hot-removed. Multiple versions of a single SE can be running without conflict, so “hot” upgrades can be done easily. You can have one version of a service that requires version 1.1 of another service running at the same time as some other service that requires 1.2 of the same service. This is flexibility in deployment, as it means that there’s no need to “drag along” other services when upgrading if you don’t want/need to.
Services can easily be managed and monitored via the administrative capabilities of the bus.
Binding Components Binding components are how we talk outside the backbone – they take messages from the backbone and transmit or receive via an external protocol to non-ESB services – e.g. via REST, JMS, HTTP, file system, etc etc. There are dozens of existing binding components to support just about every protocol we’d care about.
Of course, all this power comes with a price: you have to learn to manage and deploy your ESB of choice, and even the simplest ESB is still a fairly complex beast. ServiceMix is no exception. The good news, however, is that it’s possible to pretty much ignore much of the available complexity in order to get started small, then learn more as you need it to start to leverage the power available.
A caveat, however: It’s important to use an ESB the way it’s intended, and not try to shoehorn things that don’t fit into it’s structure. If you find yourself struggling to write services, or having to write a lot of Binding Components, chances are you’re making inappropriate design decisions, or that you haven’t separated the concerns of a Service Engine from a Binding Component fully. This is not uncommon, as in many environments these two concerns are handled by a single component. If you’re using to writing REST services, for instance, with things such as JRA or Jersey, you’ll find it very odd to separate the processing from the “presentation” (even when that presentation is simply into XML or JSON to the REST client).
Once this technique becomes second nature, however, the true power of an ESB becomes clearer.
A whole fleet of buses… Multiple ESB instances can be deployed and clustered, providing highly scalable and fault-tolerant system. The communication between ESB instances is handled entirely by the ESB itself – nothing about our components needs to be aware that they’re working in a clustered environment.
From my latest look at ServiceMix and it’s new release, I suspect I’ll be taking a deeper dive in the near future.
(Thanks to Craig Walls for pointing out that excellent quotation – and for all his help in understanding OSGi!)
If you ask most software teams, they’ll tell you that they would like to be producing high quality software. If you ask them if they are, they might hem and haw, but their answer will depend, more times than not, on subjective criteria – how they feel about the software their producing.
It’s important for good software teams to have pride in what they produce, no question, and most developers very much want to produce a quality product, but having the feeling that you’re producing quality is not enough. We need to actually establish that we are producing quality software. But how?
First we have to define what quality software is, as we can’t say yes or no if we don’t know what the question means. In this post I’m going to kick that definition around a bit, then in later posts we’ll talk about how that definition can be applied to our software to make objective measurements of quality.
Here’s where we run into our first bit of difficulty, as the definition of quality software is slightly different depending on who you ask. Everyone thinks they know what it is, but it’s surprisingly difficult to specify when you get asked.
Our best bet then, is to try to determine what the most common attributes of quality software are, the ones that most people would agree are part of a quality software system, or at least would agree have a significant impact on quality. Let’s list a few, keeping in mind that some of these factors are much more important to some situations that to others.
Requirement The best applications as far as quality must be said to begin with high-quality requirements. A topic unto itself, high-quality requirements must be descriptive, not prescriptive – that is, describe what is required in sufficient detail that it can be clearly understood, yet not say or imply exactly how that functionality is to be implemented, in order to leave the programmer with enough flexibility. A good requirement should also include clear criteria on how to verify if the desired functionality has been achieved – e.g. a good and consistent definition of “done”. There are many more attributes in this area, which we’ll explore in a future post.
Quality of design A design that fulfills the requirements and takes into account the other attributes that make up quality is often hard to evaluate. Most developers agree, however, that quality software must start with a quality design. Again, this is a large topic that we’ll explore in more detail at another time.
Quality of conformance to the design Something a little easier to quantify is the actual conformance to the design of the system that implements it. Does the code do what the design said it ought to do?
Reliability Software that continues to operate the same way despite the passage of time and the variation of environment shows the trait of reliability. If the software is deterministic, e.g. it responds the same way to the same inputs every time (unless of course it’s not supposed to – like a game with a random element), then it is showing some of the attributes of reliability. If it’s a service, then it’s uptime can be used as a gauge of reliability. Reliable software is considered to be of higher quality than unreliable software.
Maintainability How difficult is it to maintain (e.g. fix and/or improve) the software? Various things about how the software is structured, what language it is written in, how it’s modularized, and many other factors can help give us an idea as to the software’s maintainability. Easier maintainability is a widely accepted attribute of quality software.
Completeness Quality software is complete: it not only implements every part of the design, it includes the necessary pieces the allow it to stand either without a lot of supporting systems, or at least with a small well-defined set of such systems. Software that stops short of the complete design can’t be said to have this aspect of quality.
Verbosity The verbosity, or rather the lack of it (e.g. the terseness) of the actual source code can be a contributing factor to quality, at the correct level. Too terse and we can have obscure and unreadable source, too verbose and we’ve got the same. Where the sweet spot is depends a lot on the programming language being used, but it’s often an aspect of quality that directly affects readability and maintainability.
Portability Although not an attribute in some cases as far as portability from one environment to another, it’s generally considered a good thing if software is not overy dependent on it’s environment – and this is also an indicator of portability. Higher quality software will be able to be used in a variety of different environments more easily than lower quality software.
Consistency An overall consistency of both design and implementation is certainly related to quality – if we have three pages in our UI design, for instance, they should share some common attributes.
Testability/verifiability The ability to test software is a critical quality metric. If it’s easy to test, it’s almost certainly of higher quality than if it’s hard to test. Software that can be verified as correct is even better, although somewhat rare and generally requiring a functional programming environment.
Usability Usability is an aspect of quality that is often the hardest to quantify, although certainly not impossible. It doesn’t only refer to software that’s intended to have a user-facing portion, but may actually apply to software designed as a web service or utility – the usability in this case is more a function of how easily other systems and programs can connect to it and consume it’s services.
Efficiency Although related to performance to some degree, quality in efficiency is more concerned with making the best user of what resources are available, rather than the overall speed of processing. You might measure aspects of efficiency by looking at memory footprint, CPU utilization, and so forth. A higher degree of efficiency would usually translate into a lower percentage consumed of the available resources.
Security Although the requirement for this type of quality can vary, it is generally true that software that takes into account security and is by default secure is considered of higher quality than software that is insecure. Witness OSV and Windows, for example (donning flame-proof suit)
Coupling An aspect of quality closely related to usability in many cases is the degree of coupling, sometimes also called the degree of dependance. If a software application or service requires a lot of other services in order to do it’s job, it often is exhibiting a lower quality in terms of coupling compared to a system that has few dependencies, as the system with more dependencies can be harder to install, deploy and maintain. In a SOA environment, of course, a service can reasonably expect to depend on other services, but as few as possible is generally best.
How the coupling is handled in terms of packaging and distribution can also be an important quality aspect.
Composability This aspect is best demonstrated by the traditional Unix command-line utilities, such as grep, sed, pwd, and so forth. They do one thing, and do it simply but well – but they are eminently composable, being designed to be strung together to handle more complex tasks. If a service or application exhibits high composability, it is easily able to be fit into a series or of such services to perform more complex tasks than any one of them can handle alone.
A high-quality component is often one that can stand alone and perform a useful task, but also be combined with other packages as required easily and quickly.
Ensuring that all of the criteria (or at least all the ones we care about in our situation) usually requires a certain level of quality and consistency to our process, as well, but that’s another whole discussion.
Apache Maven is a lot more than a “build tool”, and one of it’s major strengths is it’s ability to manage dependencies.
Maven’s not just for external dependency management, though – it can help us work faster and more easily with our own modules as well as those written by others. In fact, it’s “internal” dependency management is actually far more powerful for most development shops.
Every dependency Maven manages is identified with 3 pieces of information – it’s group id, it’s artifact id, and it’s version. Group id is often some sub-domain of the company it’s working on, e.g. com.point2.somemodule, and the artifact id helps identify the specific module with that group, like rest-api or such.
Possibly the most interesting part is the version number, though, as this is where the real power of Maven comes to the fore. Versions allow us to maximize the opportunity for parallel development without descending into unversioned chaos. Each version represents a specific point in time in a library’s development – and, most importantly, allows us to “re-assemble” our application to a known state at any time (not re-build it).
Let’s take a for-instance to see how this might work…
Component-Based Application “Assembly” For example, let’s say I’ve got a few teams working on different modules for my new application, let’s call them “persistence”, “rest-api” and the user-interface, “ui”. Each of these modules depends on a set of common utility classes in “util”.
We can represent this through a set of triples like so:
rest-api depends-on persistence rest-api depends-on util ui depends-on rest-api persistence depends-on util (directly, and not only on the transitive dependency through persistence)
The unseen aspect here is the versioning. If we include versions in our triples, we see the picture is a bit more sophisticated:
rest-api-1.0 depends on persistence-3.1 rest-api-1.0 depends-on util-1.1 ui-1.0 depends-on rest-api-1.0 persistence-3.1 depends-on util-1.0
Now we have a fully defined dependency graph that we can assemble into an application, say app-1.0. At any time, if we want a copy of the app in 1.0 state, we re-construct it from this deployed modules, no need to build any source code, and we’ve got the exact same app, every time.
Get it in motion… Now let’s look at this in a dynamic development environment, where we’re trying to maximize sustainable velocity:
Although there are dependencies between each module, we don’t want to hold up one team by forcing them to build the other teams modules unnecessarily. We also want each team to choose if they want to work with the very latest version of the other modules, or working against a fixed and stable version for a time instead.
The “ui” team, for example, might be refactoring JavaScript code that’s relying on version 1.0.3 of “rest-api”, while “rest-api” in turn is already working on 1.0.4 – and it uses 1.1.0 of “persistence”… it can get tangled in a hurry without a way to manage it, and we don’t want to be artificially discouraged from writing modular code just because it’s hard to keep version numbers straight.
Enter Maven again. Instead of forcing everyone to just always work with the latest version of every other module (which can bring productivity to a screeching halt in some situations), we allow each time to decide what dependency they will include in their POM (Project Object Model) file.
What if I want the very latest version of “persistence” while I work on “rest-api”, with changes checked in by other developers while I’m still working? This is where the SNAPSHOT version comes into play. Instead of declaring a dependency on 1.1.0, I declare a dependency on 1.1.1-SNAPSHOT. This represents the latest “edge” code for the referenced dependency.
Now we have a graph that looks like this:
app-1.0-SNAPSHOT depends on ui-1.1-SNAPSHOT rest-api-1.1-SNAPSHOT depends on persistence-3.1-SNAPSHOT rest-api-1.1-SNAPSHOT depends-on util-1.1 ui-1.1-SNAPSHOT depends-on rest-api-1.1-SNAPSHOT persistence-3.1-SNAPSHOT depends-on util-1.1
As you can see, we have a mix of stable versioned modules (util in this case), and “on the fly” versions. Yet at the same time we’re assured that major changes that break backwards compatibility will not be seen, as we indicate such changes with a change in our major version number (e.g. 1.X to 2.0).
Then we can set up a CI job (say on TeamCity, Bamboo, or whatever your CI system of choice is) to automatically build and deploy our SNAPSHOT version of “persistence” to our local Maven repository (within our company firewall). The SNAPSHOT version actually turns into a date/timestamped version when it’s deployed to Nexus, and Maven is clever enough to fetch for us the most recent of these SNAPSHOTs every time we build. The “persistence” team checks in some code, CI builds it and deploys the resulting SNAPSHOT jar to our repository, and we get it automagically the next time we build, even though we’re working on rest-api, not persistence.
When we’re ready to “stabilize” our dependencies, we simply switch from the SNAPSHOT to a specific version. Maven has a pre-defined “release” process that guarantees, among other things, that every released version has no remaining SNAPSHOT dependencies, is tagged to version control, and verified via all it’s tests. More than a build tool indeed…
We could of course just put all the modules we’re going to depend on in an aggregator POM, and build everything every time we make a change, but this is hardly efficient, and limits our development velocity unnecessarily (and of course we might not all be in the same source tree, or even the same version-control repository). We want to be building smaller pieces, not bigger ones.
A critical part of this process is our company-local Maven repository – here I mean not just the developer-local repository on each developers own workstation, but a product like Nexus that holds a company-wide copy of all required jars for a build. By doing this, we can guarantee a consistent copy of all our required dependencies without having to depend on the availability of outside repositories, such as ibiblio. It’s not a bad idea to in fact *only* permit access to the local repository for building releases, which ensures this policy is not violated accidentally – while at the same time keeping the “external” maven repo’s available to developers for experimenting and prototyping. Once something gets used in production code, however, it gets stored in the “inside the firewall” Nexus repo (and backed up from there). This avoids the bad practice of checking jar files into source control (it’s called “source” control for a reason).
Testing, Testing… To add a new aspect to the problem, let’s say that it’s not only production code we depend on, but helper classes for tests as well. If it’s difficult to set up a fixture for a certain kind of test, that might be a code smell in and of itself, but that’s also another story. If we have some test helpers that reside in our dependent modules, we won’t be able to see those helpers in our tests in another module, as we’re only depending on that module’s production code, not test code.
We can easily tell Maven to also bundle up the test code from a certain module, however, and make it available to us in a jar file, like so:
<build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <executions> <execution> <goals> <goal>test-jar</goal> </goals> </execution> </executions> </plugin> ... </plugins> </build>
Now when we build, we’ll get a test jar as well as our regular jar, which we can depend on like so:
<dependency> <groupId>com.point2.core</groupId> <artifactId>somemodule</artifactId> <version>1.0</version> <scope>test</scope> <classifier>tests</classifier> </dependency>
Now our test classes in the module declaring the above dependency can see the test helpers in the somemodule module – but we’re still not including test code in our production jar.
Again, I have to emphasize that this level of coupling might indicate a deeper issue, but if you do need to do this, it’s good to know how
Maven also includes facilities to analyze and clean up a complex dependency tree, remove unnecessary dependencies, and keep the whole project manageable.
In summary, Maven can handle extremely complex dependency management for us in a fully declarative and versioned manner, allowing us at any moment to see exactly what our project depends on, both in production and test code. In conjunction with a CI system (like TeamCity) and repository server (like Nexus), we can automate the deployment of intermediary and full-release versions to the point where we save significant time, and never build code that we’re not actually working on, allowing us to concentrate on the task at hand and leaving the heavy lifting to Maven.
This allows us to only ever build the code we’re actually changing – never code that’s already available in another library, reducing our developer cycle time significantly. It also means we’re spending more time “assembling” software from re-usable components than re-compiling (and probably re-testing) code that’s already verified and available in object form.
Maven: not just for breakfast anymore.
