One of the ways to ensure quality in a software project is to find a set of coding standards that your team can agree on then put automated checks in place to ensure they are adhered to.
In this post I’d like to take a very small example of such a stanard, and show how you can use several different tools to help ensure and measure compliance.
The Example The simple rule we wil use is the use of curl braces on if statements and similar blocks. Our team reached a concensus that we would always prefer this form
if (condition) { block }
over this
if (condition) block
even when there is only one line in the block, where the braces are optional. I’m in the habit of not doing this, so I kept doing it the wrong way, making code just that little bit harder for the rest of the team to read and maintain.
We have an existing codebase of hundreds of thousands of lines, some of which adhere to this rule, some of which doesn’t.
First, I wanted to measure adherence to the rule, so I chose the Sonar quality assurance tool.
Sonar Sonar installation is straightforward: download the distribution, unzip it and run one script. Sonar runs as a web application, so you can immediate go to your browser and see the initial page, like so:
I’m using Sonar (for now) in it’s default mode with a bundled Derby database. If I were going to use it more heavily, I could install a separate MySQL database, and run Sonar itself inside a container of my choice (it comes bundled with a basic web container so you can get going right away).
For my machine, I ran
bin/macosx-universal-64/sonar.sh console
“Console” runs Sonar without backgrounding the task. Normally you’d use “start”, which works just like a Unix service, and is suitable for running from an init.d-type script immediately. The first time around, Sonar takes a few moments to fire up, as it creates all it’s required database tables and populates them with an initial set of data to get your started.
Sonar can measure many hundreds of different metrics, but for the purposes of this simple experiment I only want the one, so I create a custom profile, called “team”.
I set this profile to be the default, then click to edit it.
Now I can search for the braces rule in checkstyle by setting filter criteria:
You can click the rule name to see a description, and to set any optional parameters on the rule.
I activate the rule, and leave it at the default “Minor” priority.
If I now return to my profile, I have a link to access the generated checkstyle.xml file, the configuration file that is used by checkstyle to configure its rules. This will be important later on.
Now I can go to my project and analyze it by running “mvn clean install sonar:sonar”. With the default configuration, this works only when the build is taking place on the same box that Sonar is running, but it’s simple to set up a settings.xml file to tell the Sonar plugin where to find the Sonar instance, so you can have a single shared Sonar instance for the whole team. This is covered nicely in the Sonar documentation.
One minor hiccup: The released version of the Sonar plugin for Maven doesn’t work with Maven version 3 (still in Beta), which I was using. You can build the source version if you need this support, or I’m guessing it’ll be released about the time Maven 3 might be.
Now that I’ve analyzed my project, I can go back into Sonar and see how well I adhered to my rule:
As I see I have a violation, I can drill down and see the violation in context right in the code:
This is handy for checking the state of a rule against an existing codebase, but I’d like to have more immediate feedback. No problem, I can integrate Sonar into both my IDE and my build process very easily.
IntelliJ IDEA: Immediate Feedback
There’s also a new plugin for IntelliJ IDEA for Sonar. To install, just go to the “Available” tab in your plugin configuration within IntelliJ, right-click and select Download and Install. You’ll have to restart IntelliJ to activate the plugin.
Now you can find the Sonar entry in configuration, and set up a URL to point to your Sonar installation, along with a username and password to log in to Sonar. The default is “admin/admin”.
Having done that, I can now see violations directly in my code, as I open each file, like so:
In the above, I’m hovering over the little “price tag” annotation in the left gutter to see the message. You can also see the usual IntelliJ “yellow bar” on the right side of the screen, which shows the same message.
This lets me know of a rule violation before I even attempt to build, so when I forget my braces, I don’t have to wait to be chastized, IntelliJ can do it for me immediately
Maven: Ensuring Conformance
There is a Maven Checkstyle Plugin that allows us to make a run of checkstyle part of the build process of our project. I configured this plugin in my POM like so:
4.0.0 com.jglobal experiment jar Experiment 1.0.0-SNAPSHOT UTF-8 junit junit 4.5 test maven-compiler-plugin 1.6 1.6 org.apache.maven.plugins maven-checkstyle-plugin 2.5 org.codehaus.plexus plexus-resources 1.0-alpha-7 http://localhost:9000/rules_configuration/export/java/team/checkstyle.xml true true org.codehaus.sonar sonar-maven-plugin 1.8M1 codehaus Maven Plugin Repository http://repository.codehaus.org/ default false never
As you can see, instead of providing a file system based location for the configuration file for Checkstyle, I just pointed the Checkstyle plugin at my Sonar-generated checkstyle.xml, this way my IDE, Sonar and my build stay in perfect sync, and I have a nice GUI for adjusting my checkstyle rules, instead of having to hand-edit XML.
Now when I run my build, Maven will analyze my project against the checkstyle rules, and report any violations. With code as we’ve seen above, it of course reports a violation, but the build still succeeds.
This is fine, but now that I’ve verified we are adhering to this rule, I’d like to actually fail the build if we slip back. Also easily done – in Sonar, I just change the level of violation for my one and only rule to “Critical” in my profile. Now when I run my Maven build, sure enough, the build fails, noting the fact that I’m not adhering to a Critical rule.
As my codebase gets cleaner and cleaner, I can up the priority of the rules I really care about until there are no more violations – then Maven and Sonar can guarantee it stays that way. Of course, in a multimodule build (e.g. with an aggregator POM), I might want to create a parent POM, and specify all this configuration there. (It’s generally not a good idea to have the aggregator and the parent in one pom, use two).
This is of course a trivial example of Sonar, IntelliJ and Maven working together. Sonar can apply over 700 rules right out of the box, and more can be added via plugins easily.
What I find good is that I can synchronize all my tools off one common and agreed-upon rulebase, without having to tweak XML files all over the place to gradually increase my quality standards over time.
If you ask most software teams, they’ll tell you that they would like to be producing high quality software. If you ask them if they are, they might hem and haw, but their answer will depend, more times than not, on subjective criteria – how they feel about the software their producing.
It’s important for good software teams to have pride in what they produce, no question, and most developers very much want to produce a quality product, but having the feeling that you’re producing quality is not enough. We need to actually establish that we are producing quality software. But how?
First we have to define what quality software is, as we can’t say yes or no if we don’t know what the question means. In this post I’m going to kick that definition around a bit, then in later posts we’ll talk about how that definition can be applied to our software to make objective measurements of quality.
Here’s where we run into our first bit of difficulty, as the definition of quality software is slightly different depending on who you ask. Everyone thinks they know what it is, but it’s surprisingly difficult to specify when you get asked.
Our best bet then, is to try to determine what the most common attributes of quality software are, the ones that most people would agree are part of a quality software system, or at least would agree have a significant impact on quality. Let’s list a few, keeping in mind that some of these factors are much more important to some situations that to others.
Requirement The best applications as far as quality must be said to begin with high-quality requirements. A topic unto itself, high-quality requirements must be descriptive, not prescriptive – that is, describe what is required in sufficient detail that it can be clearly understood, yet not say or imply exactly how that functionality is to be implemented, in order to leave the programmer with enough flexibility. A good requirement should also include clear criteria on how to verify if the desired functionality has been achieved – e.g. a good and consistent definition of “done”. There are many more attributes in this area, which we’ll explore in a future post.
Quality of design A design that fulfills the requirements and takes into account the other attributes that make up quality is often hard to evaluate. Most developers agree, however, that quality software must start with a quality design. Again, this is a large topic that we’ll explore in more detail at another time.
Quality of conformance to the design Something a little easier to quantify is the actual conformance to the design of the system that implements it. Does the code do what the design said it ought to do?
Reliability Software that continues to operate the same way despite the passage of time and the variation of environment shows the trait of reliability. If the software is deterministic, e.g. it responds the same way to the same inputs every time (unless of course it’s not supposed to – like a game with a random element), then it is showing some of the attributes of reliability. If it’s a service, then it’s uptime can be used as a gauge of reliability. Reliable software is considered to be of higher quality than unreliable software.
Maintainability How difficult is it to maintain (e.g. fix and/or improve) the software? Various things about how the software is structured, what language it is written in, how it’s modularized, and many other factors can help give us an idea as to the software’s maintainability. Easier maintainability is a widely accepted attribute of quality software.
Completeness Quality software is complete: it not only implements every part of the design, it includes the necessary pieces the allow it to stand either without a lot of supporting systems, or at least with a small well-defined set of such systems. Software that stops short of the complete design can’t be said to have this aspect of quality.
Verbosity The verbosity, or rather the lack of it (e.g. the terseness) of the actual source code can be a contributing factor to quality, at the correct level. Too terse and we can have obscure and unreadable source, too verbose and we’ve got the same. Where the sweet spot is depends a lot on the programming language being used, but it’s often an aspect of quality that directly affects readability and maintainability.
Portability Although not an attribute in some cases as far as portability from one environment to another, it’s generally considered a good thing if software is not overy dependent on it’s environment – and this is also an indicator of portability. Higher quality software will be able to be used in a variety of different environments more easily than lower quality software.
Consistency An overall consistency of both design and implementation is certainly related to quality – if we have three pages in our UI design, for instance, they should share some common attributes.
Testability/verifiability The ability to test software is a critical quality metric. If it’s easy to test, it’s almost certainly of higher quality than if it’s hard to test. Software that can be verified as correct is even better, although somewhat rare and generally requiring a functional programming environment.
Usability Usability is an aspect of quality that is often the hardest to quantify, although certainly not impossible. It doesn’t only refer to software that’s intended to have a user-facing portion, but may actually apply to software designed as a web service or utility – the usability in this case is more a function of how easily other systems and programs can connect to it and consume it’s services.
Efficiency Although related to performance to some degree, quality in efficiency is more concerned with making the best user of what resources are available, rather than the overall speed of processing. You might measure aspects of efficiency by looking at memory footprint, CPU utilization, and so forth. A higher degree of efficiency would usually translate into a lower percentage consumed of the available resources.
Security Although the requirement for this type of quality can vary, it is generally true that software that takes into account security and is by default secure is considered of higher quality than software that is insecure. Witness OSV and Windows, for example (donning flame-proof suit)
Coupling An aspect of quality closely related to usability in many cases is the degree of coupling, sometimes also called the degree of dependance. If a software application or service requires a lot of other services in order to do it’s job, it often is exhibiting a lower quality in terms of coupling compared to a system that has few dependencies, as the system with more dependencies can be harder to install, deploy and maintain. In a SOA environment, of course, a service can reasonably expect to depend on other services, but as few as possible is generally best.
How the coupling is handled in terms of packaging and distribution can also be an important quality aspect.
Composability This aspect is best demonstrated by the traditional Unix command-line utilities, such as grep, sed, pwd, and so forth. They do one thing, and do it simply but well – but they are eminently composable, being designed to be strung together to handle more complex tasks. If a service or application exhibits high composability, it is easily able to be fit into a series or of such services to perform more complex tasks than any one of them can handle alone.
A high-quality component is often one that can stand alone and perform a useful task, but also be combined with other packages as required easily and quickly.
Ensuring that all of the criteria (or at least all the ones we care about in our situation) usually requires a certain level of quality and consistency to our process, as well, but that’s another whole discussion.
Why do we want to go fast in the first place? Well, if we’re not accumulating technical debt, our quality is still within the bounds we’ve set for ourselves, and we’re satisfying user stories, then we want to be able to accomplish as much as we sustainably can each sprint. This way we can deliver value faster, and people tend to like to pay for that kind of thing
Well, first, we need a road: the basics of an agile environment need to exist before we can go very fast at all. If we’re working on the build system every sprint and trying to get the basics of a story understood, we’re still in road construction, and we shouldn’t expect much in the way of speed until we get some of this basic asphalt laid down and smooth. These includes such basics as a good development setup for our team, a basic understanding of agile principles and a grasp of the technology stack we’re using, and the support of a continuous integration server, to mention a few. If we’re attempting to bounce along over the potholes without setting up a proper environment for rapid delivery of software value, we’ll reap what we sow.
Assuming we’ve built the road, then, what things tend to hold us back? Just like on a real road, the only things stopping us from going faster and faster (to the mechanical limit of our vehicle, or in our case, our keyboards and brains) are either externally-imposed limitations (e.g. a speed limit and cops to enforce it), or our own ability to control the pace without going off the road. In software construction, as in life, we can go off the road in many varied ways, but they all tend to be spectacular, destructive, and painful. Unlike the real road, we can be cruising along for some time before we discover we’ve left the asphalt behind and are sailing over a cliff.
We’ll start by assuming that our corporate environment has eliminated externally imposed speed limits and political roadblocks – not always a safe assumption, but lets assume for the moment that we’re one of the luck developers who work in such a situation.
Our top-level speedometer, to overuse our analogy a bit, is our velocity, measured in features per iteration, or complexity points per iteration – in other words, how much business value are we adding per time period?
The most common way to go off the road is for quality to slip. This can be detected in one of a number of ways, including an ever-increasing defect rate. If most of your sprint is taken up by fixing defects or paying off code debt, then you’re probably trying to go too fast (or you’ve not finished laying the road after all). Of course, it’s possible you’ve just got a few bad drivers on your team, but we’ll assume that’s easier to see (if not necessarily easier to fix). What’s worse than seeing quality slip? Not seeing quality slip, even though it is. We can’t measure quality directly, per se, but we sure can measure a lot of other things. Once we know what the normal position of each guage is (e.g. once we establish reasonable code standards that we can measure), then we can watch them to get early warning of things going awry.
Just like on the real road, we need two categories of things, it seems, to help us go as fast as safely possible: I’ll call them headlights and guardrails.
Headlights
The most basic tools here are user stories, acceptance criteria/tests (ideally executable ones), and metrics such as defect rate and velocity measurements. None of these are trivial or straightforward, and it’s easy to think you’ve got a good view and suddenly discover you’ve been accumulating code debt without realizing it. The only proper reaction at that point is to slow down and correct the problem, as we’ll discuss below.
A good business understanding of the goals and epics behind our user stories gives us more range to see further ahead, and going fast requires looking further ahead, while at the same time paying attention to where you are at the moment.
Just like when driving we must be aware of the road immediately ahead, our user stories give us the close-focus we need to be doing the immediately useful thing. We can’t discard these in favor of looking further ahead exclusively, or we’ll never get to where we want, but we can combine that with an awareness of both the near future and an understanding of the overall destination to make better decisions in our day-to-day work.
If we concentrate exclusively on the user stories in hand for each iteration we can find we’ve lost sight of the forest, and may have a hard time fitting together features that should blend into an overall product. If we concentrate only on the distant horizon and not on the user story we’re working on we’ll never get anything done. The proper balance lets us go fast.
We don’t want to be like the driver in the joke with the punch line that ends “we’re lost… but we’re making bloody good time”!
Looking a bit further ahead also allows us to anticipate curves and obstacles in the road, and be ready to hand them when they arrive. If we know, for instance, from our long-range planning that we intend to scale our application to thousands of users, we might make different decisions than if we’re aware that a single user on a desktop box is the intended audience – even though neither of these factors is really represented directly by each user story we work on.
Executable acceptance tests from a tool like Greenpepper, Fitness, RSpec, or the like can be valuable headlights, freeing developer time from the repetitive manual verification and allowing BA/Customer Proxies to have control over the acceptance process – again freeing up developers to develop, and maximizing team velocity. As was mentioned in a recent stand-up meeting: if you’ve manually tested once, you’ve probably already spent more time than it takes to set up an automated test to do the same thing repeatedly, not to mention you’ve probably enjoyed it a lot less
Guardrails
There’s a big difference between guardrails and a stone wall built across the road ahead, however – it’s not hard to let a testing tool or technique turn into a straightjacket, with tons of brittle and hard-to-maintain tests that don’t help us at all. We need the right tool for the right job, and used the right way.
If we have guardrails ensuring the basics of our code quality, we can go faster with the confidence that when we look back at the end of each sprint we will not have accumulated more code debt that needs to be paid back later. For example: if we establish a test coverage metric that ensures we have a breaking build if our code coverage goes below a certain minimum level (I propose this always be 100%, but that’s another post), we can move forward with the assurance that there’s no code that’s being left untested, so we won’t find ourselves in the distinctly non-TDD-like position of having to go back and write tests for existing code, burning time that should be able to be used for the next story.
We can also refactor with better confidence if we know for a fact there are tests watching over our shoulders, ready to break should our refactor not be true. Refactoring code that is, at least in part, untested should always be an unacceptable risk.
If we have some checkstyle, PMD, FindBugs or other static analysis tools checking that our cyclomatic complexity is within bounds, that our class size and line length are readable, and other critical maintainability and coding standards factors are met, we can plunge forward without the fear of a huge cleanup being required just to make the code understandable down the road a ways.
Of course, just like guardrails and headlights are not infallible in the real world, all the tools and checks in the world don’t ensure good quality code. One area that’s particular hard to ensure quality within via automatic mechanisms is design. You can have code that’s 100% covered, passes every checkstyle rule known to man, and still represents a terrible design. This is where the human factor comes into play – the automation merely ensures that you’re spending valuable human attention span on the stuff that really requires a brain, as opposed to things that can be verified mechanically.
Discipline is the glue that makes all of this work together – often times developers themselves will have the “smell” of something done not quite right, but not feel like they’ve got the latitude to dig into it and clean it up, so they save it until the mythical “later”, which sometimes never comes. Management and team leads must also be disciplined enough to have the patience while that kind of refactor happens – with the firm knowledge that they’ll get paid back by better productity and a lower defect rate over the mid to long term.
A final warning: It’s easy to let headlights become leashes and for guardrails become cubicle walls. Many agile practitioners are concerned, and rightly so, that adding tools and techniques can turn into a new dogmatism and inflexible methodologies. It’s up to us in the trenches to make sure we don’t let this happen, while at the same time getting all the juice we can out of helpful techniques and tools.
Properly applied, though, headlights and guardrails can be valuable tools in letting us reach our maximum velocity, while still arriving safely at our destination.
