It turned out this was a longer road than I had anticipated, but reviewing my reasoning was helpful, so I’m finally providing some of the reading material related to it here, as well as my own conclusions for consideration.

First, it’s good to understand the differences between stubs and mocks (and other kinds of test fakes). Martin Fowler explains this quite precisely in this article.

Sub-call verification (e.g. orchestration testing) is an anti pattern and impedes refactoring. Mocks use behaviour verification at a low level, and encouraging testing it.

Another description of the difference, with a slightly different take is in this article.

Another analysis of the difference, and the anti patterns imposed by mocking is given in this article

A succinct description of the coupling issue is discussed here

In addition to the issues described above, there are two pragmatic issues that affect my preference for mocking actually depend more on my current language of choice: Scala. One issue is that the most capable mocking framework I know of, Mockito, doesn’t play well with Scala – it can’t, for example, properly mock methods defined in a trait. As a result, because I TDD, I’m tempting to not use some of the more powerful features of Scala just because my mocking framework of current choice doesn’t handle them well. Of course, new frameworks coming along, like ScalaMock, might make this argument moot. When I found myself creating behaviour-less traits just to provide easier mocking in my tests, I realized there was a problem.

The other issue comes from the fact that it’s so darned easy to do stubs in Scala – given the low-ceremony ability to override and create closures and dynamic instances, it’s been my experience that mocks are actually more code and less comprehensible in Scala. This is not at all the case in Java, however, where mocks can be massively more succinct than stubs.

I subscribe to the original intent of object-orientation, which is that objects are encapsulations of state and behaviour, not data structures, and that the way to communicate between them is messages, not method calls. As a result, I find the verification of behaviour at the low level counterproductive. I prefer to verify the behaviour of objects (services, domain objects, etc) at a high level – e.g. does this thing DO what I want it to do, as opposed to asking “does it do the thing I’m testing the WAY I think it does”. These are two very different questions.

At the same time, I have a strong preference for functional programming, which also supports easier testing, making the issues of mocking less critical, as the stubs I need to test more fully functional code are trivial compared to what highly procedural code using mutable objects requires.

After all, why do we test? This is a larger question, and a topic for another blog post, but for me, the answer is two-fold: to give me confidence that the system does what I specify it to do, with some level of confidence, and to allow me to refactor freely and frequently while maintaining that confidence. As a result, the last thing I want to do is to ensure the system does what it does the same way it did it a minute ago – I don’t care how it does the thing it does, only that it does. In fact, I must not care about the how, or I can’t refactor without re-writing large hunks of my tests.

This preference to thinking about the what instead of the how is also, incidentally, what causes my preference for using Guice when managing services and dependency injection: because Guice allows me to quickly add new dependencies and change constructor signatures without having to go back and re-write tests, it allows me to go faster than manual module creation does, so I prefer it.

Whether I have annotations telling me my code is injected or a module class somewhere doing the injection manually for me is less interesting, although at least the annotations tell me in the class I’m looking at that dependency injection is taking place.

In any event, that preference is a side issue to the mocking/stubbing preference.

I hope this material provides some food for thought.

As many in the industry have observed, software is getting (arguably, already is) incredibly important to the normal functioning of our daily lives, and it appears this trend is likely to continue for some time. This alone, even if not for the many other reasons, is sufficient to take software craftsmanship seriously, and to explore it’s meaning and it’s disciplines.

I’ve had the good fortune to make my living in the software development industry for almost three decades (it’ll be 30 years this June, if I remember correctly), and in that time I’ve seen what I would call good craftsmanship, and I’ve seen what I will charitably refer to as – well – “not” good craftsmanship, and I’ve tried to pay attention to the differences between them. I’ve seen some technologies and techniques come, then go, then in some cases come again, a couple even for the third time. As has been many times observed, if we do not study history, however brief, we are doomed to repeat it – for better or worse.

I’ve seen what works, and what doesn’t, and I’ve thought and talked about why, and I’ve decided to try to summarize those experiences into an exploration of the subject, concentrating on the underlying principles or beliefs of good software craftsmen and the practices that usually derive from those principles.

I’m going to summarize my musings here in the blog, but I’m also planning on writing a full-length book on the subject. Tentative title “Software Craftsmanship: Principles and Practices”. I’m not sure yet if it’ll be only an eBook or if dead trees will be involved, but that’s just the details of delivery.

Any work of this nature is going to be far more valuable to other developers if it includes as many perspectives as practical, so I welcome input, comments, collaboration, discussion, dissent, digression, debate, disbelief and all that other stuff as well.

My audience for this book will be the aspiring software craftsman. Software is in the unfortunate position of being such a new discipline that there’s not a large body of knowledge easily grasped by the apprentice that’s not specific to an area of technology. To use a medical analogy, we’ve got books on treating broken fingers aplenty, but not that many on the general experience of being a doctor. Of course, if we imagine a world where the idea of doctors has only been around for fifty years or so, this lack is more understandable. Still, how can we develop more good software craftsmen (and craftswomen, of course), if those of us that have been kicking around for a while don’t record and share our experiences?

In any event, please expect a torrent of thinking on this topic to be poured out here starting soon, and please drop me a line if you’d like to get involved in the process. You’ve been warned!

This has made me want to once again rant and rave about the many negative attributes of this pattern. Instead, I’m going to slow down and try to enumerate them more clearly

First, let’s examine what pattern we’re talking about: In it’s most common form, it is the pattern of assigning an incrementing numeric value as the synthetic key to a persistent entity of some sort, often in a relational database. One of the reasons it’s often found in relational databases is that it’s easy: many databases provide an automatic facility for assigning incrementing numeric keys to a table.

I won’t even go near the debate of synthetic keys vs. natural keys for persistence, so I won’t count that for or against incrementing numeric keys.

Can’t be easily distributed
A huge flaw in incrementing numeric keys is that their assignment is, by definition, not distributable without taking special steps. Even if the database in question is assigning the keys for you, all new records must be inserted into the same instance of the database to ensure duplicates are not assigned.

Incrementing numeric keys also don’t shard well, as their distribution is completely predictable, causing some problems with large datasets that are avoided with other more easily hashable types of keys.

Single point of contention
Whether it be a database or the infamous “next number table”, the place where the key is assigned with an incrementing key is a point of contention, something any scalable system wants to avoid at all costs.

Implies order, but doesn’t make it explicit
An incrementing key implies sequence or order, but does not explicitly define order. That is, if you sort the entries by the key, they are (supposedly) in order of insertion. The key isn’t meant for this, however, and it is far better to use an actual meaningful order, such as a created date/time stamp or other sequence that actually has a purpose in the domain model.

Temptation to assign meaning
The most insidious problem with an incrementing numeric key is that it is often too tempting to assign meaning to the key, where none actually exists. If the key is supposed to be a meaningless unique identifier, then use a meaningless unique identifier, not an easily recognized incrementing value that will tempt users of the data (if you expose the key – arguably a second problem) to infer “ah, this number is greater than 1000, therefore this record was inserted after June the 12th”, or, worse, setting a new starting point in the number for a certain class of data “keys greater than 1000000 are created by customers, less than 1000000 are created by the system”. I’ve seen every kind of crazy variation of this kind of thing, and they’re all trouble, and none of them could have happened with a nice incomprehensible GUID instead of an incrementing numeric id.

Fixed Size
While it is entirely possible to set an incrementing numeric key to a reasonable size, allow for all possible expansion, often this is not done. This will frequently lead to that horrible moment in the future where you realize the data set is going to scale much larger than was anticipated, and the key must be resized. Often this happens as a result of a bizarre tendency to want to “save space” by trading off a couple of bytes.

To my mind, any one of these disadvantages is enough to avoid the trap of the incrementing numeric key, but all of them put together add up to a pattern I’d most certainly avoid unless there were highly compelling reasons for it, and “because it’s easy” is not a compelling enough reason.

Alternatives
Not one to complain of a problem without presenting some possible solutions, I thought I’d quickly mention some of the alternatives to the numeric incrementing key:

Natural Key
One way to avoid a synthetic key using a numeric incrementing value is to avoid a synthetic key altogether. If you use a naturally unique value from the data in question as the key, the entire problem goes away – although, arguably, other problems can arise, but these are outside the scope of this post.

GUID or ObjectId
My favorite solution to the key issue, if a natural key is for some reason not appropriate, is to use a randomly-generated GUID. In the bson libraries supplied with MongoDB’s driver, the ObjectId value is such a GUID, and generates a 24-character random hexidecimal key. The chance of a collision is vanishingly small, and this value makes an excellent key for both key-value and relational databases.

Java’s own libraries also provides a GUID-generator, also suitable for the same purpose.

Conclusion
I hope this article gives you a bit of food for thought about numeric incrementing keys, and I encourage you to consider the available alternatives next time this issue comes up in your design work.

In the last many years, I’ve built systems that consist of a set of services. These services are used by a web UI and various other client applications to interact with users, load data, process that data in various ways, and export some of that data to other systems.

A fundamental goal of using a service-oriented architecture is to keep the cost of new changes as low as possible. That’s not the same as going as fast as possible, but it does often result in a high and sustainable pace of change as well. The advantage of this is that it allows the whole organization to respond to changes in the market quickly and without excessive cost.

That cost is broken into two parts – the apparent costs, which is the developer and operations time costs to implement and deploy a new feature, and the non-apparent costs, which are the accumulating technical debt of each change. The apparent costs are easy to measure: how many hours, days, weeks did this change take, and how many developers and other staff did I pay to do the change.

The non-apparent costs, relating to technical debt, are much less obvious. This cost is not directly measured in dollars at first – it is measured in dollars later. The best way to see it is to consider the cost difference per feature over time. Let me use an example to illustrate: Let’s say that we’re good at breaking up feature requests into approximately equal-sized pieces, and that the average cost of a feature with a given team is, say $100. If we’re accumulating no technical debt at all in a given sub-system, then our cost per new feature remains about $100 (not taking into consideration any changes in the cost of talent over time for the moment).

If, on the other hand, we are accumulating technical debt, then the cost per feature over a few months might rise to, say, $110. Then in another couple of months is $120, then $140, and so on. Each new feature is only worth so much to our business, to look at the other side of the equation. Let’s say, for the sake of argument, that a given new feature is worth $150 per month more in revenue. Simple economics tells us then that paying $150 for that feature allows it to pay for itself in a single month – so it makes sense to write it, even if the cost for that feature has risen to $150.

Unfortunately, it’s not this simple: the revenue from a feature is not easy to measure or estimate, and it does not remain constant over time – a shiny new feature might net us $150 more the first month, then $140, then $120… until the shine is gone and it’s not contributing to our product at all (arguably, that’s the time to take it out again, but that also costs money – and it’s hard to know).

It also matters, often a lot, WHEN a new feature is released. We might gain $150 profit from a feature if we release the month before our competition, but it might be worth much less after everyone else has it. In fact, we might still need our new feature just to keep up with the joneses – we might have to pay to develop it despite no increase in profit, we might need it just to avoid losing customers to the competition.

We can make a number of observations here. It is clearly desirable to get each new feature for as low a cost as is sustainable – that is, we’d like to keep paying about the same per feature over time. At the same time, we have to carefully select where we spend our money per feature so we have the right feature at the right time in order to maximize our revenue – this means the time to implement a feature is as important as its cost, sometimes more so. We might be willing to pay more to have a feature sooner, and we must be able to make that decision.

On any given system, it is common for the cost per feature to rise over time. This is due to a number of factors, but a critical one is the underlying design of the system, its level of coupling and cohesion, its indépendance from other systems, etc. The RATE of the increase over time is the most useful measure of the effectiveness of the design – if it’s increasing sharply, we may hit what I’ve called previously a point of “technical debt bankruptcy”, where its not financially responsible to add new features, as we can’t possibly make enough money from them to make it worth the ever-increasing costs.

Increasing costs also make the order of features more critical – if we get a less important feature done before a more important one, we’ve increased the cost for the more important one, hastening our collision with the bankruptcy point all the more.

I go through all of this in order to highlight the importance of being able to produce new features sustainably, at a predictable pace and at either a constant or a slowly-increasing cost. The delta over time of the average cost per feature is one good metric of the technical debt – often, some non-feature work and refactoring can in fact reduce that delta, at least for a while, so we sometimes see a pattern of bursts of new features, then a slower period where we “pay down” some of that debt, then go again, without letting it get out of hand. This is also a sustainable pattern.

Creating new features more rapidly, with less thought to design and architecture, looks good at first – we get features more rapidly, and for an apparently low cost. The affect on the delta, however, is exponential in this situation – the next feature doesn’t cost a bit more, it costs a LOT more, as we’re having to be constrained by the decisions we made in a hurry last time. That’s not to say that it’s not sometimes the right decision to go fast, and to acknowledge that we’re accumulating technical debt.

The mistake that I’ve seen over and over again, however, is to disavow the existence of the delta, of the cost of technical debt entirely. To not take it into account when making decisions is to make decisions without all the facts.

One of these decisions is the difference in importance between various design and architecture decisions, such as service-oriented architectures and event-driven architectures. One of the primary features of a well-crafted SOA/EDA is the ability to decouple systems – by this I mean that changes and new features in one system do not necessitate changes and impact on another. You can change your customer-relation-management system and its features without breaking (or even affecting) your shopping-cart system, for instance. This means that the overall cost of the change to your CRM is lower, as you don’t need to take into account the fact that you’ll break the shopping-cart – or any other system.

One of the ways that SOA/EDA creates this advantage is by specifying the communication between systems – the CRM, for instance, might respond to events from the shopping cart when customers buy things. It has no idea how the shopping cart works, or what its process for selling is, it only knows the well-defined API between it and the cart, whether this be events, REST calls, or some other mechanism – the main thing is that one system does not – in fact, must not – know the “internals” of the other.

This is true in good system design in general, not just in a SOA architecture. When designing with objects, for instance, there is the concept of “information hiding”, where the internal details of the implementation of even a single object are not exposed carelessly to the outside world. They are, instead, exposed only through carefully crafted interfaces, e.g. an API. The idea scales up to entire services just the same.

So, you might by now ask, what does this have to do with ad-hoc reporting?

Ad-hoc reporting has traditionally relied on access directly to a persistent store of some kind, typically a relational database, to be able to produce reports and replies to queries that were not anticipated when the system was built (hence the ad-hoc part). In order to do this, and to formulate a meaningful report, the report creator has to understand the meaning of the stored data, especially how it relates to other stored data, so that sets of data can be joined. This means that the domain design must be represented not in the logic of the application, but in the schema of the database. Not just the logic of one domain, or one service, either, but the logic of any that you wish to join together for ad-hoc purposes.

Good service-oriented architecture that relies on an API treats persistence as an internal implementation detail of the service – e.g. it’s the service’s business, and its alone, how it manages to store its data. Whether it’s in a database, a file, or in some kind of magical cloud storage doesn’t matter at all – and MUST not matter at all – to the clients of the service, or we’ve lost a key advantage.

But what is to stop a service from simply storing its data in a manner that can in fact later be used by ad-hoc tools? Where’s the harm, you might ask?

The harm is what brings us back to the discussion of technical debt again.

Let’s say we design our CRM service to store its data in a nice relational database, so we can support ad-hoc queries later. Now we leave the CRM service and go to the shopping cart service – same thing, we should store our data in a nice normalized schema, so we can later come along with SQL and do our queries.

Right away we’ve made some fatal assumptions: the domain model must now be entity-based, in order for ad-hoc queries to be meaningful. What I mean is this: Let’s say in our shopping cart we decided to create our customer domain object from a series of “customer-edit” events. Our service stores the events, and uses the CQRS pattern to create the customer domain object from the event entities. How do we get a list of customers from our ad-hoc reporting tool? Well, we’ve got two choices – either create the customer in some kind of view, essentially duplicating the logic that creates the customer domain object in the service, or don’t store the customer that way – store the customer as the completed domain object, or a relational representation of that object (as a customer is rarely “flat” in practice).

Now we’re seeing where the decision to design for ad-hoc reporting is skewing our service design – it’s making us consider doing things in a non-optimal manner in order to keep the world in a state where ad-hoc reporting can operate, and causing us to “bleed” domain knowledge from our service into a database schema.

Let’s say we’re OK with that decision, that we don’t mind losing a bit of design freedom in order to have our reporting choices.

Now let’s consider the CRM service. It also deals with customers, and might also have used some kind of event architecture to do its work. Let’s put it in a relational database as well, and structure its schema so we can do our reporting. It is very probable that the customer we’re dealing with in the shopping cart is the same customer (conceptually) as we’re dealing with in the CRM. Ok, then we’d like to be able to join between these two elements, and produce ad-hoc reports. This means that the CRM system must share a data definition of the key of the customer table with the shopping-cart system – e.g they must agree on data-type, and on length, and on the way the key is produced, etc.

Now we’ve started coupling these applications: any change to the customer key in CRM means a corresponding change to the customer key in the cart. Well, at that point, why don’t we just merge the two tables and have just a single definition of customer? Wouldn’t that be simpler? At first blush, it certainly seems so. Ok, now we’ve coupled a bit further, essentially making CRM and shopping cart into a single system. Where’s the harm? This brings us back to technical debt. A change to the CRM now means we must consider the impact of that change on the shopping cart, increasing the cost of that change. How much? It’s hard to say, it depends on the change, but this is a simple example – this pattern leads the way to the “big ball of mud” design, where you essentially have a single database for all services, at which point the temptation to simply use the database as a means of communications between applications becomes too great – and once you’ve done this, you don’t have lots of small applications, you just have a single application, one giant monolith that is nearly impossible to maintain, with exponentially increasing costs per change.

Another cost must be taken into consideration as well – when the ad-hoc reports are created, they are of course created at a point in time, and the schema of our mega-app is at a certain point. As the schema changes (and it will, albeit more slowly over time, as it becomes too expensive to consider easy schema changes), the ad-hoc reports, for which we went to all this trouble, stop working. I’ve never seen a “one-time” report that didn’t get saved and run over and over again, and every time the schema changes, this report breaks, and the user who used the “easy” reporting tool comes screaming down to find a developer to help him fix his report.

What we’ve essentially done is created another application: ad-hoc reporting. It is every bit as bound to the schema as all the other apps, and requires as much, perhaps more, maintenance over time. The perceived “savings” of having users do their own reports is eventually lost altogether, and is usually a net loss, costing more than if the users simply requested reports via the APIs of the services in the first place (even though that takes developer time to create).

This doesn’t happen all at once, of course. An expert team of developers that moves in this direction can manage it quite well, at least for a while. When the pressure to develop more features more quickly increases, however, it’s common that less experienced developers will start to introduce subtle couplings, that go unnoticed until you try to go fast, then suddenly something breaks.

The logical end of this pattern is to put more and more of the “logic” of the mega-app into the database, first in the schema, by having richer and richer “domain” objects stored (for instance, adding an “account total” field to the customer because it’s easier than adding it up every time – then when the logic to add it up changes in the app, it must change in the database as well). This then leads to the madness of stored procedures and triggers, as we try to “program” the anemic domain model of the database. At this point, costs per new feature are astronomical, the bug rate is through the roof, and I usually get called as a consultant

There are actually other losses that I’ve not even detailed when we move towards a shared schema: The rapid and ever accelerating cost increases of a shared schema is only the most visible. The subtle costs of making your developers have to think more about each and every schema change are perhaps, in the long run, worse. For instance, if I’m working on a service and I know that the only thing that uses its persistence mechanism is that service, I can tune the persistence for maximum advantage – I don’t have to worry about all the other use-cases of that store, there aren’t any. If I want to change the schema, I don’t have to form a committee, as long as the service continues to pass its integration tests, I can change it all I want, now, right away, and in as radical a manner as I need. This is one reason that developers love post-relational database systems such as MongoDB and Cassandra. It’s almost effortless to change the schema, as it’s defined in your application code, where it ought to be. If the domain design is only the business of the application, then where else would it be? If the domain, on the other hand, is shared between many applications, or even two, then I immediately have artificial constraints on change, which increases cost and decreases velocity. Key-value/post-relational databases also avoid that horrible impedance mis-match we call object-relational mapping layers, which are again an impediment to design freedom and easy change. Schemas that are shared and are outside the application have a tendency to fossilize – they become so hard to change that in the end we don’t change them at all, and make poor choices in our applications to “work around” their inadequacies, further reducing quality, increasing cost and time.

What are the alternatives to this scenario? How can we build systems to support the need for reporting without dissolving into a ball of coupled goo? Are these goals mutually incompatible?

First, it’s a false economy to say that we’ll design for easy ad-hoc reporting to save the time for developers to write reports: As we’ve seen, if we make the system coupled and schema-transparent enough to make it easy for power users to write reports, we’ll waste far, far more developer time than if we just admit we have a requirement for reporting and put it in the backlog. To think there’s a “cheap” way to get decent reporting is to defy reality – there just plain isn’t, and we should be professional enough to admit it.

If we’re ok with a tightly coupled system that costs a lot to change, but that support easy ad-hoc, that’s fine, let’s make that tradeoff and be upfront about it.

What if we want our cake and to eat is as well, is there any middle ground? Sure there is, and it’s the same one that’s been around for decades: it’s called a data warehouse.

Given enough access to the APIs of the services that make up our decoupled SOA/EDA-driven system, it’s straightforward to build a simple data warehouse optimized for ad-hoc queries. What we’re really doing is building a new set of clients to our services that have a different read model – this time, the read model is optimized for reporting. We build this warehouse one of two ways: where events exist that have the data we need, we listen to those events and build, where no events exist we execute timed batch jobs that query APIs – no, not the underlying database, the API – and write the results into our warehouse.

The warehouse is NOT the authoritative source of the original data, and that’s a good thing. It means we can produce a consistent and meaningful view of the data, which is not possible when reading directly from our production database (as it’s always changing and may be inconsistent at any given time). We also avoid any potential performance impact on production system, or, worse, locking tables with a read lock if we’re trying to get consistency and blocking readers and writers in the production system. You can index the data warehouse to optimize reads, which might be exactly the opposite of what you want in your production data stores.

Often the data warehouse is done with a relational database, even if the production data stores are not. For one, this gives the freedom to choose the best persistence technology for the job when building services, which contributes to cost savings, but it also speaks to the relative maturity of tools for querying in an ad-hoc manner available for relational databases. Although some tools exist for doing the same in the post-relational world, they’re not as mature as their relational cousins.

Is this data warehouse technique some work? – yes, of course. Is it worth it? I believe so quite firmly, yes. This work can be done without impacting the services and features being built for production at all.

You get the advantage of exactly the schema you want for ad-hoc purposes, even if that schema is only loosely related to the entity designs that the source data services use. You don’t have any impact at all on the services that comprise your production system, either in design or in performance, at any time. You control and have visibility as to the amount of time, money and resources that are going in to supporting ad-hoc reporting, so you can make informed decisions on how much to spend, instead of hiding those costs where they’re just as real, but much harder to see.

Conclusion

SOA and EDA are not just the new shiny thing, the latest fad. They are a fundamentally different and better way of building systems.

They are techniques which are exactly opposed to the idea of shared databases, anemic domains, and externalized fossilized schemas. Attempting to shoehorn those attributes into a SOA/EDA architecture results in many, if not most, of the advantages of SOA/EDA being lost, and the resulting frankenstein is simply bad software, costing more to develop and maintain than the advantages are worth.

If what you want is ad-hoc reporting capability, be honest about it, call it a feature, and build it into your system without compromising your design integrity and costing far more in the medium to long term. Anything else is simply unprofessional, in my opinion.

When you’re writing tests, you may have occasion to segregate some tests from the others – perhaps your functional tests, integration tests, or ui tests, or just tests that run more slowly than the rest of your suite.

There are a number of ways to tackle this (one good way is with SBT sub-projects, which I’ll cover in another post), but another option is to “tag” your tests. The ScalaTest doc covers tagging here, but what’s not as clear is how you do this and still run your tests with SBT.

Let’s say you have tagged some tests with the tag “Ui”, so you’ve got test declarations that look like this:

describe("some thing I want to test") {
it ("should do a thing", Ui) {
....
}
}

I only want to run this test (and all other tests tagged “Ui”) in one particular job on my CI server.

What I need to do is pass the -l and -n options to ScalaTest, but I want to do this only when a certain system property is passed to my SBT instance. E.g. if I have a property “ui”, I can pass it to SBT with “-Dui=true”. How do I get this property to set the right option for ScalaTest so that when ui is true, ONLY my Ui-tagged tests run, and when ui is false, all the other tests EXCEPT the Ui-tagged tests run?

Here’s what I did in my build.sbt file:

testOptions in Test ++= (if (System.getProperty("ui", "false") == "false") Seq(Tests.Argument("-oDF"), Tests.Argument("-l"), Tests.Argument("Ui")) else Seq(Tests.Argument("-oDF"), Tests.Argument("-n"), Tests.Argument("Ui")))

What this says is that I always want the arguments “-oDF” passed to Scalatest, but that when ui is false, I want “-l Ui” and when ui is true, I want “-n Ui”. I default ui to false, so if it’s not specified at all, thats the same as false.

Now I can simply set a system property for my Jenkins builds, and the right tests are run at the right time.

I thought this might be helpful to others, as it took a bit of digging to figure out the exact syntax.

Note that this appears only to work with Scalatest 1.7RC1 (or later, probably), and SBT 0.11.2 or later.

To do this safely, you of course need the confidence that your changes are not going to break production, so you need a set of comprehensive, reliable and performant tests.

Organizing these tests into the proper “gauntlet” for the change to pass through has produced what we’re now calling the “development pipeline”, as it does kind of resemble a pipe with valves in it.

Now, I’ve been using Tomcat for a long time, and it’s always done a pretty good job for me. One quirk that has always existed, in one way or another, was that it is occasionally necessary to restart Tomcat, no matter how well-behaved and non-leaking my applications are. You can make this quite seldom with carefully tuned memory arguments to your JVM, especially including the options that allow garbage-collection and re-use of permgen space, but I’ve never been able to simply make it unnecessary to eventually restart Tomcat.

Sometimes, Tomcat will even go down “hard”, becoming unresponsive and unable to be killed through it’s normal admin process, which is even worse, as this makes automated redeployment unreliable and slow.

I don’t think any of this is particular to Tomcat, either – I’ve seen variations on the same kind of issues with a great many Java servlet containers over the years, to a lesser (Jetty) and greater (Weblogic) degree.

In the last while, I’ve been developing web service applications based on the execllent Akka framework, and Spray on top of it. Spray has recently added a very thin container replacement called “Spray-Can”, which you can read about in detail on the Spray site.

In combination with the SBT “assembly” plugin, I’ve found it’s possible to deploy my apps in an entirely different way. Although there is, technically, still a container, it feels quite containerless in practice.

I build my application into a single executable JAR file, as opposed to the .war format so familiar to Java developers.

When this jar is run, my application becomes available on the configured port (8080 by default, but whatever I set it to, even at runtime).

This allows me to create a very simple script to keep my app running at all times, and to easily facilitate downtime-free upgrades, even on a single machine. Let me detail this…

My /etc/init.d script simply runs my executable jar file in a loop: e.g. something like this:

while true
do
  java -jar /apps/dir/myapp*.jar
done

Then I place the initial version, say 1.0, in the /apps/dir and run my script. Up pops my app.

Now I have another /etc/init.d start a second copy of the app, either on the same machine on a different port or on a second machine entirely. (I’ve found the lower memory requirements of this allow at least 3 copies on a single machine, though, compared with a full container).

Now I use something like Apache with mod_rewrite/mod_proxy, or balance or, my personal favorite, pound on the front end, so the concern of where my app is visible as far as URLs is entirely removed from the app itself. The application itself comes up on the context root “/”, on, say, port 8080 and 8081. Then my pound config simply redirects URLs of the form /url/to/app to one of these two ports, with failover/load balancing. My app neither knows nor cares it’s on the URL /url/to/app, that’s pound’s concern. It can even be behind HTTPS for all it cares – pound (or apache) takes care of that too.

Now when I want to deploy, I simply copy a new jar into the proper location in /apps/dir, e.g. myapp-2.0.jar.

Now I send a “poison pill” to my app in the form of a URL that causes it to shut down. My pound router filters out this call from outside the local network, so client’s can’t send it, and I send it directly to the running app (e.g. on port 8080 or 8081). This causes the app to terminate and exit completely, cleaning up any connections it needs to on the way. The VM terminates, and the shell script immediately starts up the new version on the same port.

All this time the second node on 8081 is still going strong with the old version, of course, so the clients never notice the bump.

Now I do the same to node two. When I say “I” here, I’m of course referring to my trusty Jenkins server, which is doing all this for me automatically.

Now I have all of the advantages of my app, in a highly cluster-able lightweight fashion, without using a servlet container at all.

I used to do something similar with the lightweight winstone server (which Jenkins itself uses, incidentally), but for Spray apps, the servlet API is something I don’t need, and I can run a much higher performance multi-core Akka stack instead, with all the same “containerless” advantages.

I recommend you have a look!

In this post I’d like to explore a specific principle that flows pretty readily into a practice that I think is very common among software craftsmen:

Solving the Right Problem
Craftsmen often have a somewhat different perspective on what they do as developers. They understand enough of how IT fits into the business picture to be aware of what is really needed when given a project or even a single story.

Part of the value of an experienced professional in software development is their ability to not only perform the exact task they’re set, but also to help determine if the problem is being solved in the right way. Feature definition is not just a business analyst or customer proxy rattling off a list of acceptance criteria, it’s a give-and-take negotiation. Part of that negotiation is cost-based – the developer can give detailed estimates on specific parts of a story, and the customer proxy can decide to change parts of the story to maximize the value they get for the effort expended.

In addition, however, the experienced craftsman has to have a context of the overall project, and the business advantage that is expected from the development effort. He can then blend this with the technical understanding of *how* it’s going to be done to help figure out if there is a better (e.g. faster, easier, higher quality, more maintainable) way to do accomplish the same thing.

Even beyond finding a better way, the true craftsman can determine if the problem being addressed is even the right problem.

Generally, the user has a good idea as to what they want to accomplish, but not necessarily how they want to accomplish it, or how the features for accomplishing it might be turned into actual working software. This is where your craft comes into play, helping to “translate” the users desires into actual requirements that make the best sense given the context of the existing system of what’s possible.

Sufficient experience often allows a craftsman to look at the requirements user is putting forward and to understand the underlying need for functionality. This may be very different than what you’re actually being told, and to gain experience is the guide for the craftsman to be able to steer the user from what they think they want to what they actually want. This is not at all the same as telling the user that he’s “wrong”, It’s more a matter of being technique being able to communicate with the user at their own level, and to help them explore the problem at hand more thoroughly in the context of what’s possible with software and automation, to arrive at a superior definition of the problem to be solved.

This is what I like to call “solving the right problems”, and it is generally a core principle of software press craftsmanship.

Don’t Grab the Wheel

One way to block this process, and one which I have seen happen frequently in actual practice, is for the customer, customer proxy, or other management, to oversteer technical choices that are not entirely within their domain.

Sometimes this is a trust issue, in the sense that management may not trust the developers, in this case the craftsman, to make the correct choices. Sometimes it is out of a misguided sense of fear, which manifests itself in a desire to have influence over choices that management feel would be “safer”. Frequently, this means selecting and “standardizing” on obsolete, unsuitable, and unproductive technologies and techniques.

The irony, of course, is that in the process of specifying these unsuitable technologies and techniques, the overall risk in the project goes up dramatically.

Software is unique and a bit strange in the fact that it’s the one industry where the customer generally tells the vendor not only what problem they want solved, but how to solve it, and frequently even with what tools to solve it with.

It is somewhat like going to your doctor with a tummyache and telling him “I want my arm amputated, and I want you to use this spoon to do it with”. Oh, and I want it immediately and I want to pay 4c for the service, and it better not hurt. Or going to a master carpenter and telling him “here’s a chunk of pine, make me a mahogany bannister for my staircase, and here’s a blunt butterknife to do the carving with”. While both of these sound absurd, they are no less absurd than hiring a software craftsman and telling him he must develop on a laptop using PHP and MS-Access and deploy on a five-year-old Windows 95 box with insufficient memory. (Not knocking any specific technology here, just pointing out that it’s bad to not make use of your developers expertise).

To get the best out of your craftsman, let him (or her, as I keep pointing out) do the driving. There may be necessary constraints of course, so explain them – but don’t grab the wheel out of his hands.

It lets you very easily test your Scalatra filters or servlets, but the number of things you can assert on is a bit limited. You can check status (to ensure you get a 200 instead of a 404 or a 500, for instance), and you can assert things about the contents of the response body, such as if it contains an expected string. All well and good, but not enough for my test-driven proclivities.

So I did a bit of tinkering, and came up with a nice clean way to grab things such as cookies, response values being passed to the template, the name of the Scalate template being called, and so forth.

I want to be able to write a test like this:

test("edit the organization data") {
    get(ORGANIZATION_URL) {
      status should equal(200)
      assert(template.endsWith(OrganizationController.ORGANIZATION_TEMPLATE))
    }
  }

For instance, where I assert that the template that got called was a certain template that I expect.

It turns out this isn’t very hard – just stick another Filter in the chain when you set up your test, like so:

   addFilter(classOf[TestFilter], "/*")
   addFilter(new CommonFilter, "/*")

And yes, I am using a different method invokation on the second line – I do this intentionally, so I can control creation of the servlet class under test, as I use Guice-servlet to inject all it’s service dependencies, and at test-time, I do this with implicit parameter – but that’s a whole ‘nother post

Anyway, TestFilter, as you might imagine, gives me “hooks” that I can assert on later. It looks like this:

object TestFilter {
  var lastURI: String = null
  var values: Map[String, AnyRef] = Map()
  def template = values("scalateTemplates").asInstanceOf[List[String]].head
  val cookies = ListBuffer[Cookie]()
}

class TestFilter extends Filter {

  def destroy() {}

  def init(conf: FilterConfig) {}

  override def doFilter(request: ServletRequest, response: ServletResponse, filterChain: FilterChain) {
    val wrappedResponse = new ResponseWrapper(response.asInstanceOf[HttpServletResponse])

    val httpRequest = request.asInstanceOf[HttpServletRequest]
    lastURI = httpRequest.getRequestURI
    filterChain.doFilter(request, wrappedResponse);
    val names = httpRequest.getAttributeNames
    while(names.hasMoreElements) {
      val name = names.nextElement.asInstanceOf[String]
      val value = httpRequest.getAttribute(name)
      println(name + ":" + value)
      values += (name -> value)
    }
  }

  class ResponseWrapper(val response: HttpServletResponse) extends HttpServletResponseWrapper(response) {
    override def addCookie(cookie: Cookie) {
      super.addCookie(cookie)
      cookies += cookie
    }
  }
}

That’s all there is to it – now I can call my servlet under test just like in the first code snippet, and assert on values being sent to the template, on the name of the template, on cookies, and so forth.

That’s it! Hope this turns out to be useful to some other Scalatra aficionados out there!

Developers are often tweaking and re-selecting pieces of their stack – the newest thing comes along, we like to try it out. But usually we’ve got a “go-to” stack for getting real work done, and constrain our experiments to a separate area until we’re sure something has what it takes to get included in our favorite stack.

There’s usually two major types of components in our stack – the bits we actually deploy, and the bits we use to actually develop with – the latter includes pieces we usually don’t actually ship with the finished product.

For instance, a classic J2EE stack might consist of the “deployable” pieces: Oracle’s standard Java EE 6, Hibernate, JSPs (technically a part of J2EE in any case), and Tomcat as a servlet container.

And the development stack might includes Eclipse, Maven, and various plugins to make Maven do our bidding.

There are a number of attributes I wanted in my own personal favored stack. This included:

Easy management of dependencies: I like knowing and controlling exactly what bits go into my finished application, and unless I can control all of the dependencies accurately, I can’t achieve acceptable quality.

Fast cycle-time: I want a stack that during development provides me with a rapid cycle time. E.g. the time between making an edit and seeing the results in either an application or test to be as short as possible.

Easy testability: I need my stack to facilitate testing, as my normal mode of development is BDD/TDD, where I write code because a tests tells me to, by and large. I want my testing tools to support my choice of test frameworks as well.

Modern language support: My choice of language for my own development is Scala, so I need a stack that supports Scala well. I won’t go into my reasons for choosing Scala, that’s a topic I’ve covered in another blog post.

So, without further delay, here’s my favored stack. First, the deployable bits:

Scala: I’m currently using Scala 2.8.1. Scala allows me to use a REPL for easy experimentation, which helps with my fast cycle-time requirement as well. The 2.9 release of Scala makes the REPL even more capable, but I haven’t tinkered with it much yet. I’ve talked at length about why I choose Scala in other posts, but it’s not for lack of knowing and having seriously tried other languages and environments. I can’t say I’ve tried them all, but I’ve tried quite a lot, and Scala remains my top choice.

Scalatra: Scalatra is a lightweight non-intrusive web app framework that lets me create web applications quickly, easily, and with maximum flexibility. I can write both RIA-style applications with REST/AJAX/COMET functionality or non-RIA request/response style applications, while maintaining statelessness and supporting scalability.

Scalate: With strong integration with Scalatra, Scalate is a powerful template system, perfect for generating XHTML for web applications in an extremely DRY and expressive manner. It’s like an up-to-date version of Velocity, but with Scala goodness and much more powerful.

SBT: A long-time Maven aficionado, I was hard-pressed to consider a different build system. I was, however, honest about Maven’s shortcomings, of which there are quite a few. I tried SBT and went back to Maven several times. I was intrigued enough with the advantages to come back again for another try, though, and I’ve now mastered it enough, and am getting enough benefits, that I won’t switch back.

Casbah:Casbah is the “missing link” between the Java MongoDB drivers and Scala. It allows database objects for Mongo to be readily constructed with Scala’s expressive map syntax, and type-safe retrieval of fields, not to mention it’s advanced query support.

Squeryl: On the occasion I need to interact with relational databases, Squeryl has become my tool of choice – but I’ve also dabbled with a few alternatives. One alternative I won’t go back to under any circumstances is Hibernate, or even worse, Entity EJB’s.

OpenJDK: Open JDK is a capable and independent JVM that supports Scala well.

MongoDB: I’ve used relational databases for several decades, but now that I’ve worked with key/value systems such as Mongo, I seldom find a job that RDBMS is better suited for.

Winstone: The Winstone servlet container allows me to create a single simple executable jar that incorporates both my app and the servlet container necessary for it to run in a single super-lightweight package that is easy to deploy and manage. Given it’s light footprint, it allows me to run multiple servers on a single system and provide a cluster “in a box”.

Linux: For deployment, I choose Linux. I’ve never found a business situation that wasn’t better served by Linux than by Windows, ever. Enough said.

Now the development stack:

IntelliJ IDEA: IDE support for Scala is not fantastic yet, but it’s quite good and getting better. The best of the lot is IntelliJ IDEA with it’s Scala plugin. It’s not perfect, but it’s quite capable. I will admit I frequently find myself being tempted to use TextMate, Kod, or, more recently, Sublime instead, however, but the completion and refactoring tools keep me sticking with IntelliJ.

SBT & JRebel: SBT’s own support for continious deployment and testing is excellent, but when combined with the (free for Scala) JRebel tool, it’s downright unbeatable. I can work on a webapp, make code or UI changes, flip to my browser and hit refresh and see the result. I can tell SBT to keep running my tests every time they need to run, and immediately see any breaks as they occur.

Jetty: SBT uses a built-in Jetty servlet to auto-deploy webapps for continuous development, so Jetty is part of my stack as well.

Mac OSX: For development, I work on a Mac system much of the time. I do have a desktop Linux system, but the impressive portability and power of my MacBook Air is hard to compete with.

The above are my current weapons of choice for web application development. So, what’s in YOUR stack, and why?