Sunday, September 26, 2010

Free Your Domain - Better Models of the Problem Domain Part One

I touched on techniques to aid in building a better model of the problem domain in a previous post. The post briefly covered two points, the first was that we can do many things 'eventually', we can 'defer' them, thus wresting more control over when we do things with our architecture; the second was that we can simplify our abstraction or model of a problem, keeping it unpolluted. I will try to provide more detail on the second point in this post.

First let us look at what software architecture is. As a discipline I can't help but think of it as a declaration of war on a problem. The problem domain can be almost anything you can think of, and can range in difficulty from trivial to almost infinitely complex. If the problem is trivial then we are very likely going to crush our enemy without much effort. On the other hand if the problem is complex, we need to be more careful, we need to consider both tactics and more importantly strategy. We need to choose the appropriate weapons to aid us, and we need to use all the tricks we know, and then some.

So, to recap, we need to choose the right weapon, something versatile, deadly, powerful. Thats easy, let us choose Scala, and actors. Our tactics and perhaps to a lesser extent our strategy depend to some extent on our choice of weapon. Likewise our weapon choice may depend on our tactics and strategy. As it happens, at work we are comfortable that Scala gives us a better range of tactics and strategy than any other language for most use cases.

Now we are on our way to giving the problem a good hammering, but we don't want to be overconfident. Now we need to ensure our strategy is good, our tactics are sound, and that we have a few tricks up our sleeve.

The legendary strategist Sun Tzu said 'all warfare is based on deception'. We are going to deceive in two very important ways. First we are going to deceive in terms of hiding the composition of our forces (this is represented by our model of the problem domain). Second we are going to deceive in terms of when we are going to attack the problem (we are going to defer, using actors), more on this in the next post.

A Deceptively Simple Model of the Problem Domain - Conceal the Composition of your Forces

We all hate a polluted and overly complicated model of the problem domain. What is it that causes this complexity ?

- lack of thought
- lack of understanding of the problem domain
- persistence

The first two I will not comment on, but persistence is worth looking at. There are so many problems caused by the persistence facet in any project I could almost write a book about it :-) Ours at work boil down to the need to persist in a relatively flexible way, sometimes relational, sometimes post relational.

By the time we have prepared our domain for persistence in just a relational store we have either written an essay in XML, or given our code a lethal injection of annotations. Simply put, we lose the will to live. It is really quite upsetting. Worse than upsetting, it is actually in my opinion dangerous to the health of the project. The elegance, intention and pretty much all other meaningful parts of the model or abstraction of the problem domain are literally lost in a maze of tangled concerns. Thats bad mmkay ?

Now we are going to take our first steps towards a better model of the problem domain. First we are going to use Scala's case classes, and only Scala's case classes to model the problem domain. No tangled persistence gibberish. Yes I consider it to be gibberish when it has invaded my sacrosanct model, where it clearly does not belong.

It is time for some pseudo code. Please note this is overly simplified and not particularly robust, it is intended to be concise :-)

sealed trait SimpleEntity {
var id = 0L

case class CannonFodderGruntType1(description: String) extends SimpleEntity

case class CannonFodderGruntType2(description: String, weapons: Int) extends SimpleEntity

Urm, that is it. Hard to find a mechanism in any language which offers so much in so little code. I won't list the goodies we get for free here, they are well documented.

At this point, it looks like we have a pushover army, unable to persist itself. That is true, for now. We are deceiving our enemy, concealing the true composition of our forces until the last possible minute. Enter Scala's implicit conversions.

Once we have decided how we want to persist our domain we use a handful of lines of code to tell it how it should persist itself. If I use as an example Squeryl, an excellent SQL DSL library.

class SquerylPersistenceWrapper[T <: SimpleEntity](grunt: T) extends SpecialKey

implicit def wrapSimpleEntity[T <: SimpleEntity](grunt: T) = new SquerylPersistenceWrapper(grunt)

At work we have wrappers for JDBC based store integration and perhaps in the future other alternatives.

Suddenly the enemy (durability) sees that we are more than a match for it. At the last minute we persist our domain, as we wish, and there is nothing our enemy can do about it. Better still, our domain model is concise, unpolluted and therefore has a better chance of remaining meaningful long into the future.

To wrap up, here is very typical sample of what a domain layer looks like after a lethal injection of annotations.

case class CannonFodderGruntType1(
@Id @GeneratedValue(strategy=GenerationType.AUTO, generator="CFGT1_SEQ")
@SequenceGenerator(name="CFGT1_SEQ", sequenceName="CFGT1_SEQ")
@Column(name="CFGT1_ID", nullable=false)
@BeanProperty var id: Long) extends SimpleEntity {

@Column(name="DESCRIPTION", nullable=false, length=254)
@BeanProperty var description: String = _

def this() = this(null)

Urm, yes, well, something along those lines :-) The intent is most definitely in my opinion lost.

Friday, August 20, 2010

Type Class Utopia Part One

I have been working through a personal dilemma recently, namely that of deciding what the perfect solution would be for making the ultimate type class implementation. I have seen a series of dialogues based around Scala type classes in particular over the last few weeks, and it is this which made me want to look back to Haskell, to see if Scala has 'caught up' with it.

I can only speak with the benefit of experience on type classes in two languages, Haskell and Scala, an old, elegant, 'firm favourite', and a more recently found 'get things done', powerhouse. My Haskell days are mostly behind me now as I must admit to never really being able to use it in enterprise. Nonetheless it is still to me the most elegant, pure, beautiful language. My Scala experience is limited to a little over three years, with only two years worth of putting it to work in the enterprise.

If we start with Haskell and a contrived example, a Difference Engine, somewhat simpler than Charles Babbage's version. By chance we have a need at work to do something along these very lines, basically a pluggable calculation engine.

data CalculableEntity = CalculableEntity { x :: Int, y :: Int }

class DifferenceEngine a where
calculateDifference :: a -> Int

instance DifferenceEngine CalculableEntity where
calculateDifference (CalculableEntity a b) = a - b

calculate a = calculateDifference(a)

So, that is pure, elegant, beautiful even. Moreover it is easy to take it further and calculate on the Double type or any other, we just need a new instance working with the new type. So far there is no form of obfuscation in my opinion, it can not as far as I can see be made any simpler to a developer trying to understand intent. Is this perfection ? Let us come back to that when we have explored it further a little later.

Let us look at the Scala equivalent.

case class CalculableEntity(x: Int, y: Int)

trait DifferenceEngine[T] {
def calculateDifference(ce: T): Int

def calculate[T](t: T)(implicit de: DifferenceEngine[T]) = de.calculateDifference(t)

object DifferenceEngine {
implicit object CEDifferenceEngine extends DifferenceEngine[CalculableEntity] {
def calculateDifference(ce: CalculableEntity) = {
ce.x - ce.y

This does basically the same thing. Personally I look at this and immediately see clutter. Now some of that we can excuse as Scala marries the functional and object oriented paradigms, and excellently too. One consequence of this is that Scala's type system can not be as elegant as Haskell's. What worries me more though is the use of implicits. Implicits offer us great power, but with great power comes great responsibility, and all of these extra keywords in my opinion add more clutter, further obfuscating the intent of the code.

Here it is almost like we are losing our identity a little with all of this wrapping and the sprinkling of magic keywords. I understand that we are delegating to the compiler, but in so doing we are polluting the reference manual by which our audience will understand our intent.

So ends round one, and in my opinion Haskell comes out on top. I think Charles Babbage would have backed Haskell thus far. In part two I will explore where Scala may have an edge over Haskell.

Tuesday, May 11, 2010

Eventually everything, and actors

I tweeted a little while ago about the appeal of 'eventual' everything; eventually persisted, eventually synchronised etc etc. I am still of a mind that this is a good thing, it lets cross cutting concerns be handled in a truly elegant way, especially when coupled with the actor paradigm.

In recent days at work we have been tasked with architecting a new logging / auditing module for our platform, and we had to jump through the usual hoops in thinking out a solution. We have what is often called a kernel, and many (surely to grow in number) satellite modules. Some persist to a relational store, some to a search index, some to a post relational store. As per usual we need to be able to audit actions on any of these, with clearly defined verbs dictating what is possible within our system. We need to think of the usual CRUD operations, and also 'preview' and 'publish' operations.

We immediately thought about the possibility of 'eventually' logging / auditing, as in our case there is no need for real time reporting on logged / audited information. Additionally, by decoupling auditing / logging to an even greater extent than using aspects, by using actors, we can keep our codebase simple and elegant, letting a hive of workers do our work for us, in a scalable way with redundancy built in whether it is on one machine or across many.

A further advantage is that completely independently from the code which fires auditing or logging operations we can model in a simple real world way the kind of delegation required to perform all these many tasks which need to occur in our platform. We simply abstract our auditing 'messages' conveniently handled by Scala's case classes, and fire them at a worker or set of workers who know how to handle them, and can choose how they want to persist them. No need to wrap unpleasant aspects across the multiple projects which comprise our ever growing platform.

We are betting on actors in a big way, and this paradigm is delivering in solving real world problems in an elegant way. Right now there are several actor implementations, with more in the pipeline. For us Akka provides a massive number of integration possibilities, and great performance.

Sunday, April 25, 2010

What does Scala have to offer the workplace?

Update: Phil Bagwell of the Scala team has done a far better job than me with writing this up over at :

I posted very briefly a little while back on our experiences with Scala. The positive results of our experiences were duly noted. I thought it was time to give a little more back to the community, and those considering this move. This post attempts to put some thoughts together on:

- who are we and what do we do ?
- why Scala ?
- what do we do with Scala ?
- how to transition ?
- what benefits will you see ?

Of course this commentary is subjective, it is what we as a team have found, and we had a particular scenario in place at our company 'Thatcham Motor Insurance Repair Research Centre'. Furthermore the management have been very open minded and supportive of the technical team's decisions regarding technology stack options.

At Thatcham we conduct research and produce data based on that research. We research efficient, safe, cost effective repair of vehicles, and work with manufacturers to influence the design of new vehicles. If you want to know more, you can visit our site: (please note this is an old site currently under redesign by our team on a new Scala powered platform).

Our team is small, but over the last year we have been tasked to rewrite the entire suite of research software within the company, and to create a new publishing platform to disseminate the data gathered by the research software. These two parallel projects are long running, comprising many mostly web services based modules. The need to rewrite our software stems from a transition to a more web services oriented technical world, and a desire to architect a more coherent extensible platform. A small team needs to avoid firefighting, and must think things through clearly in order to not just survive, but evolve.

Why Scala? In all honesty our first concern was the sheer bloat involved in creating such a large architecture with Java. We initially had a split down the middle of the team between JVM land and .NET land. It is noticeable these days how far plain old Java is lagging behind. Why not use something better than both, with all the choice and power of the venerable Java ecosystem on tap?

We use Scala exclusively across the server side, and these days, that is where the majority of the action is. We have traditional ORM oriented modules, distributed modules, publishing modules and traditional webapp modules. The question in fact is what can't we do with Scala? Perhaps the most crucial of our modules or layers, are the distributed modules, here we make use of Scala's excellent concurrency ready concepts, and in particular actors / message passing. Our chosen implementation of actors is, mainly for the very capable integration capabilities it offers. Effectively Scala and in particular Akka have made us cloud ready, scalable and confident that we are flexible.

How do you get started with Scala, and more importantly bring it in to the enterprise ? Start slowly, Java and C# devs can all get fairly proficient quickly. Ensure the management understand the different ways Scala can be put into use. You can write Scala in a Java style, then harness it's power to a greater level over time. Finally, don't be afraid to ask your questions on the mail lists, both the Scala and Akka list members are very helpful.

How has using Scala benefited Thatcham ?

- the power of functional programming, more choice than just OOP
- collapsing the inheritance hierarchy, alternatives to inheritance, reducing coupling
- expressive syntax, express your intention, solve problems cleanly
- everything is a library, make the language do what you want, how you want it to do things
- enriching the toolset, upgrade your tools, 'pimp my library'
- dropping heavy, cumbersome dependencies, Scala does what Java needs many libraries and frameworks to do
- simple lightweight compile time checked composability and dependency injection
- feature rich actors / message passing, 'free your domain' from pollution, be flexible, scalable

Of note, we have measured a reduction in lines of code by approximately one third thanks to Scala's expressive syntax, and the concise nature of functional programming. Our reduction in dependency baggage has lead to our deployable artifact's more than halving in size where we are not hooked up to ORM. Perhaps more importantly we have more options at an architectural level, thanks to the capabilities of a more evolved language, and developing and architecting are just more fun, which is an often overlooked side effect of the transition.

All of this, and we can still leverage the undeniably rich JVM ecosystem, all those well implemented frameworks. In short we can do more with less code, and we can do it better.

If you want to know more, post a comment, I will be happy to give any help I can.

Friday, March 05, 2010

Scala Revisited

Some considerable time has passed since I last blogged. In part, this is because of just how busy I have been working on excellent projects at work, and in part because I must confess to tweeting these days.

There is still definitely a place for blogging if you want to say a little more though so here goes.

In short, using Scala for real in the work place has worked out better than I could ever have imagined. Not only is it a joy to work with in hobby coding world, but it is a real world powerhouse problem solver. After a successful proof of concept project, and a prototype converted project, the company have now agreed to adopt Scala for all server side development. Furthermore, because of the powerful capabilities of the language, our rules systems will probably end up being written in Scala.

This has enabled us to drop the never quite delivering on the promises .NET platform. Too many times we came up against brick walls, or found partially or incorrectly implemented specs. We now have the brilliant position of being able to use the powerful and varied open source projects in the JVM ecosystem.

Scala is a joy to use. This joy has persisted for me from when I first tinkered with it just over two years ago to the present day.

Long may it continue.