I've been quiet for a while... here is why...

Hi again.

Sorry for my extended silence on this blog.  My excuse is that overall blogging activity is drastically reduced due to becoming a father and starting a business... also we are blogging over at http://thebitheap.blogspot.co.uk/ these days from a business perspective.

I will very occasionally put content here when it is not directly related to our company (bheap), but don't hold your breath :-)

In the meantime, I hope you are well!

Software, jamming the signal

Here I examine, on a micro-scale, the reasons for complexity and limitations in software engineering in the enterprise.

Todays programming languages are just amazing, sure they will continue to get better and better, but right now they are pretty damn amazing. It is impossible to learn everything there is to know about even a handful of them, and if we are honest, without the luxury of time most of us are sadly lacking in proficiency. In this Pantheon (considered either by beauty, elegance, capability or a dozen other criteria) I personally would place at least Scala, Clojure and Haskell.

So, what is the excuse ? Why the complexity and limitations in software engineering ? The answer is simple. Us. Human beings. In fact, it is so bad I cannot even categorise between those of us who are technical and those who are not, (as a techie I am of course biased towards blaming those who are not technical, or technical _enough_ for all the problems).

We are slaves to so many bad habits, and philosophies that it is almost a miracle we ever get anything right. I am guilty of it too. I will not try to list the bad habits and philosophies that I am guilty of as there are so many :-) , but I will list those I have witnessed in others. To avoid being purely destructive I will also point out possible fixes I have seen (both ignored and implemented).

So in no particular order, and with no particular definition of complexity here are some bad habits, and their causes:

1) artificial technical constraint imposition

• agenda (insidious and ubiquitous)

2) artificial deadline imposition

• lack of strategy

3) poor role definition and allocation of resource to roles

• lax recruiting
• lack of understanding of the capabilities of resources
• using the wrong tool for the job

So, what kind of evil do these habits let in to your organisation ?

• rushed, unmaintainable, unstable code
• lack of exploitation of the magic and power of languages, toolkits and frameworks
• the death of motivation
• the death of projects

How do we fix the three problems isolated above ?

1) Ensure the agenda of 'the business' is always placed above the agenda of any individual. This requires good communication between key project stakeholders and business domain experts and the technical teams who will be delivering on projects. Managers need balls to sort this one out, shots may need to be called to replace individuals who are 'jamming the signal'.

2) Ensure the business and technical teams work together to define sound strategy. This needs to go through some rigorous processes, and needs buy in. It is ridiculous to impose deadlines on technical teams without consulting with them. An act of lunacy.

3) Sometimes, due to either agenda or perhaps just incompetence, a resource may act like a signal jammer. Communications break down, and a project starts to fail. Once again this needs to be recognised at a higher level. Restructuring is _always_ the right thing to do here. Why use the wrong tool for the job ?

Just as a system comprised of quality components with a clear signal works beautifully, so too does a team comprised of quality resources with clear communications.

Business Startup

So, it is out there in the wild now. I am no longer a full time permanent employee. A colleague and I have started a company and are going to write software and consult. Exciting times.

We will be employing Scala, Akka, iOS and Android, so plenty of bases are covered.

There are all sorts of reasons for us choosing to do this now, including:

- being our own bosses
- pursuing a shared set of philosophies
- prospects for exciting and varied projects
- making some money !

We have had to wear so many hats and pull so many threads in to make it all work, but it has been fun so far. Today we signed our first contract. We can't wait to get to work with our new model.

It is early days yet, (content and styling are placeholders only) but our site is up. Improvements and material to come. We will link in our company twitter details and blog when they are ready.

Free Your Domain - Better Models of the Problem Domain Part One

I touched on techniques to aid in building a better model of the problem domain in a previous post. The post briefly covered two points, the first was that we can do many things 'eventually', we can 'defer' them, thus wresting more control over when we do things with our architecture; the second was that we can simplify our abstraction or model of a problem, keeping it unpolluted. I will try to provide more detail on the second point in this post.

First let us look at what software architecture is. As a discipline I can't help but think of it as a declaration of war on a problem. The problem domain can be almost anything you can think of, and can range in difficulty from trivial to almost infinitely complex. If the problem is trivial then we are very likely going to crush our enemy without much effort. On the other hand if the problem is complex, we need to be more careful, we need to consider both tactics and more importantly strategy. We need to choose the appropriate weapons to aid us, and we need to use all the tricks we know, and then some.

So, to recap, we need to choose the right weapon, something versatile, deadly, powerful. Thats easy, let us choose Scala, and actors. Our tactics and perhaps to a lesser extent our strategy depend to some extent on our choice of weapon. Likewise our weapon choice may depend on our tactics and strategy. As it happens, at work we are comfortable that Scala gives us a better range of tactics and strategy than any other language for most use cases.

Now we are on our way to giving the problem a good hammering, but we don't want to be overconfident. Now we need to ensure our strategy is good, our tactics are sound, and that we have a few tricks up our sleeve.

The legendary strategist Sun Tzu said 'all warfare is based on deception'. We are going to deceive in two very important ways. First we are going to deceive in terms of hiding the composition of our forces (this is represented by our model of the problem domain). Second we are going to deceive in terms of when we are going to attack the problem (we are going to defer, using actors), more on this in the next post.

A Deceptively Simple Model of the Problem Domain - Conceal the Composition of your Forces

We all hate a polluted and overly complicated model of the problem domain. What is it that causes this complexity ?

- lack of thought
- lack of understanding of the problem domain
- persistence

The first two I will not comment on, but persistence is worth looking at. There are so many problems caused by the persistence facet in any project I could almost write a book about it :-) Ours at work boil down to the need to persist in a relatively flexible way, sometimes relational, sometimes post relational.

By the time we have prepared our domain for persistence in just a relational store we have either written an essay in XML, or given our code a lethal injection of annotations. Simply put, we lose the will to live. It is really quite upsetting. Worse than upsetting, it is actually in my opinion dangerous to the health of the project. The elegance, intention and pretty much all other meaningful parts of the model or abstraction of the problem domain are literally lost in a maze of tangled concerns. Thats bad mmkay ?

Now we are going to take our first steps towards a better model of the problem domain. First we are going to use Scala's case classes, and only Scala's case classes to model the problem domain. No tangled persistence gibberish. Yes I consider it to be gibberish when it has invaded my sacrosanct model, where it clearly does not belong.

It is time for some pseudo code. Please note this is overly simplified and not particularly robust, it is intended to be concise :-)

sealed trait SimpleEntity {
  var id = 0L
}

case class CannonFodderGruntType1(description: String) extends SimpleEntity

case class CannonFodderGruntType2(description: String, weapons: Int) extends SimpleEntity

Urm, that is it. Hard to find a mechanism in any language which offers so much in so little code. I won't list the goodies we get for free here, they are well documented.

At this point, it looks like we have a pushover army, unable to persist itself. That is true, for now. We are deceiving our enemy, concealing the true composition of our forces until the last possible minute. Enter Scala's implicit conversions.

Once we have decided how we want to persist our domain we use a handful of lines of code to tell it how it should persist itself. If I use as an example Squeryl, an excellent SQL DSL library.

class SquerylPersistenceWrapper[T <: SimpleEntity](grunt: T) extends SpecialKey

implicit def wrapSimpleEntity[T <: SimpleEntity](grunt: T) = new SquerylPersistenceWrapper(grunt)

At work we have wrappers for JDBC based store integration and perhaps in the future other alternatives.

Suddenly the enemy (durability) sees that we are more than a match for it. At the last minute we persist our domain, as we wish, and there is nothing our enemy can do about it. Better still, our domain model is concise, unpolluted and therefore has a better chance of remaining meaningful long into the future.

To wrap up, here is very typical sample of what a domain layer looks like after a lethal injection of annotations.

@Entity
@Table(name="CANNON_FODDER_GRUNT_1")
case class CannonFodderGruntType1(
  @Id @GeneratedValue(strategy=GenerationType.AUTO, generator="CFGT1_SEQ")
  @SequenceGenerator(name="CFGT1_SEQ", sequenceName="CFGT1_SEQ")
  @Column(name="CFGT1_ID", nullable=false)
  @BeanProperty var id: Long) extends SimpleEntity {

  @Column(name="DESCRIPTION", nullable=false, length=254)
  @BeanProperty var description: String = _
  
  def this() = this(null)
}

Urm, yes, well, something along those lines :-) The intent is most definitely in my opinion lost.

Type Class Utopia Part One

I have been working through a personal dilemma recently, namely that of deciding what the perfect solution would be for making the ultimate type class implementation. I have seen a series of dialogues based around Scala type classes in particular over the last few weeks, and it is this which made me want to look back to Haskell, to see if Scala has 'caught up' with it.

I can only speak with the benefit of experience on type classes in two languages, Haskell and Scala, an old, elegant, 'firm favourite', and a more recently found 'get things done', powerhouse. My Haskell days are mostly behind me now as I must admit to never really being able to use it in enterprise. Nonetheless it is still to me the most elegant, pure, beautiful language. My Scala experience is limited to a little over three years, with only two years worth of putting it to work in the enterprise.

If we start with Haskell and a contrived example, a Difference Engine, somewhat simpler than Charles Babbage's version. By chance we have a need at work to do something along these very lines, basically a pluggable calculation engine.

data CalculableEntity = CalculableEntity { x :: Int, y :: Int }

class DifferenceEngine a where
    calculateDifference :: a -> Int

instance DifferenceEngine CalculableEntity where
    calculateDifference (CalculableEntity a b) = a - b

calculate a = calculateDifference(a)

So, that is pure, elegant, beautiful even. Moreover it is easy to take it further and calculate on the Double type or any other, we just need a new instance working with the new type. So far there is no form of obfuscation in my opinion, it can not as far as I can see be made any simpler to a developer trying to understand intent. Is this perfection ? Let us come back to that when we have explored it further a little later.

Let us look at the Scala equivalent.

case class CalculableEntity(x: Int, y: Int)

trait DifferenceEngine[T] {
  def calculateDifference(ce: T): Int
}

def calculate[T](t: T)(implicit de: DifferenceEngine[T]) = de.calculateDifference(t)

object DifferenceEngine {
  implicit object CEDifferenceEngine extends DifferenceEngine[CalculableEntity] {
    def calculateDifference(ce: CalculableEntity) = {
      ce.x - ce.y
    }  
  }
}

This does basically the same thing. Personally I look at this and immediately see clutter. Now some of that we can excuse as Scala marries the functional and object oriented paradigms, and excellently too. One consequence of this is that Scala's type system can not be as elegant as Haskell's. What worries me more though is the use of implicits. Implicits offer us great power, but with great power comes great responsibility, and all of these extra keywords in my opinion add more clutter, further obfuscating the intent of the code.

Here it is almost like we are losing our identity a little with all of this wrapping and the sprinkling of magic keywords. I understand that we are delegating to the compiler, but in so doing we are polluting the reference manual by which our audience will understand our intent.

So ends round one, and in my opinion Haskell comes out on top. I think Charles Babbage would have backed Haskell thus far. In part two I will explore where Scala may have an edge over Haskell.

Eventually everything, and actors

I tweeted a little while ago about the appeal of 'eventual' everything; eventually persisted, eventually synchronised etc etc. I am still of a mind that this is a good thing, it lets cross cutting concerns be handled in a truly elegant way, especially when coupled with the actor paradigm.

In recent days at work we have been tasked with architecting a new logging / auditing module for our platform, and we had to jump through the usual hoops in thinking out a solution. We have what is often called a kernel, and many (surely to grow in number) satellite modules. Some persist to a relational store, some to a search index, some to a post relational store. As per usual we need to be able to audit actions on any of these, with clearly defined verbs dictating what is possible within our system. We need to think of the usual CRUD operations, and also 'preview' and 'publish' operations.

We immediately thought about the possibility of 'eventually' logging / auditing, as in our case there is no need for real time reporting on logged / audited information. Additionally, by decoupling auditing / logging to an even greater extent than using aspects, by using actors, we can keep our codebase simple and elegant, letting a hive of workers do our work for us, in a scalable way with redundancy built in whether it is on one machine or across many.

A further advantage is that completely independently from the code which fires auditing or logging operations we can model in a simple real world way the kind of delegation required to perform all these many tasks which need to occur in our platform. We simply abstract our auditing 'messages' conveniently handled by Scala's case classes, and fire them at a worker or set of workers who know how to handle them, and can choose how they want to persist them. No need to wrap unpleasant aspects across the multiple projects which comprise our ever growing platform.

We are betting on actors in a big way, and this paradigm is delivering in solving real world problems in an elegant way. Right now there are several actor implementations, with more in the pipeline. For us Akka provides a massive number of integration possibilities, and great performance.

What does Scala have to offer the workplace?

Update: Phil Bagwell of the Scala team has done a far better job than me with writing this up over at :

scala-lang.org

I posted very briefly a little while back on our experiences with Scala. The positive results of our experiences were duly noted. I thought it was time to give a little more back to the community, and those considering this move. This post attempts to put some thoughts together on:

- who are we and what do we do ?
- why Scala ?
- what do we do with Scala ?
- how to transition ?
- what benefits will you see ?

Of course this commentary is subjective, it is what we as a team have found, and we had a particular scenario in place at our company 'Thatcham Motor Insurance Repair Research Centre'. Furthermore the management have been very open minded and supportive of the technical team's decisions regarding technology stack options.

At Thatcham we conduct research and produce data based on that research. We research efficient, safe, cost effective repair of vehicles, and work with manufacturers to influence the design of new vehicles. If you want to know more, you can visit our site: www.thatcham.org (please note this is an old site currently under redesign by our team on a new Scala powered platform).

Our team is small, but over the last year we have been tasked to rewrite the entire suite of research software within the company, and to create a new publishing platform to disseminate the data gathered by the research software. These two parallel projects are long running, comprising many mostly web services based modules. The need to rewrite our software stems from a transition to a more web services oriented technical world, and a desire to architect a more coherent extensible platform. A small team needs to avoid firefighting, and must think things through clearly in order to not just survive, but evolve.

Why Scala? In all honesty our first concern was the sheer bloat involved in creating such a large architecture with Java. We initially had a split down the middle of the team between JVM land and .NET land. It is noticeable these days how far plain old Java is lagging behind. Why not use something better than both, with all the choice and power of the venerable Java ecosystem on tap?

We use Scala exclusively across the server side, and these days, that is where the majority of the action is. We have traditional ORM oriented modules, distributed modules, publishing modules and traditional webapp modules. The question in fact is what can't we do with Scala? Perhaps the most crucial of our modules or layers, are the distributed modules, here we make use of Scala's excellent concurrency ready concepts, and in particular actors / message passing. Our chosen implementation of actors is www.akkasource.org, mainly for the very capable integration capabilities it offers. Effectively Scala and in particular Akka have made us cloud ready, scalable and confident that we are flexible.

How do you get started with Scala, and more importantly bring it in to the enterprise ? Start slowly, Java and C# devs can all get fairly proficient quickly. Ensure the management understand the different ways Scala can be put into use. You can write Scala in a Java style, then harness it's power to a greater level over time. Finally, don't be afraid to ask your questions on the mail lists, both the Scala and Akka list members are very helpful.

How has using Scala benefited Thatcham ?

- the power of functional programming, more choice than just OOP
- collapsing the inheritance hierarchy, alternatives to inheritance, reducing coupling
- expressive syntax, express your intention, solve problems cleanly
- everything is a library, make the language do what you want, how you want it to do things
- enriching the toolset, upgrade your tools, 'pimp my library'
- dropping heavy, cumbersome dependencies, Scala does what Java needs many libraries and frameworks to do
- simple lightweight compile time checked composability and dependency injection
- feature rich actors / message passing, 'free your domain' from pollution, be flexible, scalable

Of note, we have measured a reduction in lines of code by approximately one third thanks to Scala's expressive syntax, and the concise nature of functional programming. Our reduction in dependency baggage has lead to our deployable artifact's more than halving in size where we are not hooked up to ORM. Perhaps more importantly we have more options at an architectural level, thanks to the capabilities of a more evolved language, and developing and architecting are just more fun, which is an often overlooked side effect of the transition.

All of this, and we can still leverage the undeniably rich JVM ecosystem, all those well implemented frameworks. In short we can do more with less code, and we can do it better.

If you want to know more, post a comment, I will be happy to give any help I can.