Monthly Archives: November 2013

How to use partial application like DI?

I had an interesting question today on twitter from someone having some issues trying to do what I discuss in my 8 lines of code talk

Unfortunately though while the Event Store does manual wire up it does not also use partial application in that code so its not a great example so I asked for a gist with example code to see what the problem may be.

https://gist.github.com/ndonze/7675989

As you can see the types are the big problem here as it wants a List<Func<IMessage, IHandlerResult>>() unfortunately the compiler will not like this (some languages can actually do this! C# is not one of them). I replied with the following code that works for the particular case

This code is very similar to the first bit (still registers up handlers etc) but it has one small trick being played in it. In particular when registering a handler you will notice:

_dictionary.Add(typeof(T), x => func((T)x));

Internally the Dictionary stores as an internal interface for IMessage. This line of code causes the cast to happen. If I were to write it out as a full function it would look something like:

static MyReturn Convert(IMessage message) {

OtherHandler((ConcreteMessage) message)

}

This allows all the internal things to be treated and dispatched as IMessage but the outside handlers to use a ConcreteMessage. We can go one step further than this though (and probably should). Note that the code is no less testable, we are just passing in dependencies in a different way.

Why should I TELL you what IMessage must be?

This same code can also work for command handlers as well as queries. In order to do this we will take direction from F# (as well as many other languages) and introduce a type that represents void. We need to do this as System.Void is not allowed to be used for generic arguments.

https://gist.github.com/gregoryyoung/7677751

Now we can use this in conjunction with our dispatcher to handle commands as well as an Action<TCommand> is equivalent to a Func<TCommand, Nothing>

Now we can use the same code in either place!

Durability of Writes

Just finished reading some upcoming posts by Ayende I think they will be out this week and are an interesting series. They are in regard to durability of writes in particular with a journal. In dealing with Event Store we have done a lot of work with this. In particular his discussions get into looking at the differences between appending a log in batches with occasional fsync/flushfilebuffers vs other methods such as o-direct/write-through etc.

If only it were so simple. IO is hard. There are many places between you and the disk and getting everything right (pun intended) requires a ton of testing/knowledge. It gets even more fun when you consider the number of subsystems under you (are you even talking to a local disk? does that disk have caching? is the caching safe? does the controller ignore fsyncs?)

By the way: promotion this is actually the subject of my talk at Build Stuff in Vilnius Dec 9-11 hope to see you there 😉

Let’s start with what is good about batch writes + fsync, this is the default in event store. It is the most compatible and likely to work on an unknown system. Forget about even locally connected drives how will directio work with 4kb page aligned writes when you are talking to a networked device (really your head will start hurting). It has however some big problems. The largest problem is that you are not a good neighbor. When you fsync you do not only fsync your file you fsync all writes that are going to that disk. This can cause all sorts of interesting latency issues where you get spikes because something else is writing to the disk. Your constant fsyncing will also affect performance of everything else running on the box. On a side note your constant fsyncing can sometimes make other systems that forget somewhat less likely to fail!

A second issue with fsyncing/flushfilebuffers is that it also flushes all metadata changes to the disk. This can cause a large number of writes that you may not need to be done to be written to the disk. This is especially bad when you consider that it can cause seeks in the process.

I have just finished implementing for windows O-DIRECT aka unbuffered IO for windows as an option. I have started working on the posix implementation as well. We will be running it through our circles of hell (power pulling clusters) to validate durability etc for a few weeks before release. Once done it will be available for everyone to use OSS is great!

This is one aspect that is often not considered in such decisions. Sure it may only be 500loc but have you actually made sure it really is durable?