Projections intermission

Yesterday I was meeting with a company. We were going through some of
their problems and looking at whether the Event Store and in
particular stream querying might be a viable solution to any of their
problems. It turned out one of the problems was a perfect example of
where projections can make a big project tiny.

The Problem

People put “Bid Strategies” in through a web application. These
strategies are then used for blind live bidding of prices of some
commodity. Sometimes a bid is won. When a bid is won, sometimes
“something good” happens later. Users want real-time feedback to their
web browser on the bidding process as they are changing it. Thousands
of bids happen per second.

This is a very stereotypical problem in many systems. There is some
process that’s happening very quickly and we want to summarize that
process and give real time feedback to a user. This is a typical
example of where projections can really shine.

The Solution

Let’s start by defining the events that will be happening in this system

StrategyStarted { strategyId }
StrategyEnded { strategyId }
BidPlace {bidId } //probably some bid information as well
BidWon {bidId }
SomethingGoodHappenned {bidId}
IntervalOccured {time

We will integrate with the current system by writing these events into
the event store. There will be a stream for each strategy. As they are
not interested in a long period of time we will also set $maxAge on
the streams to one day (all data older than one day can be deleted).

Now we have the data coming into the Event Store. It can easily handle
a few thousand transaction per second. But how do we get summarized
real time data out to the user? This is where projections can come
into play. This projection will use a lot that is not yet discussed in
the projections series. Don’t let that scare you we will explain what
its doing.

fromCategory('strategy').
    foreachStream().
    when( {
                $init : function() {
                              return {
                                   "id" : 0,
                               "bidsPlaced" : 0,
                               "bidsWon" : 0,
                               "goodthings" : 0
                                   }
                       },
                StrategyStarted : function(s,e) { s.id = e.body.strategyid},
                BidPlaced : function(s,e) { s.bidsPlace += 1; },
                BidWon : function(s,e) { s.bidsWon += 1; },
                SomethingGoodHappenned : function(s,e) {
                                         s.goodthings++;
                                         linkTo('goodthings-' + s.id, e);
                                      },

                IntervalOccurred : function(s,e) {
                                          emit('liveresults-' + s.id,
                                                        {
                                                                "strategyid" : s.id,
                                                    "goodthings" : s.goodthings,
                                                    "bidsPlaced" : s.bidsPlaced,
                                                    "bidsWon" : s.bidsWon,
                                                    "time" : e.time
                                                         });
                                           s.bidsPlaced = 0;
                                           s.bidsWon = 0;
                                           s.goodthings = 0;
                                      }
             });

Wow now that’s a little more complicated than the other projections we
have been looking at! Let’s get into a bit of what it does.

fromCategory() -> this selects all streams that are in a category
foreachStream() -> this says to run this on all streams in the
category (in parallel!)

The when() is basically what we have looked at before but we have now
said we want it to run independntly against every strategy stream in
the system (each has their own state variable). It is then calculating
the counts of bids/bidswon/goodthings that happen within the interval
of time which is overall pretty basic logic.

At the end of the interval it puts a new message out to
liveresults-{strategy} with the results from the interval. For good
measure it also puts out all “good things” to a separate stream as
links. Basically the whole backend of the system is done now. It will
scale, is highly available, and fully durable.

Connecting the client

To hook up a client to this we just need to access the streams and
update the UI for the user somehow. Luckily there is an easy way to do
this. Every stream is an atom feed. Instead of coming up with some
super fancy custom way to get to the client. We can just use something
like jFeed. The client is now receiving all the events via javascript
and will just need to draw the results on the screen. How long does it
take to draw pretty results? Well thats up to the person doing it.

Summary

As you can see projections can be very useful in certain categories of
problems. In using them here a highly available scalable system
processing thousands of messages/second can be built and put into
production in less than an afternoon using nothing but javascript.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: