Mighty Moose Free

BeefyMoose FlexAs some of you may know Svein (@ackenpacken) and I have decided to make Mighty Moose free. This is not a decision that is being take lightly and much thought has gone into it. This post is to explain why we went free and the future.

Almost two years ago now sitting in the Dubliner in Oslo we started having a discussion with Einar about AutoTest.Net. Wouldn’t it be cool if AutoTest.Net could only run the tests that needed to be run when a piece of code changed. We looked in the market and there was no software that could do it well (we only actually found one named infinitest in the java space and it was at the time (I have not tried it lately) very easy to trick).

We discussed with quite a few people the idea and were continually told that “its impossible”. We like “impossible problems”. After some quick spikes we even discussed it as being “non trivial but not hard” we figured a prototype could be working in about a month (given the usual software +-50% “fluff”). I love that now its no longer considered impossible but an expected behaviour of such tools

I returned to Montreal and started working on the bit that did the selection, it was dubbed the “minimizer” and worked off of static analysis (we realized very early on that code coverage is not a good solution to the problem of which tests to run as it describes the last run which may or may not have anything to do with the current run). After one month there was a prototype there, unfortunately it ran way too many tests. Often it would produce results of 200 tests when you really would expect 8 to run. After another 4-6 months of work it started to give pretty reasonable results in most situations. Sounds like a classical software estimation process to me 🙂

But enough reminiscing …

Mighty Moose is from this point forward free. Not “free in beta” or “free with a bunch of functionality turned off” but free (license is being updated as I write this). We announced this rather quietly at NDC.

Many people have asked me “why would you go free? you could charge for what you have.”. This is true. Continuous Testing tools will be taking off and yes we could charge for it however its not quite as simple as that.

To start with support is generally included with a paid product such as a dev tool. Let’s try some really quick math. Let’s say we were wildly successful the first year and got 2000 paying daily users (this would mean we would likely have at minimum double this actual users as you often give away free versions to user groups etc). This would suggest we have very deep penetration into the .NET TDD community (how many TDD’ers do you think there actually are?)

So 4000 users. What could we possibly charge for such a tool? USD$250 would seem to be about the average pricing our research suggested. So we would in the first year have a top line revenue of $500k USD. That sounds like a lot but lets start breaking that down further. For 4000 users how many dedicated support guys would be needed?

One interesting thing I have found is that very often a tool such as Mighty Moose (or any continuous testing tool) is the messenger of a problem not really the problem itself. This could be versioning issues with mSpec (I believe this is fixed now), never having run a redirected build before (this needs to be done as VS loves to lock files), or even just a new release of the CLR/Unit Testing Framework/etc. If you were to look at the width of support of a continuous testing tool its build automation + unit test running + profiler/analysis. This is very wide considering the price of the tool.

Along with this as dev tools are built to be very flexible quite often you will need to do some level of configuration on an existing project in order to get things working properly (out of box can only go so far .. even if it works you likely need to optimize things). It is not hard to trick any such tools. Just put a post build step that manipulates your output e.g.: moves files around (every such post build step would need custom integration written to understand it).

Even worse the support guys you have need to not only be developers but be good developers. In other words the role is basically 50% support 50% developing new features (dev team does support). You could try with just regular support guys and front line vs second line support but this will probably just annoy many users though it could possibly lower blended cost of support (e.g. front line + dev team). These methodologies however also annoys your developers as they are now doing direct support.

In order to get around the support cost question you are left with three possible options. The first is a no go from the start, support contracts. Developers don’t want to pay yearly fees on support costs. The second is to reach high levels of growth but an issue here is how many people actually write tests (let alone do TDD) in .NET. This brings the third model that most developer tool vendors use, I release a new major version of the software once per year.

This “major version release” strategy basically allows people to “upgrade” to the newest version. Generally I also will sunset support with this model to force upgrades (or at least lower support costs for older users). This model has been quite successful for many companies but comes with its own drawbacks. The largest drawback is that I as a company need to release a version of software whether or not I can come up with things that are really of value to my customers. Often I will end up with a few really valuable things but just listing those is not enough to get people to upgrade (think about a change log with 7 items on it…) so I start piling in crap to make a more impressive change log. This crap comes at a cost to the 80% of my users that don’t care about the particular feature and I end up building “mega tool”. This pattern can be seen over and over again whether we talk about Visual Studio itself, Microsoft Word, or any of the popular VS adding products.

This actually leads to a second issue as well. When in beta we have focused very heavily on features that only really interest the top 5% of developers (not a good base for a product, for a product we would be much more interested in the early/late majority areas of the s-curve). One question we get quite often is “how well will things work on my solution with 300 projects”. Now that we have said that we are free I can honestly tell you “I don’t care, when you have a solution with 300 projects in it you have bigger problems to worry about than getting continuous testing working”. Flashy new tools are not the answer, you should stop trying to hide your pain and start doing root cause analysis.

If we were to go down the lines of a paid product we would basically be forced to stop innovating. Our business goals would not be well aligned with our personal goals of changing the way people code. Our bread and butter would be the early/late majorities who want to use things in a much different way (they want slow incremental change) than the early adopters and on much different looking code bases. An example of this can be seen in a common feature request in our graphs of “making them look better and come up faster when there are more than 1000 nodes to show” (they come up with very small nodes). Yes we can do this (and spend 3 months of developer time working on it) but I am entering a point of alleviating pain instead of dealing with the underlying symptom (how on earth do you understand code that puts 1000 nodes in a graph, the graph will not solve this issue for you). This is also 3 months of developer time that we were not working on something actually useful.

I personally believe that the future is not in making tools that alleviate pain that many have but in helping them identify root causes of the pain and resolving it.

8 Comments

  1. David L
    Posted June 23, 2012 at 4:01 pm | Permalink | Reply

    Having seen signs of how much effort went into building it, and having tried it in the beta i can only say that this is a very big donation you have given to the community. I sure hope the people that adopt it will recognise this.

    Thanks Svein and Greg 🙂

  2. parajao
    Posted June 23, 2012 at 6:11 pm | Permalink | Reply

    Be sure to embed a “Donate” button, good products need to be supported.

    • Posted June 23, 2012 at 8:39 pm | Permalink | Reply

      I learned a long time ago that donate buttons no matter what color you make them are very well camoflaged they can hide from 99.9% of users 🙂

      • parajao
        Posted June 23, 2012 at 10:14 pm | Permalink

        Lol! maybe an “Adopt a Moose” button…
        No really, THANK YOU all; I still can’t believe you are really giving this for free…

  3. Steve F
    Posted September 11, 2012 at 11:48 am | Permalink | Reply

    Hi, MM looks very cool but I’m experiencing some unexpected behaviour. Is there a mail group or something for support? I know it’s free, so I’m not expecting too much in the way of support 😉

    • Posted September 19, 2012 at 11:53 am | Permalink | Reply

      yes there is a mailing list linked off the webstie continuoustests.com and there is a support link on the website.

      Cheers,

      Greg

      • Steve F
        Posted September 19, 2012 at 12:19 pm | Permalink

        Ooops, forgot I had NoScript running :$

  4. Gregg
    Posted October 17, 2012 at 1:50 am | Permalink | Reply

    Thanks so much for this, I’m running it full time now.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: