Monthly Archives: February 2012

Theme?!

Some people have asked me why the theme to this website looks so well awful. There are three distinct reasons for this.

1) Its “Good Enough”
2) We as developers are used to seeing pre-css’ed scripts
3) I wouldn’t want the amazing graphics I came up with to distract you from the content!

btw: the theme is meant to be a starting point for custom CSS I just never put anything in. I like it better this way.

Powerful Questions

Powerful questions are key to good analysis sessions.

From http://www.theworldcafe.com/pdfs/aopq.pdf a great quick read btw.

“Questions can be like a lever you
use to pry open the stuck lid on a
paint can. . . . If we have a short
lever, we can only just crack open
the lid on the can. But if we have a
longer lever, or a more dynamic
question, we can open that can up
much wider and really stir things
up. . . . If the right question is
applied, and it digs deep enough,
then we can stir up all the
creative solutions.”

Powerful questions dig into underlying assumptions, they create interest, and most of all they get people interested and good discussions come forth from them.

Question why

I am looking for what powerful questions you use in your analysis process so we can create a list (grabbing some from an old thread in my email as well). Here are some examples:

What is the smallest possible thing we can do to deliver this business value?

What is the need this system fills, not “what it does”

If I turned off the server tomorrow who would be the first person to notice and why?

How would you verify that this is working correctly?

what is the earliest point you can know whether the system has any value to you? How will we do this?

Why are we starting here?

Please keep them coming, the idea of powerful questions can quickly unlock the door to great discussions involving our software process.

The Context Game

After the last post The Gibberish Game, I figured I would drop in some more posts about analysis games as they can be fun (as well as quite valuable in getting information about a domain out of people.

Many times people have a very hard time with understanding that they do have contexts in their organization. This game shows different context and can be used both as an analysis game or more for a business side workshop (more on that later).

Context

The Context Game

Take a team of developers and a group of business people from different sections of the company (developers are familiar with “asking questions” etc). Break apart the group into sub-groups of one (sometimes more) business person and a developer. The groups should be grouped based on business function (this is very important, we will see why later).

Have the groups break apart into different places (a time boxed session). Give them a loaded word from the problem domain. In an insurance domain you might pick the word “policy” other good words might be “customer”, “student”, “goal”, “campaign” to name a list of some I have used previously.

The role of the developer is to capture information in this game not to get into discussions bringing about common understanding. Capture a definition of what the term means to the expert. If the term is something tangible, it can be fun to also draw a picture during the time that you are separated.

Get the group back together and have each of the teams present their idea of what the term means (and hang their picture on the wall if they have one).

Likely you will get drastically different points of view over what a term actually means. This is completely normal. Some questions to ask the group might be “How do we communicate when we all have such different definitions of a common word, have we ever played the telephone game? Can you remember any situations where communication issues have come up because of different definitions?”. Continuing questions might include seeing if anyone has different words to describe any of the other view points, e.g.: “Sales called it a Policy, Claims calls it an Origination for them a Policy has this further definition …”.

This game can very quickly show you context differences within the organization. This game can also be used as a business exercise with the choosing of a very loaded term like “Ideal Customer”. It can show the different points of view of members of the meeting allowing others to understand where people are coming from.

The game is best used when you feel that you have multiple contexts being discussed at once inside of a problem. It can quickly identify the different contexts and make them very apparent to the individuals involved. The drawing of the item if possible helps with memory retention of the concepts (visualization), it also helps with making the workshop a bit more fun.

The Gibberish Game

I figured I would drop up a quick post on a game I like to play with domain experts. Very often when getting into a context or discovering new concepts we have a tendency to just take the first word that comes to mind.

This is quite an unfortunate scenario though as often the word really sucks. What is worse is that often the word carries mental baggage with it because it is used in another context or has some definitional qualities that carry more or less information than you want to be using in the context that is being discussed. In other words

Selecting an existing word early in the process shapes our thinking.

We as creatures are easily biased. So here is the game:

Holy gibberish

The Gibberish Game

Refuse to allow a real word to be given. When going through use cases etc make a word up out of some sounds and use that for the first few hours of discussion.

Once you have gone through some analysis. Revisit and try defining the words more specifically.

Replace the word with a real word. You will be amazed how often it is different than the one you would have first selected.

The goal of the game is choosing different words until we have far more information. This can help eliminate some cognitive bias. This game can be quite good for getting confusing discussions laser focused especially when you are stuck on “what should we call this thing”

be careful you can’t use the same make believe words over and over again either otherwise they gain baggage themselves.

I played this game with a client this fall. We chose toglies and glotys to describe some concepts. When we felt that the conversation was using too broad words. Sure enough at the end we had different words. The game can be fun too (try recording yourself talking about the marital patterns of toglies).

I didn’t make sure to check up with them to share this but I can’t imagine how toglys and glotys could be covered under a NDA 🙂 btw doesn’t gloty sound like something you would order in a Polish restaurant?

We don’t need to always be so serious, even if we are wearing suits.

Sculptor “DDD” Toolkit

Just had someone send me a link to this.

Seems like another MDA “DDD” tool.

Whats wrong with this picture?

Screen Shot 2012 02 28 at 7 07 37 AM

These sentences?

“Within 15 minutes you can go from scratch to a running application, including build scripts, Eclipse projects, domain model, JPA persistence, services, Web Flow application and much more. Thereafter you can continue by evolving the design, add manual code and regenerate.”

Then I saw:
“Say Hello – Entity, Repository and Service are some of the available building blocks. Yes, it is real DDD-style.”

Here’s the link *WARNING NOT SAFE TO BE DRINKING COFFEE AS IT MAY COME OUT OF YOUR NOSE WHEN WATCHING* http://fornax-sculptor.blogspot.com/2009/08/introduction-to-sculptor.html

I imagine its probably better than that but I hate marketing BS.

Risk Management and Continuous Testing

So I just spent an hour listening to the herding code podcast on continuous testing. We were invited to come speak on it but oddly when asked at 0330 to be available at 0730; attendance is unlikely :).

One particular comment however particularly got to me from Remco Mulder (author of ncrunch).

Remco Mulder: “There are tools continuous test runners that work really hard to do impact detection and only run the tests impacted by that change. The problem with that approach though is that no tool is perfect. You may very well have the impact detection to the point that you know that every test touching a particular change point and you can say to run just those tests but that doesn’t account for external dependencies outside the testing environment. What if something changed in the database or some sql file thats not part of the solution but happens to exist on the disk somewhere. You know there is a whole range of things that can cause tests to fail that a test runner may just not be aware of at all.

Yeah that would be talking about Mighty Moose. Of course we weren’t there to be able to actually reply.

Let’s get into the meat of these comments. There is a “major” difference between Mighty Moose and nCrunch. Mighty Moose has a massive amount of effort in only running tests that need to be run through static/dynamic analysis where as nCrunch runs all of your tests but attempts to prioritize which to run based on heuristics and dynamic analysis. Mighty Moose has a huge investment in static analysis (>9 man months, its a really complicated problem).

The argument put forward by Remco would appear on the surface to make sense but its actually as we will see quite misleading.

As a side note you can configure mighty moose to run selected then background all tests just like what is being said with prioritization (e.g. static analysis becomes a form of prioritization) hence why I put “major” in quotes since we support prioritization as well.

Before Jumping In/Side Note

Before jumping into things, so my tests are dependent on data in a database and/or some obscure sql file that sits on disk outside of my solution (who knows maybe even the USB key that needs to be plugged into the machine). Is this a good thing? I could also make my tests require Eric Cartman to show up personally to where they are being run wearing a pink tutu, carrying an Antonio Banderas love doll but this is probably not such a good idea. We are basically talking about bad/super brittle tests here.

Antoniobanderas

So many tools out there have been built with the primary use case being the worst possible mess that you can imagine, this case fits firmly in that place. So you are getting occasional failures because you depend on data in a shared database that people change? Stop doing that if that causes pain (then again how often is it happening?). My guess as well is that such a code base would not survive having tests run in parallel either.

We need to remember as well, that the worst case of a test being missed (or badly prioritized for that matter) is that you find out about it later not now. This is the trade off that is at the core of Continuous Testing.

Continuing

So Mighty Moose (Static + Dynamic Analysis + other heursitics) will according to the argument given above fail to find successfully tests that are for instance dependent on data in the database. This is true. nCrunch (Prioritization based on dynamic analysis + heuristics) would also fail to be able to prioritize these tests to be run in any intelligent way. The problem is that there is not enough information as its external.

OK So they will be run … eventually with prioritization. As you keep coding new tests keep getting prioritized over the rest of those tests in the background 10-15 minutes later it tells you that you have a failing test (I mean we are talking about codebases that are hitting databases during tests so I can’t imagine these are wonderful to work in code bases with fast tests :)).

Is this actually that much better off that finding out right before you check in (what if its 5 minutes? 283 seconds?)? What is the cost of the few minutes of notice? Are there other ways of mitigating this risk? What’s the probability of this happening? What’s the cost of failure? This should sound familiar.

Risk Mitigation

The heart of Continuous Testing is risk mitigation. I do some work up front such as prioritizing your tests based on time to run/previous probabilities of failure/static analysis/dynamic analysis/etc to minimize the amount of time that you have to wait before reaching a point to continue. I also have a possibility of failure in which case I also want to minimize the time I find unexpected errors.

At any time heuristics can become a burden. I am trying to get you the best situation most of the time (this also changes from code base to code base which makes it more challenging!). The reason for this is that every form of heuristic has a cost associated with it. To instrument code or to do static analysis or to measure timings/previous failures … all of them have a cost to maintain the information to provide the heuristic. The idea is that this upfront cost works out in the long run (sound familiar to pricing strategies?).

Now what is being discussed is that constantly running tests in the background as a beta (think alpha predicting beta heuristic). The “beta” goes and actually does what the “alpha” e.g.: heuristics predicted. Is it a worthwhile endeavour? Is the cost of actually running beta to verify your predictions of value?

Another quote though was striking into what leads to the fallacious assumption of being better to run.

Remco Mulder: And you shouldn’t notice any degradation of performance as they are all using background cores

A zero cost pricing model would be a no-brainer unless it had zero benefit. Ah if only it had zero cost our lives would be so much simple. Of course we were talking about bad tests that are using all sorts of wonky external resources that we couldn’t find. My first thought my be that the disk thrashing as they pound my local database or read every file on my hard drive could feasibly be an issue (or as monitoring information on them is collected/merged into my data stores). Also buses can be an issue, memory access, locking of resources, network access … loads of things. To presume running tests on a machine has no impact on the machine is insane.

The point is we have a cost. Constantly running tests has a cost on the machine. There are possible benefits as well like any of the heuristics though it should be put into a pricing model.

For us we found that our up-front strategies (alpha) were good enough and the likelihood of failure after we say “this is what needs to be run” was so low that we didn’t need a full run (beta) in most situations. We have failure modes sure but they just aren’t hitting people often enough to justify the cost. If they were hitting problems and they wanted to reduce the cost of failure by finding them sooner they could go through and turn on our beta.

Summary

On all of these decisions for heuristics on tests we need to consider:

Probability of failure
Cost of failure
Cost to mitigate
Other mitigation strategies (e.g.: stop doing that)

There are two key points where these come into play. The first point is where “expected errors” (alpha) would occur and the second place is where unexpected (beta) would occur. If your alpha is good and your probability of failure * cost of failure in beta is low enough. We can sometimes even bypass beta. Conversely if the cost of beta is low overall (I have a solution with 10 tests) I shouldn’t bother with even having an alpha (e.g.: the prediction of beta might be more expensive than actually calculating beta … this gets to be a bit more tricky as often the information used in predicting beta say profiled data has value in its own right)

Sometimes this makes perfect sense to run all tests in the background. In some situations it does not. The ability to run without a beta search on vast number of projects should be seen not through the eyes of possible failure modes but as a sign of great heuristics (alpha) to alleviate the need for running all tests (beta) in enough circumstances that it becomes feasible to do. Conversely the sign of needing a beta and not being able to run without one is a sign of weak initial (alpha) heuristics.

Mighty Moose Demo Up

Ct

I put up a quick demo of Mighty Moose this weekend. Longer videos are in the process of uploading today so there will be more to come.

Build Times [ctd]

All I can say is wow! I never had any idea that the problem was so prevalent in the .NET space. Over 50% of people had builds ranging over 1 minute! This is a real pain point for development teams (think about your TDD cycle). I can tell you the first thing I would be doing is ctrl+right arrow to get over to my email and twitter to start checking out what is going on. This while seeming inoculate has a major cost associated with it.

I forget what I was working on. I get a push/pop on my mental stack. I also tend to get easily distracted wait shiny thing. This cost is real and can actually be quantified through the use of monitoring software. This is actually a feature we have been looking at building into a plugin (monitoring of users flow to help measure improvements).

I would love to see some discussion on why build times has been prioritized to such a low level on teams. Is it just death march mentality? Is it a frog in a boiling pot? Is it technologies that you use that have notoriously long build times? Is your software actually decoupled into reusable bits?

Build Times?

Let’s be honest (its anonymous anyways) for .NET DEVELOPERS ONLY! Think about your normal build as you work (not a production build etc)

Build Times

Was answering some questions about Mighty Moose today and figured I would just drop a quick note here about the topic.

“Mighty Moose does not work very well on my project as it takes 3-5 minutes for things to show up as my build takes that long.”

OK STOP.

Go to jail 1

If your build is taking this long you have some very serious problems on your team and with your source base. A build being used by developers to actively develop that takes this long should be your first priority to fix. There is no excuse for having a build that takes this long under any circumstances.

Note I make a difference between a build developers use to actively develop and a build on a CI server etc.

To have a build taking this long makes it impossible for developers to develop. How many times per day do you build? You are losing n MINUTES per build? Extrapolate this to a team of 7.

Oddly I have found many people doing this on purpose! The rational is using refactoring tools like reshaper and coderush. They want refactoring support across the whole codebase. Ask yourself, do you really want refactoring support across layer and tier boundaries? This sounds scary.

Another reason I commonly heart is for debugging support (that way you can step into everything). You can do this without creating one massive solution for everything, a bit of brush up on the debugging tools/support in Visual Studio can show you how to do this.

Do not create solutions that take minutes to build. Solutions are meant to be workspaces to code in, not how you produce your production build etc. Break up into many solutions to keep build times lean. You should already have these kinds of boundaries in your system.

If you corporate anti-virus takes you from 20-30 seconds to 2-3 minutes dear god find some way to get it off those machines even if it means living on a segregated network.

Mighty Moose resolution: Won’t fix. If you are in this scenario though the cascading builds in Mighty Moose though sometimes non-trivial to setup might help alleviate some of your pain as they only build projects that need to be built (and if your build is fast, they can make it even faster!)