Monte Carlo Simulation: Estimate Product Initiatives With Confidence
Scope grows like grass, so what can engineers or product managers do to communicate the duration of initiatives when there are many unknowns? That’s where the Monte Carlo Simulation (MCS) comes in handy.
When we talk about the future, we often aren’t talking about the future at all, but about the problems of today. A software engineer, trying to persuade a product manager to invest time in reducing technical debt, will lay out in great detail future operational gain by spending less time on maintenance. A typical “today” problem.
As a first step a product manager could write down all the requirements into stories to get the initiative done and get the team to estimate each ticket in Jira. Story pointing tickets of relatively small scope is fairly straight forward, but often you don’t know all you need to know from the get go.
Also, estimating every story is very time consuming. So, what do we do for initiatives with lots of uncertainty?
This article is useful to those who have to think about bigger assumptions for what the future might bring. In other words, to everyone.
The problem
The problem is that software engineers are often reluctant (for a good reason) to provide the best guess or gut feeling of a timeline. Fixed time and fixed scope is next to impossible to achieve, especially for larger initiatives.
“When will it be done?” is a fair question from the CEO, but “In three weeks time!” is often setting the wrong expectation and leads to conservative estimates by engineers, inflating timelines for fear of reprisal. There is a better way of answering this question.
Monte Carlo Simulation (MCS)
The Monte Carlo Simulation (MCS) allows us to think differently about scope and time. When we talk about probability instead of gut feeling we allow for scenarios outside a specific date. Here’s how we go about finding an 80% probability of hitting a date.
Like any model, it works best with good data. The input will be a broad selection of estimates, for example the range of days from many engineers, broken down into best case (S), most likely (M) and worst case (L).
The more variables you consider, the better. In the example table I used some basic high level variables from an initiative in the past, such as “data migration” and “unknown” as a contingency.
Now, when we plot the data onto a chart, we can see the normal distribution (bell curve). Adding the cumulative distribution (black line) can give us the answer we’re looking for, which is the days the initiative will take with around 80% of probability.
Chances of completing the initiative are
- 10% in 135 days
- 45% in 150 days
- 79% in 160 days (!)
- 100% in 200 days
Finally, how do we make sense of the forecasts? Communicating time lines using probability and scenarios instead of fixed dates is a mind shift first and foremost, which requires engineers and all stakeholders to get on board. Probabilistic reasoning helps to produce better forecasts and brings objectivity into a task otherwise fairly subjective. Things change and so should the forecast. The closer a forecast lies to the presence the more accurately we can determine its outcome.
When it comes to forecasting, Sir John Cowperthwaite has once said something quite striking. As the financial secretary of Hong Kong in the 1960's, he laid the foundations for the city’s rapid growth. When asked how growth could be achieved elsewhere, he answered: “Start by abolishing the office of national statistics.”
Cowperthwaite believed that collecting and publishing GDP data encouraged politicians to meddle in the economy, and their actions always had unintended consequences.
The same is true for any project or initiative. Timelines should not be a metric for success and the focus should always be on the outcome.
Special thanks to Tremis Skeete, Executive Editor at Product Coalition for the valuable input which contributed to the editing of this article.
