Short Overview of Estimation
Back in 2008, Bob Payne and I were working with a team learning to practice Agile Software Development. They were doing quite well at a lot of things, but their sprint velocity bounced up and down like a yo-yo. The sawtooth velocity chart indicated, since they were clearly delivering value at a pretty steady pace, that they were not very good at estimating stories. Bob suggested to me that they might do as well, with less effort, by counting the stories instead of estimating them.
We did a bit on number crunching on the data we had, and it seemed to support that hypothesis. We put out a call on some mailing lists seeking other datasets to examine.
Bob Payne and I have been considering the effectiveness of story estimates for planning via Yesterday’s Weather. We’ve looked at some historical data from a team we’ve worked with, comparing the power of story points and a simple count of the story cards. The results looked interesting to us, so we’d like to look at a wider set of data.
If you’d be willing to send us data from your team(s), we need historical data for a series of iterations of a team, containing
– the number of story points completed for the iteration
– the number of stories completed for the iteration
– the range of story points (low to high) for the stories of the iteration.
If this last value isn’t easily available, then the range of story points generally used by the team will do.
Thanks for your help.
The responses were encouraging. We submitted a session proposal to the “Agile Frontier” stage of Agile 2009. Unfortunately, it was not one of the 18 proposals accepted for that stage. We tried again, a few years later, and presented What’s the Point of Story Points at Agile 2012 to an over-stuffed room.
In this session, we said many of the things that people calling for “No Estimates” are saying.
- Estimation is not the goal, and can distract us
- Estimation wastes more time and effort than it’s worth
- Estimation encourages too-detailed planning, which then encourages “sticking to the plan” due to the sunk cost
- Estimation is fractal, and grows as we work in finer detail
- Velocity and estimates get treated as things they are not
- If we work in priority order and limit work-in-progress, estimating the amount of work that will fit a timebox is less important
- If we work on things that are an order of magnitude more valuable than the cost of producing them, then estimation of cost is less important
But a funny thing happened. While arguing against estimates, I found some residual benefit for estimation. From a business perspective, it’s quite valuable to project into the future to predict what might happen, and to make contingency plans, if needed. Making estimation less important does not make it unimportant.
Since that time, I’ve been asking different questions. In particular, what is the residual importance of estimation? I’ve come up with a number of ideas. In no particular order:
- Estimation allows us to bring past experience to bear on future work.
- Estimation is a valuable technique for making decisions, but it is not the only technique, nor is it always applicable.
- Estimates can be used to communicate to stakeholders, particularly those outside your organization, that you have the relevant experience to understand the work and have paid serious attention to their request.
- Estimates can be valuable for routine business activities such as planning release dates, managing risk, summarizing a situation for executive stakeholders, coordinating with parallel efforts, choosing between alternatives, and justifying decisions that spend someone else’s money.
- Estimates are based on assumptions, and using them as a hypothesis can provide early notice that those assumptions might be incorrect when actuals differ from the estimate.
When we have a clearer idea of what we want an estimate to accomplish, we can then ask how can we achieve the important goals simply?
I’ve previously suggested that “Continuous Estimation” might be a worthy tag, in the vein of continuous integration, continuous improvement, continuous design, and continuous planning. I think “Extreme Estimation” also has merit—pushing the limits of estimation to the extreme. The tagline “No Estimates” seems problematic, to me.
- Naming something for what it is not gives little information.
- Telling people not to do what they’ve been doing, or even appearing to tell them that, is needlessly antagonistic.
- I find that I do not want to eliminate all estimates, after all.
When I wrote the title of this article, before starting it, I estimated a “short” overview. As you can see, it’s not so terribly short. Such is the nature of estimates—they represent our best understanding, and sometimes our hopes, at the time we made them. We should always keep in mind that they are just estimates.
I think I mostly share your views here: both those that align with “No Estimates” and those about where estimates seem to have value. I see no real prospect of changing the name, though I suppose if a bunch of us started doing it it might catch on.
My own concerns are around these two questions:
1. How does a funding agency decide how much to invest in some new effort? I get that we would have that investment be small. But the agency probably wants either a very very small risk, or some certainty of some kind. Anyway, how do they decide how much to spend, and how can we help them? Or, how can we even decide whether to take their money and try?
2. Relatedly, what are some good and effective ways to wean people who have power and want estimates? Courage and saying “no” are perhaps great strategies for some people, but many feel that they do not have that luxury.
I’d like to read stories about how to deal with relatively big problems in relatively big companies, while dealing with moving toward “No Estimates”.
Just as we’ve found with XP practices, part of the problem is in helping people realize that they can in fact do them, even if imperfectly at first.
1. The easiest situation is when the estimated return is an order of magnitude greater than the estimated cost. At least, that’s true from a logical standpoint. Sometimes the greater concern is spending the budget that’s allotted to them, and having something that appears shiny and valuable when that’s done.
2. It always seems to come down to understanding what that person in power wants. This is hard to do across multiple power levels. The person in power whom you know may just be passing along the desires of the person in power over them. Sometimes you need to coach your person-in-power to deal more effectively with their person-in-power to try to get that information. The difficulty goes up exponentially with the number of levels.
Maybe an easier approach is to publish articles in airline flight magazines. 😉
I’ve found value in the “no estimates” arguments, but something bugged me about it, and not only because I work on a product that is based on using estimates.
Mike Cohn has said that what the business really wants to know is when they will get a particular feature (or capability). So I agree they are valuable for “routine business activities”. Of course they are only estimates. But when I worked almost 9 years on a team – it didn’t take that long for our estimates to be useful for planning out a couple months.
The past experience thing is really important IME. So is the idea of communicating relevant experience. If it’s something we’ve done before, it’s predictable. If it’s all new, we’re going to have to do some spikes. My team that was together so long, we built up trust with the biz that we could take a sprint or three to spike something and verify the architecture scaled. Then we could make a meaningful estimate.
Lisa, it’s excellent you’ve built such a trusting relationship. I don’t think you can build such trust by just providing numbers. I think it takes a deeper conversation about what the numbers mean, and an understanding of what use the business makes of the numbers. And when the answer is really “We don’t know,” then expressing it in numbers would likely be harmful.
Hi George, thanks for this overview. I like how you balance the benefits of estimations and not even doing them. My favourite topic is “If we work on things that are an order of magnitude more valuable than the cost of producing them, then estimation of cost is less important” – this is something we are currently working on in close collaboration with the business stakeholders. Let’s see how far we can get…
Frank, if you’re working in close collaboration with your business stakeholders, I think you’ll do really well. You’ll understand what the stakeholders really need to know, and they’ll be happy with the appropriate level of precision, knowing that when the picture changes, they’ll know about it. That’s what’s important.
Great post. To elaborate your final point about assumption.
Estimates are based on assumptions, and using them as a hypothesis can provide early notice that those assumptions might be incorrect when actuals differ from the estimate.
I find that assumption often exists within a team too. I see many times, when it comes to estimation, wildly different values of story points being estimated for a single item. For a team with a cross functional skill set, this is often an indication of different assumptions. You could argue that these assumptions should have been driven out within the team during refinement of the feature, but I find that the act of estimating a number helps focus the team.
I ask myself why it is this act that focuses the team? Maybe it’s a hangover from years gone by that means the team still feels the number is a contract opposed to an estimate and as such the process gains a certain importance.
What could we do to drive out assumption within the team without the need for an estimation / contract as an easy prompt? Well certainly more refinement. However there will be a point at which the focus that estimation brings will be more valuable than further refinement.
Should estimates only be used internally for the team, done and discarded immediately. Externally predictions of the future could then be based on cycle time with WIP limits stabilising variability.
Ben, I find that developing a list of concrete examples, or acceptance scenarios, is another way to focus the team. I talked about that in my previous post, http://blog.gdinwiddie.com/2014/03/27/definition-of-ready/
Good article. I think that estimating in story points adds no value over counting stories if your stories are all reasonably small and somewhat similar in size. If you have some big stories, you’ll need to compensate for this bad practice by starting to estimate in story points.
I have also seen the zig-zagging pattern in velocity. Usually, this was due to having too large stories that got partially done in a sprint (and then added 0 points to the velocity). In the next sprint, this story got completed quickly, giving more points than average.
This pattern often leads teams to give partial credit to half-done stories, but I see that as one of the most mortal sins in Scrum. Making stories smaller is the right approach there.
Just my 2 cents, AndrÃ©