An article in USA Today (December 12, 2012) about highway projects in New York has the sub-head, “Design errors, planning lapses drove up costs more than 14%.” Among the things listed that “drove up costs” are

• More asphalt than projected due to a math error
• More temporary concrete dividers than planned, as plans called for only half what was needed
• Unanticipated excavation costs.

It’s true that no one likes for costs to exceed estimates, but exceeding estimates is a very different thing than driving up costs. Those costs were accurate—it was the estimates that did not agree with later reality. Two of these items are mistakes, but the third is apparently a bit of information that was not known when the estimate was made. Had these mistakes not been made, and the excavation needs known in advance, the result would have been the same costs.

The same is true for estimates on your software projects.

On the other hand, the article describes other work where the state engineers “never did any quality control review that would allow them to stop and correct things instead of letting the contractors run up the cost.” This is a real problem.

As projects progress, you learn things—or, at least, you should. You learn that some of your estimates were wrong. You learn there were things you didn’t know. And you learn when the project is in the weeds and no longer producing the expected value.

You learn these things by paying attention to what happens. And, to distinguish the estimation problems from the actual problems, you need to pay attention separately to costs and value. Old-fashioned Earned Value Management gives me heartburn because it makes the simplifying assumption that these are the same—that the value you’ve received on an incomplete project is roughly proportional to what it’s cost you. That just doesn’t seem to be true. Not only are there cases like this one where true progress is stalled but the expenses continue, there are well-run projects that deliver the bulk of the value early using shrewd prioritization of the work.

I don’t know much about highway construction, but I would imagine that it would be possible to measure actual progress by completed miles of roadway. I also imagine the work is not usually done that way. I suspect that it’s common to grade the entire project, then lay down the roadbed, then pave it all, then paint the lines, “¦

People often work the same way in software development, in architectural layers, so there’s nothing of value until the end when all the pieces are integrated. The result of such a big-bang integration is that problems are hidden until it’s way too late. It’s a great recipe for cost (and time) overruns. Such overruns like to grow in the dark places where no one is watching.

The sad thing is that it’s quite easy to do better. Instead of working in layers or components, implement functional slices of the requirements. The layers and components can be built bit by bit as more and more functionality is completed. And each functional slice provides a usable road that goes somewhere.

The pace of work can be more easily monitored by working in functional slices, also. It’s relatively easy to check whether a function works or not. The sum of functionality can be tracked over time to see how fast the work is proceeding. The total amount of functionality can be estimated to give an estimate of when the work will be done. I like a simple hand-drawn burn-up chart for this, as it doesn’t mislead with unwarranted precision.

And if the rate of completion is less than you hoped, or the cost of the work is higher than anticipated, or hidden work is uncovered, then you can detect it early and make appropriate contingency plans—very similar to what the “top-performing states” do when managing highway projects.