• More asphalt than projected due to a math error
• More temporary concrete dividers than planned, as plans called for only half what was needed
• Unanticipated excavation costs.

It’s true that no one likes for costs to exceed estimates, but exceeding estimates is a very different thing than driving up costs. Those costs were accurate—it was the estimates that did not agree with later reality. Two of these items are mistakes, but the third is apparently a bit of information that was not known when the estimate was made. Had these mistakes not been made, and the excavation needs known in advance, the result would have been the same costs.

The same is true for estimates on your software projects.

On the other hand, the article describes other work where the state engineers “never did any quality control review that would allow them to stop and correct things instead of letting the contractors run up the cost.” This is a real problem.

As projects progress, you learn things—or, at least, you should. You learn that some of your estimates were wrong. You learn there were things you didn’t know. And you learn when the project is in the weeds and no longer producing the expected value.

You learn these things by paying attention to what happens. And, to distinguish the estimation problems from the actual problems, you need to pay attention separately to costs and value. Old-fashioned Earned Value Management gives me heartburn because it makes the simplifying assumption that these are the same—that the value you’ve received on an incomplete project is roughly proportional to what it’s cost you. That just doesn’t seem to be true. Not only are there cases like this one where true progress is stalled but the expenses continue, there are well-run projects that deliver the bulk of the value early using shrewd prioritization of the work.

I don’t know much about highway construction, but I would imagine that it would be possible to measure actual progress by completed miles of roadway. I also imagine the work is not usually done that way. I suspect that it’s common to grade the entire project, then lay down the roadbed, then pave it all, then paint the lines, …

People often work the same way in software development, in architectural layers, so there’s nothing of value until the end when all the pieces are integrated. The result of such a big-bang integration is that problems are hidden until it’s way too late. It’s a great recipe for cost (and time) overruns. Such overruns like to grow in the dark places where no one is watching.

The sad thing is that it’s quite easy to do better. Instead of working in layers or components, implement functional slices of the requirements. The layers and components can be built bit by bit as more and more functionality is completed. And each functional slice provides a usable road that goes somewhere.

The pace of work can be more easily monitored by working in functional slices, also. It’s relatively easy to check whether a function works or not. The sum of functionality can be tracked over time to see how fast the work is proceeding. The total amount of functionality can be estimated to give an estimate of when the work will be done. I like a simple hand-drawn burn-up chart for this, as it doesn’t mislead with unwarranted precision.

And if the rate of completion is less than you hoped, or the cost of the work is higher than anticipated, or hidden work is uncovered, then you can detect it early and make appropriate contingency plans—very similar to what the “top-performing states” do when managing highway projects.

I invest a lot in my own professional development every year. I attend conferences such as this one. I read. I converse with colleagues.

My career has spanned a number of decades, and I expect to continue to do so indefinitely. I gave a talk at XPDay Manhattan in 2007 on Sustainable Career where I explored this topic. To do so, you not only need to continue learning, you need to learn things that have a long half-life. Learning specific technologies may be valuable, but those technologies quickly become obsolete. Be sure to also learn things with lasting value, such as the principles behind specific techniques.

You need for your career to last a lifetime. Invest in yourself.

I mentioned my aspirations to my father, who was a chemistry professor. “What do you know about valence?” he asked.

“What’s ‘valence’?”

He proceeded to explain about electron clouds and the tendency of atoms to fill or empty their outer ring of electrons.

“So the valence of oxygen is 2.”

“Yes, except when it’s 1 or 4 or 6 or some other value. It’s not always simple.”

I’ve been thinking about that conversation since the end of the Agile 2011 Conference. Kevlin Henney gave a great keynote talk about, among other things, learning about code by reading successful code. “We’re not wired up to learn from failure.”

Yes, that’s absolutely right.

Then Linda Rising gave her keynote (which I unfortunately missed, having to leave for the airport) where where exhorted us to learn from failure. “Try again. Fail again. Fail better! – Samuel Beckett”

Yes, that’s true, too. How can I agree with two seemingly contradictory points?

That’s what made me think about my plans to invent the next-generation battery. Had I continued with those plans, I could have tried 10,000 combinations and been no nearer to success than when I started.

Yet Thomas Edison did invent a successful light bulb. After many failures, he was reported to say, “I have not failed 10,000 times. I have successfully found 10,000 ways that will not work.” If that was all he knew, then he’d have been little better off than I was with my battery. In fact, he’d learned more than he admitted.

During the course of those 10,000 experiments, he built up a model of what was needed and how various materials did and did not fit that model and supply those needs. If we’re paying attention and are very persistent, we can learn from most series of failures as long as each attempt varies from the others.
It’s just not very efficient that way.

We can turn a series of failures into a string of successes quite simply. Instead of expecting each attempt to produce a working light bulb or battery or whatever it is we want, expect each attempt to be an experiment that will teach us something and refine our model. A successful experiment is one that teaches, whether or not it confirms our expectations.

Looking at it that way, we’ll want to design better experiments to maximize the learning rather than trying to minimize the number of trials. This is the heart of “Fail better!”

Teams need time to acclimate to new knowledge. They need to try it on their own, making decisions without immediate help. Otherwise they come to depend on the coach making the decisions, and they don’t learn how to make them, themselves. I’ve seen this happen when I’ve been working too steadily with one team.

It’s also important to limit the presentation of new information to the rate at which it can be absorbed. Time the team spends practicing without the presence of the coach is an important part of this absorption. Without such “soak time,” the team will get lost in the details, trying to climb the proficiency ladder without learning to practice the simple things fluently.

If you try to go faster than you can, you’ll only end up going slower.

The more interesting question, to my mind, is how can we achieve a better result. For that, we need to dig deeper into the root causes rather than stopping with blame allocation.

• Did the coach or organization have limited understanding of Agile software development?
• Did they do things that are counter to Agile software development?
• Did they do the right things, but not do them well?
• Could they tell the difference between these two?
• Could they tell, along the way, whether they were improving or diminishing their ability to get “the job” done?

Certainly, a good and experienced coach can help them see sooner and more subtly whether things are helping or hurting. That’s part of “seeing the invisible stuff.” If you’re trying an “all at once” transition, that help is imperative. Why?

Consider an analogy with driving at night. The old adage is “Do not drive beyond your headlights.” At any moment, your headlights could reveal something that you want to avoid. In the distance between where you are and the limit of illumination, you need to be able to react to that something. If you’re going too fast, you won’t have time to react within that distance, and you’ll likely hit whatever you’d rather avoid.

Knowledge and experience help you to observe. They direct your observations to the right things, and they help you observe more subtle nuances of these things. In that respect, they’re like better headlights, and they allow you to go faster, safely.

So, should you find a knowledgeable and experienced coach and “bet the farm” on a successful Agile transition? I wouldn’t. Even with good references, I would be loathe to put my organization at risk depending on the skill of an outsider. And even with a good coach, my organization has to learn to do the right things well.

Instead, I’d look for a coach who, in addition to coaching my organization, coaches me in making transitions—especially this particular type of transition. I’d want them to improve my ability to see where I’m going, not just take their word that everything is OK. And while I’m learning that, I might want to start making the changes a little more gradually such that I don’t put my organization at risk while I’m learning how to make this transition.

And since that’s how I’d want to approach an Agile transition if it were my organization, that’s how I coach.

Much of my coaching depends on helping them observe these varied conditions and how the results of their actions are affected by them. Their current skills work fine when the conditions match the way they practice them. When conditions change, the same actions fail. Without keen observation, the cause of that failure is a puzzle.

I’m teaching them to see the wind. It’s invisible. It’s vital for sailing. And it’s constantly changing, both in velocity, direction, and turbulence. But it’s so hard for them to see.

I show them the cat’s paws on the water, and the wind line where the water darkens. I show them how to ease the sail out until it starts to luff and then harden. I point out the trees, and the nearby sailboats, that disturb the wind and rob it of power. These are some of the ways they can actively seek the invisible and make it observable. As they grasp these techniques and remember to do so, they learn to teach themselves to see the wind.

With practice and time, they’ll learn to notice these cues, and even more subtle ones that are difficult to describe–like the way the boat feels when the wind shifts aft and forward. They’ll learn how to react to these cues smoothly, moving tiller and sheet together in an instantaneous reaction to changing conditions.

Well, some will. Some may become great sailors. Most, perhaps all, will become competent. Right now, there are some who seem not yet ready for these lessons. I run out of different ways to tell them. With only a week, there is little I can do to remedy the situation. But I also note that one of the strongest sailors this year was struggling last year. Where she once was timid and hesitant, she now is confident. Even when conditions are challenging, she continues to teach herself how to handle the situation and doesn’t give up.

Coaching sailing is an easy parallel to coaching software development. There’s a similar aspect of teaching techniques and when they apply. There’s a similar aspect in learning when to react to changing conditions, how to react, and how much. And the building of confidence with growing successes is very similar.

And above all, there’s the similarity of learning to observe that which cannot be directly seen. And learning how to observe the effect of our actions in response to those significant but invisible changing conditions.

…blaming… …placating… …anger… …guilt… …stress… …resistance… …denying… …avoiding… …blocking… …deny… …avoid… …anxiousness… …vulnerability… …fear…

This post is, in part, to demonstrate to him that the work of Virginia Satir is not focused on the negative.  Mostly it’s to share, and rejoice in, the freedom we have to reach our goals.

The Five Freedoms

The freedom to see and hear what is here instead of what should be, was, or will be.

The freedom to say what one feels and thinks, instead of what one should.

The freedom to feel what one feels, instead of what one ought.

The freedom to ask for what one wants, instead of always waiting for permission.

The freedom to take risks in one’s own behalf, instead of choosing to be only “secure” and not rocking the boat.

This passage comes from Virginia Satir’s book, Making Contact.

It is the fifth freedom on which I want to focus, for this is the freedom of joyful change.  I can try new things, taking the risk that they might fail.  If they do, then I own that failure. But that risk of my failure is balanced by the likelihood of my success.  I weigh the balance and make my own choice.

I do not need the protection of others.  I do not need someone else to make the decision and force a change on me, or deny a change from me.  The choice is mine alone.

As long as failure is only a risk and not a sure thing, then there is also a chance of success.  That success is also mine.

Success often does not come with the first attempt.  Often we must fail a little.  We may try small experiments, so as not to bet the farm on a single roll of the dice.  We risk that which we can afford to lose.  If we lose, we learn from it and try with this new knowledge.  This is a strategy almost sure to produce success in time.

And what a sweet success that is.  Success not handed to us by someone else’s choice.  Success not borne of attempting the sure thing.  But success achieved by our own efforts and intellect.  Success that teaches us far more than any instant success.

And this success, and new knowledge, is all ours.  Isn’t that a joyful thing!

Consider the case of these newly trained CSMs realizing that this is just the beginning of the journey, and continuing to learn. They’re more likely to help projects succeed, in that case, if the organization is already competent at developing software using empowered teams in an iterative-incremental fashion. But the material they learn in the CSM class is not sufficient for helping an organization change to become competent at that—not unless they’re already quite close.

In fact, in my experience, learning Scrum is just the tip of the iceberg for many organizations who send their employees to CSM class. Usually they’re not just trying to educate more people about a process they’re already doing. They’re trying to change their process to something more successful than what they’re doing. In addition to learning a new process, they also often need to learn

• How to bring about change
• How to build and maintain productive teams
• How to successfully delegate many decisions to those doing the work
• How to divide the work in small functional increments
• Engineering practices that support working in small increments
• Engineering practices that promote the quality needed for sustainable development

These are things that go beyond the Certified Scrum Master course. They go beyond the Certified Scrum Product Owner and Certified Scrum Developer courses, too. These courses teach useful things, but not nearly enough to accomplish the change needed to make Scrum successful in most organizations. Scrum certification, and the courses behind it, are a solution only to a tiny part of the problems that most organizations face when adopting Agile software development. That’s why bringing in a coach who knows more than just Scrum is almost a necessity.

Teamwork

The fundamental practice of teamwork remains essential.  Agile succeeds in large part by reducing wasted effort, and much wasted effort is the result of various people pulling in different directions.  Even slightly different directions will create enough friction to noticeably slow the project.  It’s well worth spending some time and effort on building an effective team within the context of the project.  The two practices most essential for supporting teamwork in an Agile transition are working with a whole team in a common area.

Whole Team

While collaborative teamwork among all the developers, for example, is beneficial to any project, it’s not enough to accomplish a transition to Agile.  At a core, you’ll need high-bandwidth collaboration between your Product Owner, Scrummaster, and Developers.  This is the minimum, in Scrum terms.  In most organizations, Developers really includes both programmers and testers; in some it includes user experience designers.  Yours may include people with other titles, such as architect, database administrator, security officer, system administrator, operations engineer, technical writer….

Your project community will also consist of other stakeholders, such as executives, customers, end users, support personnel, other development teams, facilities maintenance…. There are a lot of people with interest in the project or the things that the project team is doing.

The boundary between the core team and the rest of the project community is rarely distinct.  One useful distinction is whether a person is focused principally on the project, or whether the project is just one of many things with which the person is concerned.  This distinction is not without peril, however.

All too often an organization will assign multiple projects to people where it would be much better if they were focused on one.  If you give your Product Owner, your programmers, or your testers multiple simultaneous projects, you’re likely headed for trouble.  You may be headed for trouble if you multi-task in some of the other specialties, but there’s more variation from one context to another.  As a rule of thumb, if there is any way to let someone focus on only one project, then you should do so.

Common Area

Building a team, which is different from a mere work group, takes a lot of work.  With skill, you can facilitate that teambuilding to help it happen more reliably and a little faster.  If you don’t have a skilled facilitator, though, you’re still in luck.  People have practiced self-organizing into small groups for their common good for millennia. If they’re working together for a common goal in a common area, distinct from those working on other goals, they’ll likely build a gelled team without an outside facilitator.  There are many reasons for this, from trust building to osmotic communication.

“But we can’t do that,” I hear you cry!  Why not?  Is this project, this Agile transition, so unimportant that you can’t sit together?

“Our Product Owner has an office elsewhere, and needs to attend to other things.” Maybe you’ve chosen the wrong product owner.  Maybe the Product Owner can sit in the team room for part of the day.

“Our project team is scattered around the globe.”  Don’t do that.  While it’s possible for people scattered around the globe to work together, it’s much harder.  The odds of making that work while also making an Agile transition are slim to none.  If you must do distributed projects, set up a collocated team at each location and minimize the working dependencies between the teams.

“We don’t have a space that supports that.”  Find or make one.  Move furniture.  Take down cube walls.  Set up temporary or movable partitions.  If the furniture arrangement is more important than the project, consider canceling the project.

Visible Progress

In order to see whether or not we’re headed toward our goals, we need visibility into the progress.  Usually, this boils down to the questions “Are we going to get everything done?” and “Are we going to get done in time?”  If the answer to one or more of these questions is “no” then we can make adjustments to how we’re doing things, how much (and which things) we expect to get done, and what time we expect to have them done.  We can subdivide this visibility into two simple practices: taking baby steps and getting things done.

Baby Steps

Everything we do, from releasing interim versions down to validating the code we write, we should do in small steps.  Working in small steps gives us a much finer-grained view of the progress.  It lets us view the initial trends sooner.  It lets us view changes in the trends sooner.  It gives us more opportunities to make corrections or course changes.  It gives us more practice and comparing where we’re going with where we want to go, so we get better at doing that.  It requires less work before we see that we’re doing something wrong, or doing the wrong thing, therefore there is less work to be thrown away and redone.

Done

There’s little point in measuring something as progress unless we know that it actually contributes toward reaching our goal.  Measuring amount of work done doesn’t tell us that.  Consider this scene from the movie, Captain Ron:

Captain Ron: We should be okay. ‘Cause I know we’re near land.
Martin Harvey: Great, Cap. Great. Ya hear that? We’re almost there. Explain to the kids how you know that, Captain Ron. Someone translate for General Armando.
Captain Ron: Alright, now stay with me: When we left, we had just enough fuel to make it to San Juan. And we are out of fuel.

The gap in that logic is immediately obvious, but plan-driven projects attempt the equivalent all the time. We must measure relative to the goal, not relative to the path we’ve taken.

It’s important that we measure functional slices. It’s often tempting to measure tasks that have been accomplished. Sometimes, however, these tasks turn out to be unnecessary, and they don’t contribute to accomplishing the goal.  Sometimes, while they look right in isolation, they turn out to have flaws when we look at the bigger picture.  Sometimes they don’t quite meet with other tasks, and there is a gap to be filled between the tasks.  Such a gap in unknown undoneness.

We need ways of seeing progress that won’t like to us about how much progress we’ve made, or how much is left to do.  We want to measure functional slices that go all the way through the system.  It’s easy to neglect some of the things that people think of “someone else’s responsibility” after the system is “complete.”  These things might include an installation procedure, or deployment to a different computing environment.  Sometimes people neglect something so basic as testing that the system works.

We want to see functional progress–something we can demonstrate and test, something we know is desirable.  Expending the amount allotted is not such an indication.  Getting most of the way to the goal is no guarantee that we’ll get the rest of the way there–particularly when the remaining part is fundamentally different than what we’ve been doing.  We want to always, or at least at frequent intervals, be able to stop and have something that can be used, even if incomplete.

Putting these together can be difficult for a team making a transition to Agile.  It’s hard enough to get everything working, but much more so when you’re having to do that all the time.

Fortunately there are a lot of well-known engineering practices to support combining baby steps with getting to done at each step.  I’m starting with the assumption that the team is already capable of creating working software using whatever techniques they already know.  Eventually they’ll need to learn other practices to support this new way of working, and even to get better in general.

Continuous Improvement

Being able to work as a team and measure progress in a visible manner is only the start.  At first, we’ll do these things in a rather rough manner.  As the system grows in size and complexity, that rough manner won’t sustain forward progress.  We’ve got to get better in order to keep moving.

And there’s no limit to how good we can get.  No matter how practiced we are, there’s further improvement we can make.  If we conscientiously practice our techniques, we’ll see places where we can improve them.  We can then practice those improvements, so they become a part of our routine, and we do them even when distracted by the latest disaster.  When things go wrong, it’s no time to ease off on the practices that make us more effective.

As a team, we need to take stock from time to time to see the longer term trends, both good and bad, and the issues that only become visible in retrospect.  Holding retrospectives at various tempos–each iteration, each quarter, each release–will give us various views into what we’re doing, and result in different insights.  We must use these insights to plan ways in which to improve in the future.

From these starting points of teamwork and visible progress, and proceeding with a program of continuous improvement, we can learn all of the practices that support Agile development.

Agile requires the courage to make mistakes in front of others and the maturity to admit them when they happen.

Surely this is true.  Agile is all about paying attention to what happens and adjusting for it.  We learn most when when thing are a little off from what we want, we notice that, and we correct for it.

Brian Marick responded

I now think that Agile requires the creation of an environment in which admitting mistakes doesn’t require courage.

Surely this is also true.  If we want people to notice when things are a little off, so we can make a correction, then we have to make it easy to talk about our mistakes.  If we don’t, the little mistakes get swept under the carpet and we continue on our way–until the mistakes grow so big that we can’t ignore them any more.  By this time we’re way off the path we want.

In other words, well, in my words,

I think Agile requires creating environment where admitting mistakes requires no more courage than you have.

We can’t eliminate the need for courage to admit our mistakes.  It’s human nature to gloss over even tiny ones, and hope no one noticed.  But we can reduce the stigma of making mistakes.

How fascinating!

was the comment by Willem Larsen.  Willem and Evan Gardner are spreading Evan’s “Where Are Your Keys” language fluency game.  It’s expected that you’ll make mistakes when learning a language.  The response to such a mistake is “how fascinating” signed by raising both hands in the air.  Everyone laughs.  No one feels badly for making a mistake.  In fact, it feels good to share your mistakes with a laugh and helpful colleagues rather than to hide it inside.

Is that the way it feels when you make a mistake in your team?  What could you change so it does feel that way?