I’ve led a separate retrospective for developers (programmers and testers) at a client making an Agile transition. In that situation, it seemed to me that the full-team retrospective concentrated on “acceptable” improvements—things that were easy to propose in a corporate environment. And I noticed that a number of the developers didn’t say volunteer anything. That was a clue to me that there were things to be said that people didn’t feel comfortable saying. Given the fact that the Scrummaster was a former project manager, this dynamic wasn’t unexpected. Having a separate retrospective, with a safety exercise at the beginning, allowed some issues to be acknowledged, even if they weren’t tackled.

In my friends situation, the majority of the participants are programmers who have, for the most part, worked with each other at other companies. They know each other well and are working together at this company precisely because they get along and respect each other. That’s all well and good, but the side effect is that there’s a bit of a monoculture among these developers. They readily agree with each other within a limited viewpoint.

You would think that a retrospective would bring out other viewpoints and counteract the hegemony of this group. Unfortunately, the group had settled on “majority vote” as their standard means of deciding which topics to pursue in the retrospective. How did they choose that? I don’t know, but I’d guess they voted. Ellen Gottesdiener examines a number of group decision-making approaches in her article “Decide How to Decide.” (Software Development Magazine, January 2001) While majority vote is fast and efficient, it always results in a loser, and sometimes in a persistent loser. For retrospectives, I prefer “Spontaneous Agreement” or “Consensus.” When the group I’m facilitating can’t converge on a choice that hears all voices, I’ll sometimes fall back on “Decision Leader Decides After Discussion” to give those voices a chance.

If you’re stuck in this sort of situation and you’re not the facilitator or otherwise capable of making the decision, then feedback to the retrospective facilitator might be in order. It could be that the facilitator just hasn’t noticed what is happening. It could also be that the facilitator doesn’t yet have the skills to know what to do about the retrospective being consistently owned by a sub-group. Some helpful feedback, stated in terms of observed behavior and personal impact, might be just the thing to break that log-jam. Or, it might not, particularly when there’s a difference in organizational power between the facilitator and the person who wants to help the situation.

Having a separate retrospective of people locked out of the decision-making in the larger retrospective is an escalation of this approach. If one person has been unable to influence the facilitator’s behavior such that all voices are heard, perhaps the group of disadvantaged people can dig deeper into the problem and come up with some more effective solutions. If this group continues to feel the need for a separate retrospective, though, then it indicates a continuing problem.

That’s not a good sign for the organization. Human dynamics of a given group don’t always work out smoothly. And sometimes organizations fail.  Being deaf to voices have different information, values, or preferences is a good way to fail. The universe doesn’t depend on every organization succeeding. As an individual, there’s always Martin Fowler’s advice, “Change your organization, or change your organization.“

Given-When-Then describes the scenario in chronological order, which also makes it easier to automate. Often I don’t implement it in that order, though. Instead, I’ll flesh out the Then with code fixtures first, as the results are the primary focus. The When is usually a straightforward single interaction with the system. The Given describes the business pre-requisites, and these get translated into actions that make the When interaction work. Starting with the Given can lead us astray into specifying things that don’t need to be specified for this scenario.

Sometimes we discover that there are other pre-requisites, too. These may be things that are simply taken for granted or implied by the specified context of the test. We take for granted that the system is up and running. Having money in the account implies that the account has been opened. We also assume that there are no special cases in the context that would invalidate this scenario. The account having a hold put on the account by the fraud department is not a reasonable assumption. This is a case that needs checking, but in another scenario where the special context is explicitly mentioned.

If there are other things that need to be initialized for this test that are not mentioned in the Given setup, then it’s time to pause to reflect. Could this indicate a mismatch between the world envisioned by the business and the implementation in the system? That could be something to correct. Or might it be something that is often implicit in business conversation, but needs to be explicit in the test. Maybe the type of account affects whether or not a fee is applied to withdrawals. Sometimes the business gets into the habit of speaking in shorthand that leaves out some important distinctions.

I’m still happy with “Given <this context>, when <an interaction occurs> then <a result>” as a formula for acceptance tests or executable requirements. I don’t know that this is the only good formulation. I’m still looking for others that would naturally occur to business people, and would love to hear some examples if you have them. While I don’t think Do-When-Provided is natural business-speak (I could be wrong in some businesses), it’s good to consider alternatives.

Given-When-Then is a formula. While it can help stimulate our thoughts, it shouldn’t limit them.

In the same manner, Mike Cohn’s canonical expression of a User Story, “As a <type of user>, I want <some goal> so that <some reason>,” helps us to remember to think about which user and why they’re requesting this. I generally counsel teams not to actually write their stories in this way, as it often results in a wordy and repetitious list that obscures the essence of the story instead of highlighting it. There’s nothing like a long list of “As a customer, I want <to do something> so that I can make a purchase.” The formula can lull you to sleep. I’ve seen examples of this were “making a purchase” was not really what the customer wanted, it was the way what the customer wanted was implemented. And I’ve seen “As a user…” fill in for customer, obscuring the fact that not all users of the system are customers, and not all stakeholders who want something from the system are users.

These formulae are thinking tools. Use them in ways that provoke thinking. If they’re just chores to be done or checkboxes to be checked, they’re not helping.

One of the low points was when several people, including a business analyst, product owner proxy, and the program manager, individually said that they couldn’t alter the “user stories” to cut across multiple components of the system because they were already in the computerized planning tool (and Word documents) and it would be too much work. That team did not appear to be getting much value from their “Agile approach” and had significant integration risk that was being studiously ignored.

One of the most frequently asked questions on public mailing lists and forums devoted to Agile development is “What Agile Planning Tool should we use?” There is always a chorus of answers touting this or that computerized tool, usually without asking any questions about the context. Is there one best tool?

Given the number of such tools available, either for purchase or for free, people are still seeking that perfect tool. Like Lord Franklin seeking the Northwest Passage, some pay a heavy price for that search.

What’s the big difference between the two situations described above? The first difference is that one was using a physical planning tool and the other a computer program. Should we take that as evidence that physical tools are always preferred and computer programs should be eschewed? That would be hasty without looking deeper. Our second “why was one team more successful than the other” answer is that the first team adapted their process according to the successes and difficulties they were having. The second held to a “consistent process” when it was getting in their way.

One of the biggest problems for teams trying to learn how to be successful with Agile techniques is that they often don’t recognize what’s getting in their way. They’re often trying to do things “by the book” to learn Agile practices–which is a good starting point. But they’ve not yet got enough experience to see the early warning signs of Agile impediments, and by the time they do, they often jump to the conclusion that “Agile doesn’t work.”

Let’s look at a recent inquiry on the Agile Project Management yahoogroup. The questioner mentioned that his small team was moving to Agile/Scrum and because they were partially distributed, were looking for “a product management platform at least to enable a virtualized scrum board and some basic reports like the burn down chart, etc.”

Some people opined that, because they were distributed, only an electronic tool would work. This isn’t necessarily the case. I’ve heard several reports of successfully using physical boards duplicated at each location. I can see some real advantages to this approach. One of the biggest challenges of distributed development is keeping the assumptions and attention of each location in sync with the others. Talking through the changes on the board on a daily (or more frequent) basis is an excellent means to accomplish this. Having silent changes that happen in the background within a computerized tool while you’re not looking, is not.

As it turns out, the team isn’t really distributed. The product owner and another business stakeholder is remote to the development team. The product owner makes an effort to be regularly engaged, but wants to give more visibility to the other business stakeholder. Using a distributed planning tool for this sets off warning bells in my head. It’s making the daily work of those producing the software more difficult for the benefit of someone who is insufficiently engaged in the process. I often hear the counter argument that “it’s only a few seconds to use the tool.” That may be, but those few seconds are multiplied many times over during the course of the project. They become a real drag on productivity and lengthen the cycle times. I’ve not measured the effect, but I’ve seen it clearly. Worst of all, it eventually discourages people from making improvements to their process because they’ve got so much invested in the current incarnation.

There are more insidious effects, too, when an electronic tool is being mandated just to observe from afar without engagement. The information that goes into the tool becomes a public record, and people start worrying about how to make it look good to others rather than how to use it to get work done. This is an even greater productivity friction, and it siphons attention away from the original goal.

The product owner of this team would do far better to develop a sprintly summary for the other stakeholder. Business stakeholders don’t really need visibility into the details within a sprint, anyway. If they truly do, then the sprints are too long. Broadcasting all of the minute details of how people are working just invites micromanagement and breeds distrust.

This is just one case study. In other situations, I would call attention to different details and likely offer different advice. Covering all the cases would be far too much for a blog posting. Besides, the real point is that people need to keep their ultimate goals in mind, and not trade thoughtful observation and decisions for the magic of a virtual tool.

P.S. Tim Ottinger and Jeff Langr have published some very good advice on this topic.

There are those looking at it from the point of view of usability and fitness for use. Some, often closely aligned with these, use the word “design” to talk about the visual aspects. Others, the user experience.

Others look at the manner in which the software is constructed. Those who use the title of “Software Architect” often look at the big picture aspects–what frameworks will be used and how the system will be distributed. DBAs look at the schema of the persisted data as design. Those who write the code see design in the modules, classes, methods and functions–and how they interact.

Some people think that design is something you do before building the system, often drawing diagrams to illustrate their ideas. Some go so far as to assert that if this is not done prior, the system is not designed at all. Others maintain they can do just as well designing on the fly, mostly capturing the design directly in the code, but also describing it in conversation and sketches. Still others say the real design is “as built,” and the original intent is not so important.

People use “design” to refer to all sorts of small tidbits in the development process. They abstract some of these and call them “design patterns.”

Sometimes people use the word “design” to mean “the important stuff I do while I leave the mere implementation to others.”

All this is just about software, and is surely not exhaustive. When you add physical products to the mix, the definitions multiply. Any discussion about design that doesn’t develop a shared understanding of what is meant by “design” is sure to cause difficulties. And public utterances about design will always be interpreted differently by some people.

Oops! There are no photos beyond last March. I felt sick to my stomach. That’s a lot of family memories to evaporated. Whew! The program we’ve been using to download the cameras also makes a backup copy local to the machine used to download. I’ve got all the pictures back on the photo server, and now just have to sort them into directories again.

But first, I need to fix the backup script. Why didn’t it warn me when it started to fail?

I copied a version on a different fileserver to that machine and edited it for the different source and destinations. It’s a simple shell script that mounts the USB drive, runs rsync to duplicate the data, and then unmounts the USB drive. I’d run into a problem once before where the drive wasn’t getting mounted properly, so I’d added some script to detect that and email me using msmtp if it failed. That only worked for mount errors, though, not for rsync errors.

As I hacked in detection and notification of rsync errors, I felt ill at ease. I was duplicating some of the script to send emails. The script was getting harder to read. The script was increasingly inconvenient to test. And I was nervous about introducing new errors.

I started practicing Test Driven Development a decade ago. It has become second-nature to me to test-drive the code I write. Why was this code just hacked together? Mostly because I hadn’t been thinking of it as code. It was just a script. At first, it just replicated the commands I typed on the command-line. I put those into a script to make it easy to run, and then to automate running it via the <del>chron</del> cron daemon. When I replicated it to another machine, I extracted the paths to variables to make it easier to change the differences. When it had failed to mount a drive, I’d added a check for that and a line to send me an email. That line surely crossed the border between “a file of commands” to “a program,” but I didn’t notice it at the time. Or, if I did notice it, I didn’t pay attention to it. After all, I don’t generally program in bash scripts and haven’t developed any rigor for doing so. It’s those “little” changes that “can’t possibly” hurt that bite you the hardest.

The script now “seems to be working,” but I’m not satisfied. Wouldn’t it be nice to get advance notice when the disk is getting close to full, rather than waiting until it is?

I now know I’m programming in sh, not just collecting a bunch of command-lines from my .history file. Adding the email notification of rsync errors was harder than I expected due to tiny errors made in an unfamiliar programming environment. In addition to being hard, it just felt “dirty.”

A little googling lead me to shunit2. After a little difficulty creating a tiny virtual disk under control of the tests, I quickly had a function that extracted the % full output using the du command and returned it as a number. Great! Now I can use that number to provide an early warning. And working test-driven is teaching me better shell scripting habits.

I’ve still got work to do, pulling the existing script into tested functions, but already I feel better. I’m on my way to a reliable and maintainable script.

• More asphalt than projected due to a math error
• More temporary concrete dividers than planned, as plans called for only half what was needed
• Unanticipated excavation costs.

It’s true that no one likes for costs to exceed estimates, but exceeding estimates is a very different thing than driving up costs. Those costs were accurate—it was the estimates that did not agree with later reality. Two of these items are mistakes, but the third is apparently a bit of information that was not known when the estimate was made. Had these mistakes not been made, and the excavation needs known in advance, the result would have been the same costs.

The same is true for estimates on your software projects.

On the other hand, the article describes other work where the state engineers “never did any quality control review that would allow them to stop and correct things instead of letting the contractors run up the cost.” This is a real problem.

As projects progress, you learn things—or, at least, you should. You learn that some of your estimates were wrong. You learn there were things you didn’t know. And you learn when the project is in the weeds and no longer producing the expected value.

You learn these things by paying attention to what happens. And, to distinguish the estimation problems from the actual problems, you need to pay attention separately to costs and value. Old-fashioned Earned Value Management gives me heartburn because it makes the simplifying assumption that these are the same—that the value you’ve received on an incomplete project is roughly proportional to what it’s cost you. That just doesn’t seem to be true. Not only are there cases like this one where true progress is stalled but the expenses continue, there are well-run projects that deliver the bulk of the value early using shrewd prioritization of the work.

I don’t know much about highway construction, but I would imagine that it would be possible to measure actual progress by completed miles of roadway. I also imagine the work is not usually done that way. I suspect that it’s common to grade the entire project, then lay down the roadbed, then pave it all, then paint the lines, …

People often work the same way in software development, in architectural layers, so there’s nothing of value until the end when all the pieces are integrated. The result of such a big-bang integration is that problems are hidden until it’s way too late. It’s a great recipe for cost (and time) overruns. Such overruns like to grow in the dark places where no one is watching.

The sad thing is that it’s quite easy to do better. Instead of working in layers or components, implement functional slices of the requirements. The layers and components can be built bit by bit as more and more functionality is completed. And each functional slice provides a usable road that goes somewhere.

The pace of work can be more easily monitored by working in functional slices, also. It’s relatively easy to check whether a function works or not. The sum of functionality can be tracked over time to see how fast the work is proceeding. The total amount of functionality can be estimated to give an estimate of when the work will be done. I like a simple hand-drawn burn-up chart for this, as it doesn’t mislead with unwarranted precision.

And if the rate of completion is less than you hoped, or the cost of the work is higher than anticipated, or hidden work is uncovered, then you can detect it early and make appropriate contingency plans—very similar to what the “top-performing states” do when managing highway projects.

Hat for Test Driven Development

He included a photo of me wearing my Red-Green-Refactor TDD hat. This had may be more famous than I am, having been worn by Uncle Bob Martin in Episode 6 of his CleanCoders videos.

I hope this will be done by then.

These are sad and scary words to me.  When I hear them in the context of a software development project, I associate them with a project unlikely to meet expectations.  Further, it’s likely that there is no good indicator of what can be delivered when.  With no hard evidence to tell them what’s possible, and to judge the effects of changes they try, people fall back on a strategy of hope. They no longer feel in control of the situation.

We’ve done our part.

This signals the underlying hopelessness. When you can’t make things come out the way you want, when you can’t even predict how they will come out, one natural response is to give up. Often the cultural expectations don’t allow saying “this project is in trouble.” The person who does so is the person who will attract all the scrutiny. Without the clear data to back up the perception of trouble, why stick your neck out? Others should be noticing the same thing; let them call attention to it. And when things don’t work out as “hoped,” if weve “done our part,” then someone else will get the blame.

You can’t solve a problem that no one wants to see. Instead, we’ve found an acceptable way to fail.

I spent a chunk of my career where I often worked solo on projects that might get enhanced every year or two. This taught me a lot about the importance of code readability and what sort of things helped me quickly understand and avoid overlooking important considerations. I’ve also worked on a lot of legacy code bases, so I’m well aware of the pain and problems created by code that’s not readable, or is not organized in a helpful manner.

Last week at the AgileDC Conference, someone told me they were working on code I had written ten years ago. My first question was, “Are the unit tests still working?” This code was the first production code I ever wrote test-first, and it taught me a lot doing so. I was glad to hear that they had been maintained. When I was writing them, I couldn’t get anyone else interested in test-first development.

Even more gratifying was hearing that they’d recently used some of my code as an example in a discussion of object-oriented programming. The example they’d used was where I’d wrapped an integer in a class. Why would I do that? Because it wasn’t really an integer. It was a database ID. The integer was just an artifact of the database automatically creating ID values using a sequence. It would make no sense to perform arithmetic on it. And there were IDs for various entities that should not be used interchangeably.

The integer value was intended to be used only within the Data Access Object that read and wrote the database.

A few years ago, I was asked if I could change an algorithm in some embedded assembly code that I’d written more than ten years prior, and I hadn’t even seen for over ten years. Setting up the development environment wasn’t easy, as much of the hardware was no longer manufactured. The code, however, was not hard to modify. I had implemented the algorithm using state machines and data tables, and had created semi-readable constants for all of the “magic numbers” used.

Code is read much more frequently than it is written. Writing it clearly and expressively pays off when we have to read it next week or next month. Well-written code is also much more likely to be useful for decades into the future. If you write code only for the short term, it’s likely to have only a short life. People will find it necessary to rewrite it rather than maintain it for the long term.

I’d like to share with you an interview conducted by Bill Fox for his 5 Minutes to Process Improvement Success project. My interview, “Measure Progress in a Way that’s Visible and Reliable,” is found on page 69 of the PDF.  You’ll also find interesting interviews with Karen Base, Kevin Schaaff, Hillel Glazer, Scott Ambler, Neil Potter, Bob Payne, Mike Bonamassa, Mario Hyland, Jeff Dalton, Paul E. McMahon, Karl Wiegers, Mary Lynn Penn, Ally Gill, Alan Shalloway, and Tom Cagley. And there are more to come in the future.