Lesson 136: Testability is often a better investment than automation.

(I’m quoting “Lessons Learned in Software Testing” by Kaner/Bach/Pettichord)

If anyone has practical examples of improving testability, I’d be very interested to understand, and grateful.

I first encountered the issues of testability when working with integrated circuit design in the 1980s. The same issues apply to software systems.

1.  You want to be able to easily put the system into a known state. It’s best if you can get to the state you want for your test without going through a number of interactions or other states on the way.
2.  You want to be able to drive internal nodes of the system so that you can test parts in isolation. It’s much harder to test everything through the GUI, public API, IC pins, or whatever is the normal interface.
3.  You want to be able to sense internal nodes of the system so that you can test parts in isolation. Like being able to drive internal nodes, this reduces the combinatorial complexity.
4.  The special access you add for testability does add some complexity, can provide new failure modes, and might leave some paths untested. Be aware of this and consider what these issues are for your system.

Your work gets a bit easier if you consider this as part of building the system, rather than just as part of testing it. If you build your tests first, the tests will specify the state and access you need for testability. Running these tests while the system is built will result in testability appearing almost magically. Because of that, I’d say that testability trumps automation only when you’re already late in starting.

Why? Because they are an irresponsible participant in the internet community. They are damaging the internet.

I run several mailing lists hosted on DreamHost. I’ve been having problems the last couple years with bounces from AOL for subscribers who have AOL (or CompuServe or Netscape) email addresses. The reason? AOL customers have complained about spam being sent via DreamHost mail servers, so AOL frequently (and randomly, it seems) bounces all mail from a particular mail server. Now, I don’t know how many domains are hosted on DreamHost, but it’s a lot. It’s no surprise that some of them might send some spam, or have their accounts compromised and spam sent in their name. Spam, phishing, and malware emails are the scourge of the internet, but a very hard problem to solve. Blocking thousands of legitimate email senders will not address the problem. It’s gotten so bad that I had to notify my AOL subscribers (via a GMail account, since I couldn’t otherwise reach them) that it was unlikely they would receive their emails if they continued to use AOL email. The DreamHost customer support has worked on this issue numerous times, just on my behalf. It’s a losing battle. AOL is not listening. I’ve thrown in the towel at trying to keep them connected.

An event today added insult to injury. I received a couple of malware links from one of those AOL addresses, sent to me and a number of other addresses. I replied to let the owner of that account know that it was compromised. What happened? You guessed it; AOL bounced the email. It’s OK for AOL to send malware emails, but not possible for me to report it.

I don’t seem to have any way to get AOL’s attention. I’ve tried sending mail to postmaster@aol.com before, but that seems to go nowhere. I’ve tried to get their attention via DreamHost’s support department. I’ve tried to get their attention by asking their customers, whom they are inconveniencing by blocking legitimate email, to contact AOL’s support department. It all seems a dead end.

So, I’d like to see AOL come to a dead end. I’d like it if all ISPs would start bouncing all email from AOL, just as they do with DreamHost. If AOL cannot play nicely in the internet sandbox, I’d like for the rest of the internet to shun them as a pariah. I’m very tired of their pushing work on those of us trying to be good citizens of the net. Let them now become just a painful memory.

P.S. DreamHost isn’t perfect. For the price they charge, I certainly don’t expect them to be. They’re an excellent value, though, and I’ve been mostly happy with them for a number of years. If you’d like to sign up with them, use the code GD60DOLLAR and you’ll save $60, subject to their terms for the discount.

“Most people doing Agile today are actually doing Waterfall with Agile terms. Agile is dead.”

There are a lot of people talking about “post-agile” now that the word Agile has been taken up by the masses, including those selling products and services with the word without ever doing what some might consider to be truly Agile.

What did you expect?

It’s not been so many years since the mailing lists were abuzz with people suggesting Agile to anyone who would listen, and many who would not. Why the shift from pushing Agile to walking away from it? Did you think people would “see the light” and be suddenly changed?

Let’s face it, working in an Agile fashion is hard. It requires paying attention to what you’re doing and the results of that. It requires thinking and making choices. It requires honoring the thoughts and feelings of those around you. None of this is easy stuff. It doesn’t just fall into place because you’ve decided to “do Agile.”

It doesn’t “just happen” even when you’ve been working in an Agile fashion for a decade. Every day is another day of practice. I still make mistakes. I expect that; I just strive to notice and correct them earlier.

I’m often helping large organizations, or parts of large organization, move to a more Agile process. The process of making that shift is more complicated than the process of Agile, itself. When you have dozens of people affected, each one reacts differently and in a different time scale. When you have hundreds, the same is still true. Large ships turn slowly. Large organizations even more so, as they are not “one thing” to be turned.

Often the successful transformations of large organizations do not look Agile to those who work in small ones. There are many aspects that may be compromised or suboptimal in comparison to what a small organization can achieve. Yet large organizations can learn to deliver working software more reliably and more frequently, or at least determine when they can’t, much sooner. And the working lives of those involved can be much more engaging and rewarding.

Those who are abandoning Agile because “the wrong people” have gotten involved and it hasn’t transformed the world of work into the place envisioned in your dreams, what did you expect? When has anything been a broad and unalloyed success?

Take another look at the successes, at the progress that has been made, and rejoice in that. Do not be discouraged that the struggle to succeed will never cease.

Things get a little more complicated when the next use isn’t immediate, or is under the control of external events. Perhaps it depends on what links a user clicks on the UI. No problem—we build a storage mechanism a bit more complex than a variable.

What’s wrong with that?

The problem arises when the code that uses the value has to check in two places for it. Oops! We haven’t build a cache, even if we call it that. Instead the code is just stashing the value in a temporary place. The fact that our code has to check two places is a dead giveaway that we’ve violated the single responsibility principle.

Instead, let our code call for the value as it’s always done. Build a caching facade that sits in front of the expensive call and offers the same API. This facade will have only one purpose: caching those results and returning the cached value if the same request is made. Our original code will have only one purpose: whatever it’s doing that requires it to need those results.

It’s a simple thing, and it avoids a common mistake that introduces complexity and accompanying defects.

“Shipping first time code is like going into debt. A little debt speeds development so long as it is paid back promptly with a rewrite… The danger occurs when the debt is not repaid. Every minute spent on not-quite-right code counts as interest on that debt. Entire engineering organizations can be brought to a stand-still under the debt load of an unconsolidated implementation, object-oriented or otherwise.”

In this, Ward was not referring to poor programming practices, but to choices made based on the immediate needs of the time, but which had since been found to be at odds with future needs. An example of this might be a scheduling system that supports recurring events based on Gregorian calendar constructs when the situation also requires other basis such as a Lunar calendar. The concept of technical debt, and the techniques of paying down that debt, allow us to proceed with implementation of a naive or straightforward system knowing that as new requirements come to light, we can refactor to accommodate them. If we’ve built our scheduling system to have a dependency on the Gregorian calendar, we can refactor that to introduce a construct that better fits our needs, e.g. a Scheduling calendar, of which the Gregorian calendar is just one type.

I’m going to call this domain debt.

It has become popular in recent years to refer to “code smells” (a term apparently coined by Kent Beck) as technical debt. Code smells are things about the code that may not prevent it from working correctly, but that don’t “seem right” by the heuristics of an experienced programmer. These issues may span a wide variety of issues, from a method name that seems too general to give a clear understanding of what the method does, to a overly long method that does many things at many levels of abstraction. Some of these smells are stronger than others. Robert C. “Uncle Bob” Martin has identified some Principles of Object Oriented Development which are violated by some code smells. There are other “laws” of object oriented development, such as the Law of Demeter, which are generally good practice but perhaps not completely universal.

Inattention to code smells leads to a different type of technical debt. This one is based on a naive vision of implementation rather than a naive vision of the domain. It’s my experience that this debt mounts more rapidly. Taking on technical debt in terms of the domain or requirements may be analogous to taking on an automobile load so that we have reliable transportation that allows us to access gainful employment and further our goals. Taking on technical debt in terms of implementation is more analogous to incurring debt for expenses that may not be aligned with our future well-being. A marginal case would be similar to eating out at restaurants and paying with a credit card. While we need sustenance, if we spend more on it than we can afford to immediately pay, we reduce our ability to pay for necessities in the future. Sometimes it might be reasonable if we know the situation is going to change at a particular time, but often people get crushed by debt that grows unchecked in this fashion. Sometimes there are more significant acquisitions of debt such as taking a home equity loan to finance a vacation at an expensive resort. This is surely a recipe for future trouble.

I’m going to call this implementation debt.

A third category of problem that is often termed “technical debt” in the current vernacular is code that doesn’t work correctly. It may work for some cases, but miss some boundary conditions and either give incorrect answers or blow up spectacularly. I would call this a “defect” rather than a “debt.” There’s no doubt that, in most cases, such defects are related to technical debt of the second kind, implementation debt.

In the words of C. A. R. Hoare:

“There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies.”

Most forms of implementation debt obscure the purpose of code, forcing us to deal directly with the details of implementation. The goal should be to write code that clearly states both its intent and design. To do this, we must purge it of duplicated functionality and enhance its expressiveness. Note that these are the middle two elements of Kent Beck’s four rules of simple design.

There was a time when it was generally accepted that the expression of the design was in one place, and the code merely followed the design that was expressed. Such a strategy has generally been found wanting. Such a separate design is not executable, and therefore cannot be tested to ensure it has the properties we seek in the system. Even when that separate design is excellent, there is no easy mechanism to ensure that the code follows it.

Fortunately, it has been shown that it’s possible to work from the other end. It’s quite possible to express a design adequately in the code, and use separate design artifacts to explain particular points or as an aid to conversation. To do this, we must purge our code of virtually all implementation debt, so that the design is clearly visible. Then we must keep the design debt at bay, so that our design reflects our current understanding of the problem domain—at least as far as we’ve implemented.

Notes:

1. When asked about this passage, Ward told me, “I regret using the word ‘rewrite’. We did not use the word refactoring in 1992. I coined the term ‘consolidation’ later in the paper to describe the process that now goes by the term refactoring. I never rewrote more than a few dozen lines at a time.”
2. Martin Fowler has a nice article classifying types of technical debt. I don’t think I’d ever read that before writing this one.

In casual but competitive racing on a boat the size of mine, a total crew of six is generally considered a full crew. That includes a foredeck person, two jib trimmers, a main trimmer, a helmsman, and moveable ballast who can attend to anything else that needs it. Each of these jobs has specific responsibilities, but most crew members have general enough skills to jump in on other responsibilities when needed. If the crew is smaller, people can take on multiple jobs, though at some expense of efficiency.

One of the recommendations made was to get out on the racecourse an hour before the start and sail, with that crew and in those conditions, for 30 minutes to get acclimated. Then, relax and review what was learned during those 30 minutes. Finally, discuss what is the goal and strategy of the race ahead.

Looking forward at the goal and strategy ahead is very much like chartering. Every project needs a good liftoff. Ainsley Nies and Diana Larsen have just written a great book about why and how that’s necessary.

The pre-race practice at the technical skills of sailing the boat together, and the subsequent retrospective on that practice, are a fairly obvious strategy for sailboat racing. I hadn’t considered it, but I think the same approach would be beneficial for software development. What would happen if you assembled your team prior to chartering and held your own Code Retreat or other day of kata exercises? In one day, each technical member, both programmers and testers, of a small team could pair with every other on the team. They would gain familiarity with each other’s style of working and build a foundation for trust. The katas could be performed using the language and tools expected to be used in the project, but without the pressure to produce shippable functionality.

In sailboat practice, if a tack goes slowly, crew members get confused about their roles, or lines get tangled, then you can immediately stop, discuss it, and try again. In a race, the pressure is on to get things straightened out and then focus on what needs doing next. That “next thing” is often harder than it would have been because of the delay and confusion that follows a mistake. The same is true in software development.

“You can’t improve your fundamentals during a race.” This was the statement that caught my attention to the parallels between sailboat racing and software development. The lecturer recommended other practice days to hone fundamental skills such as tacking the jib, gybing the spinnaker, and making sail changes. The correlation between a race and a development project starts to break down in this metaphor, but the fact remains that we need to practice to improve. We learn far more if we practice mindfully, thinking about and discussing what we did and what we might have done, and then practice some more.

Most software projects I’ve seen make the assumption that the developers already have sufficient skill, or they should work on their skills on their own time. These things are somewhat true, which make this assumption very seductive. It’s also true that skills can always be improved, and most projects can benefit from spending time on improving those skills. How much time and how often is a balancing act decided in the context of the current skills and abilities, and the needs of the project. Of course, spending all your time on developing skills won’t get the project done, but spending zero time on developing skills is unlikely to be the fastest way to get the project done, also. Even the best teams will find there are new wrinkles to learn on every project.

And if you don’t spend time improving your fundamentals, you’ll lose the race.

  <target name="cucumber" description="Run the cucumber tests">
    <property environment="env" />
    <exec executable="${jruby.home}/bin/cucumber" >
      <env key="PATH" path="${jruby.home}/bin:${env.PATH}" />
      <env key="CLASSPATH" file="${jruby.home}/lib/jruby.jar" />
    </exec>
  </target>

I hope this helps others.

I use a simple poll of how safe people feel. I got this from Norm Kerth’s book, Project Retrospectives (page 110). Norm describes this as part of an exercise to increase the sense of safety in the room and promote honest and open speech. In the context of a day-long-or-more end-of-project retrospective, then I can see the value of such an exercise. For an heartbeat retrospective, I don’t attempt this. I’m not optimistic about significantly raising the level of trust in a short period of time. There will be other retrospectives, and there will be time to work on trust issues in between. It’s often helpful, though, to bring the issue of trust and safety to the surface.

For getting a sense of the safety in the room, I’ll ask people to rate their feeling using this scale:

1.    I’ll just keep my mouth shut
2.    I’ll agree with whatever the group says
3.    I’ll discuss the non-controversial topics
4.    I’ll discuss almost anything
5.    I’m willing to talk about anything

I have everyone write a number on an index card or sticky note as a private ballot, and fold it so the number is not visible. I will then collect the ballots (or delegate to someone who seems both trusted and without power over others) and count the number of each value. On a whiteboard or flip chart I’ll record these with hashmarks. This gives a simple obvious view of the reported safety.

Sometime I’ll point out that this is only the reported level of safety. After all, if someone doesn’t feel safe, they may not feel safe enough to answer this question openly, either. They may give then number that they think is expected, or the one they think others will give.

Retrospectives are difficult in low-trust situations. They’re also a great tool for building trust over time.

I’ve led a separate retrospective for developers (programmers and testers) at a client making an Agile transition. In that situation, it seemed to me that the full-team retrospective concentrated on “acceptable” improvements—things that were easy to propose in a corporate environment. And I noticed that a number of the developers didn’t say volunteer anything. That was a clue to me that there were things to be said that people didn’t feel comfortable saying. Given the fact that the Scrummaster was a former project manager, this dynamic wasn’t unexpected. Having a separate retrospective, with a safety exercise at the beginning, allowed some issues to be acknowledged, even if they weren’t tackled.

In my friends situation, the majority of the participants are programmers who have, for the most part, worked with each other at other companies. They know each other well and are working together at this company precisely because they get along and respect each other. That’s all well and good, but the side effect is that there’s a bit of a monoculture among these developers. They readily agree with each other within a limited viewpoint.

You would think that a retrospective would bring out other viewpoints and counteract the hegemony of this group. Unfortunately, the group had settled on “majority vote” as their standard means of deciding which topics to pursue in the retrospective. How did they choose that? I don’t know, but I’d guess they voted. Ellen Gottesdiener examines a number of group decision-making approaches in her article “Decide How to Decide.” (Software Development Magazine, January 2001) While majority vote is fast and efficient, it always results in a loser, and sometimes in a persistent loser. For retrospectives, I prefer “Spontaneous Agreement” or “Consensus.” When the group I’m facilitating can’t converge on a choice that hears all voices, I’ll sometimes fall back on “Decision Leader Decides After Discussion” to give those voices a chance.

If you’re stuck in this sort of situation and you’re not the facilitator or otherwise capable of making the decision, then feedback to the retrospective facilitator might be in order. It could be that the facilitator just hasn’t noticed what is happening. It could also be that the facilitator doesn’t yet have the skills to know what to do about the retrospective being consistently owned by a sub-group. Some helpful feedback, stated in terms of observed behavior and personal impact, might be just the thing to break that log-jam. Or, it might not, particularly when there’s a difference in organizational power between the facilitator and the person who wants to help the situation.

Having a separate retrospective of people locked out of the decision-making in the larger retrospective is an escalation of this approach. If one person has been unable to influence the facilitator’s behavior such that all voices are heard, perhaps the group of disadvantaged people can dig deeper into the problem and come up with some more effective solutions. If this group continues to feel the need for a separate retrospective, though, then it indicates a continuing problem.

That’s not a good sign for the organization. Human dynamics of a given group don’t always work out smoothly. And sometimes organizations fail.  Being deaf to voices have different information, values, or preferences is a good way to fail. The universe doesn’t depend on every organization succeeding. As an individual, there’s always Martin Fowler’s advice, “Change your organization, or change your organization.“

Given-When-Then describes the scenario in chronological order, which also makes it easier to automate. Often I don’t implement it in that order, though. Instead, I’ll flesh out the Then with code fixtures first, as the results are the primary focus. The When is usually a straightforward single interaction with the system. The Given describes the business pre-requisites, and these get translated into actions that make the When interaction work. Starting with the Given can lead us astray into specifying things that don’t need to be specified for this scenario.

Sometimes we discover that there are other pre-requisites, too. These may be things that are simply taken for granted or implied by the specified context of the test. We take for granted that the system is up and running. Having money in the account implies that the account has been opened. We also assume that there are no special cases in the context that would invalidate this scenario. The account having a hold put on the account by the fraud department is not a reasonable assumption. This is a case that needs checking, but in another scenario where the special context is explicitly mentioned.

If there are other things that need to be initialized for this test that are not mentioned in the Given setup, then it’s time to pause to reflect. Could this indicate a mismatch between the world envisioned by the business and the implementation in the system? That could be something to correct. Or might it be something that is often implicit in business conversation, but needs to be explicit in the test. Maybe the type of account affects whether or not a fee is applied to withdrawals. Sometimes the business gets into the habit of speaking in shorthand that leaves out some important distinctions.

I’m still happy with “Given <this context>, when <an interaction occurs> then <a result>” as a formula for acceptance tests or executable requirements. I don’t know that this is the only good formulation. I’m still looking for others that would naturally occur to business people, and would love to hear some examples if you have them. While I don’t think Do-When-Provided is natural business-speak (I could be wrong in some businesses), it’s good to consider alternatives.

Given-When-Then is a formula. While it can help stimulate our thoughts, it shouldn’t limit them.

In the same manner, Mike Cohn’s canonical expression of a User Story, “As a <type of user>, I want <some goal> so that <some reason>,” helps us to remember to think about which user and why they’re requesting this. I generally counsel teams not to actually write their stories in this way, as it often results in a wordy and repetitious list that obscures the essence of the story instead of highlighting it. There’s nothing like a long list of “As a customer, I want <to do something> so that I can make a purchase.” The formula can lull you to sleep. I’ve seen examples of this were “making a purchase” was not really what the customer wanted, it was the way what the customer wanted was implemented. And I’ve seen “As a user…” fill in for customer, obscuring the fact that not all users of the system are customers, and not all stakeholders who want something from the system are users.

These formulae are thinking tools. Use them in ways that provoke thinking. If they’re just chores to be done or checkboxes to be checked, they’re not helping.

One of the low points was when several people, including a business analyst, product owner proxy, and the program manager, individually said that they couldn’t alter the “user stories” to cut across multiple components of the system because they were already in the computerized planning tool (and Word documents) and it would be too much work. That team did not appear to be getting much value from their “Agile approach” and had significant integration risk that was being studiously ignored.

One of the most frequently asked questions on public mailing lists and forums devoted to Agile development is “What Agile Planning Tool should we use?” There is always a chorus of answers touting this or that computerized tool, usually without asking any questions about the context. Is there one best tool?

Given the number of such tools available, either for purchase or for free, people are still seeking that perfect tool. Like Lord Franklin seeking the Northwest Passage, some pay a heavy price for that search.

What’s the big difference between the two situations described above? The first difference is that one was using a physical planning tool and the other a computer program. Should we take that as evidence that physical tools are always preferred and computer programs should be eschewed? That would be hasty without looking deeper. Our second “why was one team more successful than the other” answer is that the first team adapted their process according to the successes and difficulties they were having. The second held to a “consistent process” when it was getting in their way.

One of the biggest problems for teams trying to learn how to be successful with Agile techniques is that they often don’t recognize what’s getting in their way. They’re often trying to do things “by the book” to learn Agile practices–which is a good starting point. But they’ve not yet got enough experience to see the early warning signs of Agile impediments, and by the time they do, they often jump to the conclusion that “Agile doesn’t work.”

Let’s look at a recent inquiry on the Agile Project Management yahoogroup. The questioner mentioned that his small team was moving to Agile/Scrum and because they were partially distributed, were looking for “a product management platform at least to enable a virtualized scrum board and some basic reports like the burn down chart, etc.”

Some people opined that, because they were distributed, only an electronic tool would work. This isn’t necessarily the case. I’ve heard several reports of successfully using physical boards duplicated at each location. I can see some real advantages to this approach. One of the biggest challenges of distributed development is keeping the assumptions and attention of each location in sync with the others. Talking through the changes on the board on a daily (or more frequent) basis is an excellent means to accomplish this. Having silent changes that happen in the background within a computerized tool while you’re not looking, is not.

As it turns out, the team isn’t really distributed. The product owner and another business stakeholder is remote to the development team. The product owner makes an effort to be regularly engaged, but wants to give more visibility to the other business stakeholder. Using a distributed planning tool for this sets off warning bells in my head. It’s making the daily work of those producing the software more difficult for the benefit of someone who is insufficiently engaged in the process. I often hear the counter argument that “it’s only a few seconds to use the tool.” That may be, but those few seconds are multiplied many times over during the course of the project. They become a real drag on productivity and lengthen the cycle times. I’ve not measured the effect, but I’ve seen it clearly. Worst of all, it eventually discourages people from making improvements to their process because they’ve got so much invested in the current incarnation.

There are more insidious effects, too, when an electronic tool is being mandated just to observe from afar without engagement. The information that goes into the tool becomes a public record, and people start worrying about how to make it look good to others rather than how to use it to get work done. This is an even greater productivity friction, and it siphons attention away from the original goal.

The product owner of this team would do far better to develop a sprintly summary for the other stakeholder. Business stakeholders don’t really need visibility into the details within a sprint, anyway. If they truly do, then the sprints are too long. Broadcasting all of the minute details of how people are working just invites micromanagement and breeds distrust.

This is just one case study. In other situations, I would call attention to different details and likely offer different advice. Covering all the cases would be far too much for a blog posting. Besides, the real point is that people need to keep their ultimate goals in mind, and not trade thoughtful observation and decisions for the magic of a virtual tool.

P.S. Tim Ottinger and Jeff Langr have published some very good advice on this topic.

There are those looking at it from the point of view of usability and fitness for use. Some, often closely aligned with these, use the word “design” to talk about the visual aspects. Others, the user experience.

Others look at the manner in which the software is constructed. Those who use the title of “Software Architect” often look at the big picture aspects–what frameworks will be used and how the system will be distributed. DBAs look at the schema of the persisted data as design. Those who write the code see design in the modules, classes, methods and functions–and how they interact.

Some people think that design is something you do before building the system, often drawing diagrams to illustrate their ideas. Some go so far as to assert that if this is not done prior, the system is not designed at all. Others maintain they can do just as well designing on the fly, mostly capturing the design directly in the code, but also describing it in conversation and sketches. Still others say the real design is “as built,” and the original intent is not so important.

People use “design” to refer to all sorts of small tidbits in the development process. They abstract some of these and call them “design patterns.”

Sometimes people use the word “design” to mean “the important stuff I do while I leave the mere implementation to others.”

All this is just about software, and is surely not exhaustive. When you add physical products to the mix, the definitions multiply. Any discussion about design that doesn’t develop a shared understanding of what is meant by “design” is sure to cause difficulties. And public utterances about design will always be interpreted differently by some people.