On the Test Driven Development YahooGroup, Alan Baljeu asked, “It is my impression that whenever a feature is added, there may be many things which are affected by adding that feature…. And by not considering those things, you introduce bugs into the software.” In response, he got two types of advice. One was specific suggestions of software construction techniques that can reduce the occurrence of errors by reducing the number of places that need to change when adding a new feature.

The second type was general testing advice. Alan admitted, “I don’t see a way to properly identify what new behaviour will need to be covered. I want to be sure I’m not forgetting cases, but currently I’m overlooking too many of these effects.” In other words, the current test coverage is not catching all the errors.

It is a truism that testing cannot guarantee the absence of errors, in spite of the title of this post. It also seems apparent, to me and to Alan, that Alan’s testing has gaps in it. I suspect that he’s trying to cover all the cases with a single set of tests. As he says, “I have acceptance tests, but it is extremely hard to be thorough. I find them difficult because they tend to follow this pattern:
Do A
Do B
Do C
The following 1000 things should now be true: (in practice I check about 6, which is not enough).”

Peter Warshall, a master builder of watertight geodesic domes, once stated in The CoEvolution Quarterly that domes leak because they depend on a single technological skin that must be perfect. Conventional buildings have multiple layers of protection, so that the water that gets past the outer covering is caught by the next. Bruce Schneier says, “One of the basic philosophies of security is defense in depth: overlapping systems designed to provide security even if one of them fails.” The same approach seems to be the best for preventing bugs from reaching production.

First of all, Test Driven Development [TDD] is principally a design technique rather than a testing technique, despite its name. The fact that it leaves behind a suite of tests that may be used for regression testing is a bonus. The bare minimum of tests needed to drive the design, however, is unlikely to be sufficient to detect unwanted changes.

The next step is to add “negative tests” to your unit test suites. In other words, add tests that verify that things that shouldn’t happen, don’t. This is more of a testing frame of mind than a design one, though it may drive the creation of code that prevents bad things. A careful developer will check edge conditions and perhaps some invalid input, depending on the circumstances.

• Aside: Testers will generally think that the developer didn’t do enough of this, and they’ll probably be right. I find that testers tend to do a better job at testing and developers tend to do a better job of writing code. Developers are rarely in the position of evaluating the code of testers, however, while testers often evaluate the missed tests of developers. I’m not down on testers–some of my best friends are testers–but I do think they sometimes under-appreciate what developers do when they bemoan errors they find. I also think that developers often under-appreciate the defensive mind-set of the testers. People who are really good at both jobs seem to be rare, however. And I don’t know anybody who’s good at both viewpoints simultaneously.

Then you want some tests to ensure that the system works together as a whole. These are termed system, or integration, or acceptance, or customer tests. Certainly you want the correct operation of the main features to be tested with automated scripts. The negative tests at this level, however, would lead to combinatorial explosion if you tried to cover them exhaustively.

Hence the need for exploratory testing. Exploratory testing is interactive testing by a human tester who is trying to find ways to surface problems. Its proponents like to point out that it’s the best way to find bugs. I believe them, because the bugs surfaced by regression tests would already be fixed. At this point, if things are done well, there’s little need for testing that things are working correctly. Instead, the good tester will concentrate on things that are possible, but likely not anticipated by the developers. Uncovering the implicit assumptions, and finding holes in them, is the value of a good interactive tester.

Testers are people, too, and there’s always the possibility that some misfeature will evade all of these levels. Care and skill are needed by both testers and developers to minimize such surprises. From the developers’ viewpoint, one of the things I’ve found is that good, simple design helps to minimize the places where such problems can hide.

By minimizing coupling and maximizing cohesion, the things that change together tend to be visible together, where it’s obvious what needs to change. In many designs I see, it’s as you say: a change here ripples through the system and lots of places need to accommodate a new parameter or a new property of an object. To me, this is a point of pain, and I’ll want to clean it up. Perhaps I’ll create a parameter object that encapsulates the tuple that needs to be passed around. Perhaps I’ll move behavior depending on the object’s properties into the object, so the object’s collaborators don’t have to worry about the internal changes. These things are the right thing to do, not because someone said so, but because they reduce the incidence of errors.

Ultimately it all comes down to skill and thinking about the problem. Errors must be attacked at different levels: design, implementation, and integrations. Errors must be attacked from different viewpoints: what’s intended and what’s not intended. Like most things, practice is required to attain that skill.

Kudos to Alan Baljeu for asking these questions, and addressing the issues. That’s the necessary first step. When you’ve got a lot of areas where improvement could be made, try to focus on one or two that seem like they would offer immediate benefit. Try those, and monitor the situation for benefit. Also be alert for other aspects, and the techniques that would address them, that you may not have fully considered before.

A fundamental part of agility is to try something and pay attention to what happens. That’s called feedback.

Edit 3/29/2021: Updated yahoogroup links to groups.io