The Reality of Automated Acceptance Testing
Recently, Jim Shore wrote about The Problems With Acceptance Testing. I like Jim, and respect him a lot. Because of my respect for his opinions, I found it quite discouraging that he said, “I no longer use [automated acceptance testing] or recommend it.” Gojko Adzic has posted his response to Jim. This is mine.
Certainly when something’s not giving you the results you want, it’s time to make a change. That change can be to drop the practice that’s not working for you. It can also be changing the way you go about the practice, or changing what you want to accomplish. Or, instead of changing, maybe the word “refining” is a better fit.
In my experience, teams that don’t do automated acceptance testing quickly get to a point where adding new features goes slower and slower, just because it takes longer and longer to test all of the functionality. Sometimes they start trying to figure out which functionality they need to retest, and which “couldn’t possibly be broken by this change.” This starts down a very slippery slope. As Elisabeth Hendrickson says,
If the customer has an expectation, they have expressed that expectation, they have every reason to believe you have already fulfilled that expectation, they don’t want to have to manually go re-verify that you have actually done the thing that you said you did before.
Is that so much to expect?
In one of Jerry Weinberg’s books [Quality Software Management: Vol 2, First-Order Measurement, pp. 156-158], he talks about how a one-line change is more likely to introduce a defect than a larger one. This seems counter-intuitive, until you consider how people drop their guard when they think they’re making a trivial change. They don’t check everything as completely. When someone doesn’t check some aspect because “it couldn’t possibly be affected” by this change, they leave the door wide open for Murphy to demonstrate his law.
Given that I’m convinced things need to be retested, and that the shorter the iteration the more frequently they need to be retested, I’m not willing to give up on automated tests. That leaves me looking at what I want to accomplish and how I want to practice it.
Jim describes two problems with automated acceptance tests.
These two problems–that customers don’t participate, which eliminates the purpose of acceptance testing, and that they create a significant maintenance burden, means that acceptance testing isn’t worth the cost.
About the first problem, lack of customer participation, he says,
The planned benefit of those tools is that the customers are supposed to write the examples themselves, thus improving communication between customers and programmers. In practice, I found that customers (a) weren’t interested in doing that, and (b) often couldn’t understand and didn’t trust tests that were written by others. Typically, responsibility for the tests gets handed off to testers, which defeats the whole point.
I also have found that customers are not interested in writing tests, themselves. I don’t blame them for that. Nor do I think they would, in general, do a very good job by themselves. Sending testers (or developers, for that matter) off by themselves to translate discussions into test examples isn’t any better. They have the same problems that face the customer, plus the problem of the customer not trusting tests they don’t understand.
I don’t, however, think that’s the way it has to be.
I’m still working on demonstrating long term results, but I’ve had some success using automatable examples as a tool to enhance the conversation between the customer, the tester, and the developer. Not to replace the conversation, or to send one person off to write tests, but to add precision to the agreement about what needs to be accomplished. The goal is for the customer to look at the test and say “Yes, that’s what I want.”
Once we’ve got that, there are lots of tools available now, and getting better all the time, to help us turn that automatable example into an automated one. Yes, maintenance is still hard. Dale Emery has an excellent article on this topic. We, as an industry, have far to go getting this difficulty under control. I say, let’s work on it.
If we approach the development of the examples with the customer and in the terms of the customer, then we’ve accomplished the hardest part. It’s worth spending the extra effort to put these examples to work preventing defects rather than finding them after the fact.