The Reality of Automated Acceptance Testing

Recently, Jim Shore wrote about The Problems With Acceptance Testing.  I like Jim, and respect him a lot.  Because of my respect for his opinions, I found it quite discouraging that he said, “I no longer use [automated acceptance testing] or recommend it.”  Gojko Adzic has posted his response to Jim.  This is mine.

Certainly when something’s not giving you the results you want, it’s time to make a change.  That change can be to drop the practice that’s not working for you.  It can also be changing the way you go about the practice, or changing what you want to accomplish.  Or, instead of changing, maybe the word “refining” is a better fit.

In my experience, teams that don’t do automated acceptance testing quickly get to a point where adding new features goes slower and slower, just because it takes longer and longer to test all of the functionality.  Sometimes they start trying to figure out which functionality they need to retest, and which “couldn’t possibly be broken by this change.”  This starts down a very slippery slope.  As Elisabeth Hendrickson says,

If the customer has an expectation, they have expressed that expectation, they have every reason to believe you have already fulfilled that expectation, they don’t want to have to manually go re-verify that you have actually done the thing that you said you did before.

Is that so much to expect?

In one of Jerry Weinberg’s books [Quality Software Management: Vol 2, First-Order Measurement, pp. 156-158], he talks about how a one-line change is more likely to introduce a defect than a larger one.  This seems counter-intuitive, until you consider how people drop their guard when they think they’re making a trivial change.  They don’t check everything as completely.  When someone doesn’t check some aspect because “it couldn’t possibly be affected” by this change, they leave the door wide open for Murphy to demonstrate his law.

Given that I’m convinced things need to be retested, and that the shorter the iteration the more frequently they need to be retested, I’m not willing to give up on automated tests.  That leaves me looking at what I want to accomplish and how I want to practice it.

Jim describes two problems with automated acceptance tests.

These two problems–that customers don’t participate, which eliminates the purpose of acceptance testing, and that they create a significant maintenance burden, means that acceptance testing isn’t worth the cost.

About the first problem, lack of customer participation, he says,

The planned benefit of those tools is that the customers are supposed to write the examples themselves, thus improving communication between customers and programmers. In practice, I found that customers (a) weren’t interested in doing that, and (b) often couldn’t understand and didn’t trust tests that were written by others. Typically, responsibility for the tests gets handed off to testers, which defeats the whole point.

I also have found that customers are not interested in writing tests, themselves.  I don’t blame them for that.  Nor do I think they would, in general, do a very good job by themselves.  Sending testers (or developers, for that matter) off by themselves to translate discussions into test examples isn’t any better.  They have the same problems that face the customer, plus the problem of the customer not trusting tests they don’t understand.

I don’t, however, think that’s the way it has to be.

I’m still working on demonstrating long term results, but I’ve had some success using automatable examples as a tool to enhance the conversation between the customer, the tester, and the developer.  Not to replace the conversation, or to send one person off to write tests, but to add precision to the agreement about what needs to be accomplished.  The goal is for the customer to look at the test and say “Yes, that’s what I want.”

Once we’ve got that, there are lots of tools available now, and getting better all the time, to help us turn that automatable example into an automated one.  Yes, maintenance is still hard.  Dale Emery has an excellent article on this topic.  We, as an industry, have far to go getting this difficulty under control.  I say, let’s work on it.

If we approach the development of the examples with the customer and in the terms of the customer, then we’ve accomplished the hardest part.  It’s worth spending the extra effort to put these examples to work preventing defects rather than finding them after the fact.

11 Replies to “The Reality of Automated Acceptance Testing”

  1. My sentiments exactly!

    I wonder if people just get impatient. It takes months and even years of experimenting with different approaches, learning how to help customers illustrate desired behavior with examples and turning those into tests, knowing how much detail to get into with tests before coding starts… it’s an evolution. No magic happens overnight.

    My team’s been doing this 6+ years and we still are refining, every sprint.

  2. My thoughts after reading yours, Ron’s and Gojko’s responses as well as Jim Shore’s original post is could the experiment perhaps for him to try would be to use pair test creation (much like an XP pair programming)? Have the customer and the knowledgeable tester work through the test as a pair. The customer has teh trust in what was written, but didn’t actually do the “hard” part in their mind.

    As someone who is trying to get a team to strive towards automating their acceptance tests, seems lik increasing the conversation around concrete examples is the way to go.

  3. Thank, you George! All the more pressing now, really, that those of us who wish to rescue this practice get together and attempt to harvest best practices for launching it for a team. I hope that in Nashville we can come closer to speaking about it with one voice. I have seen it work wonders. How sad that powerful thoughtleaders are publicly abandoning it.


  4. I cannot see anything in either of the two objections stated above that cannot be overcome with sufficiently skilled testers and the appropriate level of communication.

    “Typically, responsibility for the tests gets handed off to testers, which defeats the whole point” – I would question why this defeats the whole point. In more traditional development processes an analyst would be present who would be skilled in converting the requirements in the language of the business into the appropriate models and requirements to drive the development of the system. If a tester possesses the necessary analytical skills then why not rely on them to create the appropriate tests based on the requirements communicated from the cusomer. To put it another way – if your analyst could create a specification that tested itself you would be very happy, so why not furnish the tester with analytical skills to allow them to achieve the same?

    In terms of the maintenance costs, I have been running with an in-house developed automated test harness for 3 years now. There is an overhead on maintenance but with careful consideration towards maintaining the tests separately from the executable code we have managed to overcome some significant rework of application functionality without having to rework large numbers of tests. Again this does require the necessary skills within the test team to treat the regression harnesses as applications in their own right, and design them with maintenance in mind.

    I would reiterate Lisa’s point that it’s an evolution, and there are obstacles to overcome, many of which we are stil tackling. I do not see, however, any of the problems as sufficient reason to give up on automated acceptance testing, which are the one of the cornerstones of our current development process.


  5. I gather from the links that James Shore is simply reflecting on the value of automating acceptance tests, not automated testing.
    He values the business collaboration associated with identifying the concrete examples – not the automation beyond a unit level.
    But surely there is still business value in being able to provide some sort of automated end to end regression acceptance testing?

    1. Pete, It may be important to consider that most orgs that Jim Shore works with are still in the throes of Agile adoption. Automated end-to-end regression testing is not generally the first thing I try to instill, either.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.