In cases where the team have only had automated recording available – or perhaps, only the skills to use automated recording tools – we’ve performed the scenarios manually, and watch them fail (for the right reasons) before we start coding. This helps us get the precision you talk about and gets the conversations going – and it can start as simply as going to a web page and getting a 404 where we “expected” our new screen (see my blog post about this: http://lizkeogh.com/2007/11/16/bdd-bug-driven-development/ )

After the code has been written, the recording tools can be used to record the pages in action.

In some ways I’ve found this even better than using automated frameworks, because it keeps the code easy to change throughout the development cycle and even tests that are written rather than recorded are often easier to do after the code’s there.

The Rate of Development is a function of several key steps, of which acceptance testing is only one piece – see approximation (A.1 – D.1) below. [This model may not agree with everyone’s experience, but it can work for a mature product undergoing typically bite size changes so as to not lose hard core users or take too long between releases. New products would typically combine many steps since they are given a broader mandate from management and have minimal legacy code and users.]

Rate of Development = Function of

[(A1. rate of new marketable idea creation + buy-in from PO's, CEOs or man/woman in the street) +
(A.2 rate of resource allocation - scrum team(s) or other) +
(B.1 rate of requirements generation - user stories, formal spec or other) +
(B.2 rate of coding and manual unit test passing and/or automated acceptance test passing - Agile or other means) +
(B.3 rate of help updates) +
(C.1 rate of combined build test passing) +
(C.2 rate of release test passing - automated mostly) +
(D.1 rate of product release - determined by business calendar and contractuals)]

Possible reasons why folks are not automating acceptance tests to include the current release content…

1. Because work flow items other than manual unit testing take up so much organizational bandwidth and time. Perhaps they play a far greater role in determining the Rate of Development and so there is no motivation to change.

2. Manual unit testing the fresh code you just worked on can be quick and efficient. Use existing automated testing for stale code you or others worked on before that may not be understood or is too vast. [Whether a mix of manual and automated acceptance testing is better than all automated is open to debate.]

3. Automation is added after release “1.1” to the acceptance test for release “1.2” . This would include only major areas of user stories from release “1.1”. This work is done while activities A.1-B.1 are in progress for “1.2”, so not a critical path item is introduced.

It has been my experience that there are a few different situations and reasons (not necessarily good nor bad) why automated acceptance tests are not used widely.

* Automated acceptance testing tools have not had the exposure that xUnit and commercial test automation tools have had. I teach students how to use tools such as Fit and StoryTestIQ all the time. Before I showed them that these tools exist they had no idea.
* Some teams automate acceptance-like tests through the use of xUnit tools and put less emphasis on the customer involvement. Unfortunately, it is common that organizations I have worked with do not put emphasis on customer collaboration or don’t know how it can work effectively. The thought of being involved in something so technical seems scary until they see how it provides them value.
* The gap between coders and testers is still quite wide outside of the small percentage of hard-core Agile teams. It is common for me to work with clients who have isolated silos of testers away from coders. Coders do not believe they should have anything to do with testing and the feeling is usually mutual from the tester’s point of view. Closing this gap is essential to taking on an acceptance testing mindset, in my opinion.

The idea of confidence in work is a question of degree from my experience working with teams. Some teams thoughts on what is possible in terms of increased “confidence” is a much lower bar than teams that I work on who are hard-core Scrum with XP technical practices and been doing it for some time. It is easy to forget what it was like before I worked in this way and there is a large gap between those who have consumed Agile as a way of doing their work and those who are stumbling into it and those who are not aware or resistant to it.

I have started using the terms “Executable Design” for TDD plus additional tooling to guide design incrementally and “Executable Specifications” to describe automated tests such as are created with Fit, RobotFramework, and StoryTestIQ. This helps me move past much of the baggage that “test” comes with in terms of dysfunctional industry cultural views of the testing function. I then direct discussions towards the thought of “quality” using this terminology and it seems to get a foothold even in very traditional organizations. It is too bad that these types of “word-plays” matter so much but sometimes change involves terminology, too.

Can’t wait to hear other people’s responses to this topic. I think it is quite interesting that automated acceptance testing is not more popular.