Should your tests mock outside services or not? I keep seeing discussions on this topic. On the one hand, it lets your tests be in control of the testing context. Otherwise it may be very difficult to create a reliable automated test.
- The external service might return different results at different times.
- The external service might be slow to respond.
- Using the external service might require running the test in a particular environment.
- It may be impossible to generate certain error conditions with the real service.
- There may be side-effects of using the real service.
On the other hand, our mock of the external service might differ from it in important ways.
- Our mock service might have a slightly different interface than the real one.
- Our mock service might accept slightly different parameters than the real one.
- Our mock service might return slightly different results than the real one.
- The real service might change behavior, and we won’t notice until we deploy to production.
This leaves us in a quandary. What to do? Read More
An open letter to a programmer who thinks that code coverage by integration tests eliminates the need for unit tests. Read More
Asked on the Agile-Testing mailing list:
Lesson 136: Testability is often a better investment than automation.
(I’m quoting “Lessons Learned in Software Testing” by Kaner/Bach/Pettichord)
If anyone has practical examples of improving testability, I’d be very interested to understand, and grateful. Read More
This week, Chris Matts tweeted, “Contemplating whether GIVEN-WHEN-THEN is back to front. The system should do <outcome> WHEN <event> PROVIDED <
stimulus context>… Hmmm.” Let’s try an example. “Given I have $500 in my account, when I withdraw $50 then I have $450 in my account” becomes “The system should show $450 in my account when I withdraw $50 provided I had $500 in my account before.” It doesn’t exactly roll off the tongue, does it? Putting the result first makes the sentence both more complex and more passive. Yet I can understand the impulse that triggered this tweet. Read More
Many organizations segregate their programmers and testers in order to achieve independent validation of requirements. If the system is tested according to an independent interpretation of the requirements than used for implementation, then errors in those interpretations may be detected.
This course of action is obviously prudent when the requirements are handed down to the development organization in the form of a document. Read More
Also while in Las Vegas for the ADP/West Conference, Bob Payne and I sat in the Agile Philanthropy booth and recorded a podcast on Acceptance Test Driven Development and the 3 Amigos. This is the latest in a series of Tips and Advice podcasts that Bob and I have done.
I’ve written about User Stories before and made available a handout that includes a page on splitting stories that, in addition to listing some splitting heuristics, includes several links to several lists of techniques for splitting stories.
What it doesn’t include is an even simpler way to split stories–the simplest way I’ve found yet. Read More
In the late 1970s, in the Co-Evolution Quarterly, the magazine successor to The Whole Earth Catalog, Peter Warshall stated that geodesic dome houses always leak. This was a bold and surprising statement at the time, coming from a man who was considered one of the finest builders of dome houses–ones that didn’t leak.
Why did he make this statement? Read More
Jim Shore has posted a response to the reactions about his previous post on Acceptance Testing in which he defends the way he and the teams he coaches are working. About the same time, Lisa Crispin posted her thoughts on the topic.
As Lisa says,
I can’t tell you the one right way to test and develop software”¦. The one right way for your team to code and test will continually evolve over the years. In fact, I’m sorry to tell you that you’ll never find the one right way; it’s a moving target, but you’ll get closer and closer to it.
This is an incredibly important point! There may be many “wrong” ways—wrong in that they fail to achieve your objectives—but there is no “right way.” So I’m happy that Jim and his teams are able to achieve the results they want. I’m not saying they’re doing it wrong. Read More
Recently, Jim Shore wrote about The Problems With Acceptance Testing. I like Jim, and respect him a lot. Because of my respect for his opinions, I found it quite discouraging that he said, “I no longer use [automated acceptance testing] or recommend it.” Gojko Adzic has posted his response to Jim. This is mine.
Certainly when something’s not giving you the results you want, it’s time to make a change. That change can be to drop the practice that’s not working for you. It can also be changing the way you go about the practice, or changing what you want to accomplish. Or, instead of changing, maybe the word “refining” is a better fit. Read More