Design for Testability
Asked on the Agile-Testing mailing list:
Lesson 136: Testability is often a better investment than automation.
(I’m quoting “Lessons Learned in Software Testing” by Kaner/Bach/Pettichord)
If anyone has practical examples of improving testability, I’d be very interested to understand, and grateful.
I first encountered the issues of testability when working with integrated circuit design in the 1980s. The same issues apply to software systems.
- You want to be able to easily put the system into a known state. It’s best if you can get to the state you want for your test without going through a number of interactions or other states on the way.
- You want to be able to drive internal nodes of the system so that you can test parts in isolation. It’s much harder to test everything through the GUI, public API, IC pins, or whatever is the normal interface.
- You want to be able to sense internal nodes of the system so that you can test parts in isolation. Like being able to drive internal nodes, this reduces the combinatorial complexity.
- The special access you add for testability does add some complexity, can provide new failure modes, and might leave some paths untested. Be aware of this and consider what these issues are for your system.
Your work gets a bit easier if you consider this as part of building the system, rather than just as part of testing it. If you build your tests first, the tests will specify the state and access you need for testability. Running these tests while the system is built will result in testability appearing almost magically. Because of that, I’d say that testability trumps automation only when you’re already late in starting.
Thank you George for writing this post. I’ve been saying the same many times but here and there people seem to be opposed to the idea because it may change ‘architecture’ so they prefer to invest into driving everything through the UI.
Stephan, I wonder about such comments. What is the architecture for if not to help produce a correctly functioning system?
I remember back in the 1980s programming a GenRad 2272 “bed of nails” rig to do “in-circuit” testing of ICs that had been installed on printed circuit boards. In order to test a chip, I had to gain control of its inputs. But most of the chips I wanted to test had other chips “upstream” from them, sending signals to the chip I wanted to test.
Most of our boards had a resistor somewhere on each chip that, when shorted, would disable the chip. So for any chip I wanted to test, I would find the upstream chips and disable them.
But some of our designers didn’t know about designing for testability, or didn’t care about it. The chips on their boards were harder to test, because I couldn’t easily control the chip I needed to test. I’d turn an input pin off, and some damned upstream chip would try to turn it on, and vice versa. That made those tests unreliable.
So before I ever thought about writing automated tests for software, I knew the benefits of making it possible to isolate the components one from the other.
Hi George,
I’m coming to this conversation a little late but i’ve have seen great results when focusing on testability. I was involved in a project where over the course of 12 months we went from a situation where the the entire engineering team was involved in regression cycles lasting 7 weeks to a situation where we could release after a couple of engineers doing 2-3 days of exploratory testing. Focusing on Testability revolutionised the way we delivered software and significantly improved the quality.