Comments on: Testing that no errors remain http://blog.gdinwiddie.com/2007/01/18/testing-that-no-errors-remain/ Effective software development Mon, 13 Oct 2008 08:51:26 +0000 http://wordpress.org/?v=2.0.6 by: gdinwiddie http://blog.gdinwiddie.com/2007/01/18/testing-that-no-errors-remain/#comment-105 Tue, 13 Feb 2007 04:27:32 +0000 http://blog.gdinwiddie.com/2007/01/18/testing-that-no-errors-remain/#comment-105 Thanks for the comments. Keith, I agree 100% on capturing the bug in an automated test in preparation for fixing it. Kelly, I'm not sure I'd push for a linear measure along any axis, even education. What's really important for finding bugs is a different set of assumptions, so you don't have the same blind spots. Merlyn, mutation testing is another useful technique. I'm not sure it's as cost-effective as exploratory testing by a good tester, however. Perhaps if a good tester isn't available, it would be a reasonable substitute. And I think 100% test coverage is a false goal. It's not a bad thing to have, but chasing after that can lead you away from more important issues. Thanks for the comments.

Keith, I agree 100% on capturing the bug in an automated test in preparation for fixing it.

Kelly, I’m not sure I’d push for a linear measure along any axis, even education. What’s really important for finding bugs is a different set of assumptions, so you don’t have the same blind spots.

Merlyn, mutation testing is another useful technique. I’m not sure it’s as cost-effective as exploratory testing by a good tester, however. Perhaps if a good tester isn’t available, it would be a reasonable substitute. And I think 100% test coverage is a false goal. It’s not a bad thing to have, but chasing after that can lead you away from more important issues.

]]> by: Merlyn Albery-Speyer http://blog.gdinwiddie.com/2007/01/18/testing-that-no-errors-remain/#comment-89 Sat, 10 Feb 2007 03:53:10 +0000 http://blog.gdinwiddie.com/2007/01/18/testing-that-no-errors-remain/#comment-89 I'd like to mention test code coverage reports and mutation testing as guides to potential problem areas. Mutation testing tools I know of are Jester for Java and Heckle for Ruby. I’d like to mention test code coverage reports and mutation testing as guides to potential problem areas.

Mutation testing tools I know of are Jester for Java and Heckle for Ruby.

]]> by: Kelly Anderson http://blog.gdinwiddie.com/2007/01/18/testing-that-no-errors-remain/#comment-18 Mon, 22 Jan 2007 19:14:05 +0000 http://blog.gdinwiddie.com/2007/01/18/testing-that-no-errors-remain/#comment-18 When automated testing is done correctly by both development and testing, finding bugs in this manner should become increasingly difficult. It will require better and better testers with more and more information and understanding of the program. I think that all automation pushes us towards needing better educated people. Those who run the robots at GM are not the same caliber as their grandfathers who turned wrenches for Henry Ford. Similarly, testers of the future will have to have higher qualifications than those today. The days of "monkeys on keyboards" testing may be coming to an end. When automated testing is done correctly by both development and testing, finding bugs in this manner should become increasingly difficult. It will require better and better testers with more and more information and understanding of the program. I think that all automation pushes us towards needing better educated people. Those who run the robots at GM are not the same caliber as their grandfathers who turned wrenches for Henry Ford. Similarly, testers of the future will have to have higher qualifications than those today. The days of “monkeys on keyboards” testing may be coming to an end.

]]> by: keith ray http://blog.gdinwiddie.com/2007/01/18/testing-that-no-errors-remain/#comment-15 Thu, 18 Jan 2007 19:26:02 +0000 http://blog.gdinwiddie.com/2007/01/18/testing-that-no-errors-remain/#comment-15 And... if exploratory testing finds a bug, See if you can write automated acceptance tests and unit tests that fail because of the bug. Then fix the bug. Then the tests should pass. If the bug comes back, those tests should start failing again. And… if exploratory testing finds a bug, See if you can write automated acceptance tests and unit tests that fail because of the bug. Then fix the bug. Then the tests should pass. If the bug comes back, those tests should start failing again.

]]>