Free Trial

Safari Books Online is a digital library providing on-demand subscription access to thousands of learning resources.

Share this Page URL

Chapter 18. Seeding Bugs to Find Bugs: B... > Watching the Watchmen - Pg. 249

The second problem with metrics is that not only are bugs unevenly distributed, but risk is, too. In every project there are some modules in which defects have serious consequences-- because they are frequently used or because the entire functionality depends on them. Again, you would like to allocate your testing efforts based on this risk, rather than achieving a specific coverage. (If you were testing a commercial airplane, you would also spend much more time on the engines than on the seats.) As a good tester, you are probably already aware of defect and risk distributions, but here's a third and final problem. Coverage metrics only tell you something about the test execution-- but not about the test itself. As an example, consider the JUnit test in Example 18-1: it creates a file, writes a text into it, and then closes it. If anything goes wrong, the file methods will throw some exception, and the test will fail. EXAMPLE 18-1. An insufficient file I/O unit test import static org.junit.Assert.*; void testFileWrite() { BufferedWriter out = new BufferedWriter(new FileWriter("outfilename")); out.write("aString"); out.close(); } This is how many systems are actually tested in practice: if it does not crash, it is probably fine. Unfortunately, this is not sufficient, as the test fails to check the result. It may well be that the file methods create a wrong file, write some random output, or do nothing at all. With such insufficient tests in place, we may easily achieve 100% coverage of any test criterion but fail to catch even the simplest defect. What the test should do is at least reopen the file and check its contents--but how would we recognize this? How do we determine how well a test does its job? This is an instance of Plato's old problem: "Quis custodiet ipsos custodes?" or "Who watches the watchmen?" Fortunately, there is a way to test a test, a way so simple and straightforward that it can easily qualify as "beautiful." Watching the Watchmen A common way to test the quality of quality assurance is to simulate a situation in which quality assurance should trigger an alarm. For instance, to test a watchdog, one could check whether the dog actually gives an alarm when facing a (staged) intruder. To test the quality assurance of a clothes company, one could insert a faulty cloth into the production chain and check whether quality assurance finds it. Secure systems are routinely tested using penetration testing, simulating an attack from a malicious source. SEEDING BUGS TO FIND BUGS: BEAUTIFUL MUTATION TESTING 249