18.1. Assessing Test Suite Quality

Engineers and managers like to quantify things, and the quality of a test suite is no exception. To measure the quality of a test suite, various coverage metrics have been developed and deployed in practice. Best known is code coverage, where we check individual statements for whether they have been executed in a test suite. Obviously, if a test suite does not execute a statement, this calls for trouble. Suppose the statement has a bug that triggers a failure each time it is executed. If the test never reaches that statement, the bug will never be found—at least not during the test.

Besides code coverage, there are more advanced criteria to aim for: branch coverage criteria ensure that each branch is taken at least once, and condition coverage criteria ensure that each (sub)condition evaluates once to true and once to false. All these criteria are easy to measure and are frequently used as quality targets: “We must reach at least 90% code coverage in all modules!”