What percentage of bugs are found by rerunning tests? That is, what’s the value of this equation:
number of bugs in a release found by re-executing tests
100 X ———————————————————————— ?
number of bugs found by running all tests (for 1st or Nth time)
Like most single measures, this one is suspect. To make good decisions, we need more information. Before repeating percentages I’ve heard and read, I’ll give an example of how the context of a measure matters. I’ll compare two projects, A and B. In each, 30% of the bugs were found by rerunning regression tests.
All of project B’s tests are regression tests, designed to be repeatable. We don’t know if they’re automated or not – that’s not relevant to this example.
Some of project A’s tests are regression tests, but some are “one-shot tests”. One-shot tests are intended to be run exactly once. Nothing is done that would enable anyone to run them again. (One-shot tests are usually manual, but they can be automated – think of a quick and dirty program that you delete after running once.)
Regression tests find bugs in two ways. They can find bugs the first time they’re run, and they can find bugs when they’re re-executed. One-shot tests have only one chance to find a bug.
Here are the contexts that led to the 30% results:
Loading ...