Predicting total number of bugs based on number of bugs revealed [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.

---

开发者_运维技巧

Want to improve this question? Update the question so it focuses on one problem only by editing this post.

Closed 5 years ago.

Improve this question

Assuming n testers were testing the same application for a given period. Each tester found a given set of bugs (Some of the bugs were detected by more than one tester).

For example:

Tester 1 found bugs {1,2,3,4,5} Tester 2 found bugs {3,5,6,7} Tester 3 found bugs {1,3,5,8,9,10}

Assuming all bugs have equal probability to be detected, can I estimate how many undetected bugs are there is my application?

Edit

Even more challenging: How can I calculate the probability of having x undetected bugs?

---

Here is a blog post I wrote on this problem: How many errors are left to find?

Of course no model can tell you precisely, but this is better than a finger in the wind.

---

Assuming all bugs have equal probability to be detected, can I estimate how many undetected bugs are there in my application?

Your assumption is not true, but, to answer your question.

For example:

Tester 1 found bugs {1,2,3,4,5} Tester 2 found bugs {3,5,6,7} Tester 3 found bugs {1,3,5,8,9,10}

You have 10 known bugs.

Tester 1 found 50% of the bugs, Tester 2 found 40% of the bugs, and Tester 3 found 60% of the bugs.

Multiplying the 3 numbers together (.50 x .40 x .60), yields .12

You can estimate that you've found 12% of the bugs, or that there are 85 more bugs to find.

So, why such a low number?

We're calculating the probable number of bugs remaining.

Let's take another example. Suppose your 3 testers found the same 6 bugs. The probability would be high that they found all the bugs.

And that's what multiplying does. Multiplying 1 x 1 x 1 yields 1.

Let's take a much worse example. Suppose your 3 testers found 6 unique bugs each. We have to assume that there are many more bugs out there, since no one found the same bug.

And that's what multiplying does. Multiplying .33 x .33 x .33 yields .04 or 4% of the bugs found.

I know that seems like a low number. But 4% is a conservative estimate when 3 people find 6 unique bugs each.

---

Appoint a Project Saboteur, see http://c2.com/cgi/wiki?ProjectSaboteur for details. The basic idea is that you deliberately inject N random bugs, then go through a test cycle. Your testing will detect some fraction of those N, and by dividing the number of known bugs by that fraction, you can estimate the total number of bugs. There are lots of gotchas to the method, of course.