| 2012-10-22 16:40:57 |
Matthew Paul Thomas |
description |
The daily error rate for an Ubuntu version (e.g. Ubuntu 12.10) is calculated as
the number of error reports received from Ubuntu 12.10 machines that day
divided by
the number of Ubuntu 12.10 machines that reported any errors in the past 90 days.
That denominator is our best guess to the number of machines that *would* have reported errors if they'd experienced any that day. But that doesn't work when the number of machines spikes or plummets.
For example, imagine that Ubuntu 12.10 was installed on zero machines until release day, October 18th. Imagine that it was suddenly installed on millions of machines on the 18th, but there were no new installations in the days afterward.
On October 18th, every Ubuntu 12.10 machine we knew about would be a machine that had reported at least one error that day. So our calculation would return an error rate of at least 1.0. That would be far higher than the real error rate, because we wouldn't be including all those machines that would have reported errors but didn't encounter any.
Following this pattern, where x = the real error rate, on October 19th the result of our calculation would drop to about x + 1/2, on October 20th it would drop to about x + 1/3, on October 21st it would drop to about x + 1/4, and so on.
Reality isn't so stark, for three reasons:
(1) a small percentage of machines now running Ubuntu 12.10 were running pre-release versions
(2) Ubuntu 12.10 wasn't released until about 6pm on the 18th
(3) even those waiting for the release didn't all install on release day.
Nevertheless, our error rate calculation shows this problem quite well:
* September 18th to October 17th, it was between 0.10 and 0.15
* October 18th (12.10 being released about 6pm), it was 0.28
* October 19th, 0.66
* October 20th, 0.56
* October 21st, 0.45
We can expect the calculation to continue declining until it reaches the "real" daily error rate (probably between 0.10 and 0.15, like it was before).
That this is the problem is backed up by looking at the error reports themselves. There is no spike in relative frequency of any individual problem that pre-release testers weren't seeing (e.g. a problem with the Ubuntu installer). There are just more reports of every problem.
This would also explain the smaller spike seen at beta 1, from an error rate of 0.08 on release day September 6th to 0.15 on September 7th.
This bug will be fixed once our calculation no longer produces an error rate that spikes on release days. A mathematician could probably give us a good way to do this. A hack that might work would be to not count a machine until it has been known for at least 1/x′ days, where x′ = the error rate for the previous day. That way its first report would not artificially push the error rate upwards. |
The daily error rate for an Ubuntu version, e.g. Ubuntu 12.10, is calculated as
number of error reports received from Ubuntu 12.10 machines that day
divided by
number of Ubuntu 12.10 machines that reported any errors in the past 90 days.
That denominator is our best estimate of the number of machines that *would* have reported errors if they'd experienced any that day. (We don't have any way of counting machines that never report errors.) But this estimate falls apart when the number of machines spikes or plummets, as it did with the release of Ubuntu 12.10 on October 18th. <https://errors.ubuntu.com/>
For example, imagine that Ubuntu 12.10 was installed on zero machines until 12:00 a.m. on release day, October 18th. Imagine that it instantly became installed on millions of machines, but there were no further installations in the days afterward.
On October 18th, every Ubuntu 12.10 machine the error tracker knew about would be a machine that had reported at least one error that day. We wouldn't be including all those machines that would have reported errors but didn't encounter any. So where x = the real error rate, our calculation would instead return an error rate of about x + 1.
Continuing this pattern, on October 19th the result of our calculation would drop to about x + 1/2, on October 20th it would drop to about x + 1/3, on October 21st it would drop to about x + 1/4, and so on.
(I'm probably wrong with that part, because I'm ignoring a bunch of things -- for example, that the machines that experienced errors previously are more likely to experience errors in future.)
Reality isn't so stark, for three reasons:
(1) a small percentage of machines now running Ubuntu 12.10 were running pre-release versions
(2) Ubuntu 12.10 wasn't released until about 6pm on the 18th
(3) even those waiting for the release didn't all install on release day.
Nevertheless, our error rate calculation shows this problem quite well:
* September 18th to October 17th, it was between 0.10 and 0.15
* October 18th (12.10 being released about 6pm), it was 0.28
* October 19th, 0.66
* October 20th, 0.56
* October 21st, 0.45
We can expect the calculation to continue declining until it reaches the "real" daily error rate -- probably between 0.10 and 0.15, like it was before.
This theory is supported by looking at the error reports themselves. There is no spike in relative frequency of any individual problem that pre-release testers weren't seeing (for example, a problem with the Ubuntu installer). There are just more reports of every problem.
This would also explain the smaller spike seen at beta 1, from an error rate of 0.08 on release day September 6th to 0.15 on September 7th.
This bug will be fixed once our calculation no longer produces an error rate that spikes after release days. A mathematician could probably give us a good way to do this.
Some possible approaches, most hackish first:
(a) Take a machine into account only once it has reported at least two errors.
(b) Take a machine into account only once it has been known for at least 1/x′ days, where x′ = the error rate for the previous day.
(c) Weight a machine's individual error count by the number of days since its first error report -- ignore it altogether on the first day, multiply it by 1/2 the next day, 2/3 the day after, 3/4 the day after that, etc. |
|
| 2012-10-22 17:07:59 |
Matthew Paul Thomas |
description |
The daily error rate for an Ubuntu version, e.g. Ubuntu 12.10, is calculated as
number of error reports received from Ubuntu 12.10 machines that day
divided by
number of Ubuntu 12.10 machines that reported any errors in the past 90 days.
That denominator is our best estimate of the number of machines that *would* have reported errors if they'd experienced any that day. (We don't have any way of counting machines that never report errors.) But this estimate falls apart when the number of machines spikes or plummets, as it did with the release of Ubuntu 12.10 on October 18th. <https://errors.ubuntu.com/>
For example, imagine that Ubuntu 12.10 was installed on zero machines until 12:00 a.m. on release day, October 18th. Imagine that it instantly became installed on millions of machines, but there were no further installations in the days afterward.
On October 18th, every Ubuntu 12.10 machine the error tracker knew about would be a machine that had reported at least one error that day. We wouldn't be including all those machines that would have reported errors but didn't encounter any. So where x = the real error rate, our calculation would instead return an error rate of about x + 1.
Continuing this pattern, on October 19th the result of our calculation would drop to about x + 1/2, on October 20th it would drop to about x + 1/3, on October 21st it would drop to about x + 1/4, and so on.
(I'm probably wrong with that part, because I'm ignoring a bunch of things -- for example, that the machines that experienced errors previously are more likely to experience errors in future.)
Reality isn't so stark, for three reasons:
(1) a small percentage of machines now running Ubuntu 12.10 were running pre-release versions
(2) Ubuntu 12.10 wasn't released until about 6pm on the 18th
(3) even those waiting for the release didn't all install on release day.
Nevertheless, our error rate calculation shows this problem quite well:
* September 18th to October 17th, it was between 0.10 and 0.15
* October 18th (12.10 being released about 6pm), it was 0.28
* October 19th, 0.66
* October 20th, 0.56
* October 21st, 0.45
We can expect the calculation to continue declining until it reaches the "real" daily error rate -- probably between 0.10 and 0.15, like it was before.
This theory is supported by looking at the error reports themselves. There is no spike in relative frequency of any individual problem that pre-release testers weren't seeing (for example, a problem with the Ubuntu installer). There are just more reports of every problem.
This would also explain the smaller spike seen at beta 1, from an error rate of 0.08 on release day September 6th to 0.15 on September 7th.
This bug will be fixed once our calculation no longer produces an error rate that spikes after release days. A mathematician could probably give us a good way to do this.
Some possible approaches, most hackish first:
(a) Take a machine into account only once it has reported at least two errors.
(b) Take a machine into account only once it has been known for at least 1/x′ days, where x′ = the error rate for the previous day.
(c) Weight a machine's individual error count by the number of days since its first error report -- ignore it altogether on the first day, multiply it by 1/2 the next day, 2/3 the day after, 3/4 the day after that, etc. |
The average daily error rate for an Ubuntu version, e.g. Ubuntu 12.10, is calculated as
number of error reports received from Ubuntu 12.10 machines that day
divided by
number of Ubuntu 12.10 machines that reported any errors in the past 90 days.
That denominator is our best estimate of the number of machines that *would* have reported errors if they'd experienced any that day. (We don't have any way of counting machines that never report errors.) But this estimate falls apart when the number of machines spikes or plummets, as it did with the release of Ubuntu 12.10 on October 18th. <https://errors.ubuntu.com/>
For example, imagine that Ubuntu 12.10 was installed on zero machines until 12:00 a.m. on October 18th. Imagine that it instantly became installed on millions of machines, but there were no further installations in the days afterward.
At the end of October 18th, every Ubuntu 12.10 machine the error tracker knew about would be a machine that had reported at least one error that day. We wouldn't be including all those machines that would have reported errors but didn't encounter any. So where x = the real error rate, our calculation would instead return an error rate of about x + 1.
Continuing this pattern, on October 19th the result of our calculation would drop to about x + 1/2, on October 20th it would drop to about x + 1/3, on October 21st it would drop to about x + 1/4, and so on.
(I'm probably wrong with that part, because I'm ignoring a bunch of things -- for example, that the machines that experienced errors previously are more likely to experience errors in future.)
Reality isn't so stark, for three reasons:
(1) a small percentage of machines now running Ubuntu 12.10 were running pre-release versions
(2) Ubuntu 12.10 wasn't released until about 6pm on the 18th
(3) even those waiting for the release didn't all install on release day.
Nevertheless, our error rate calculation shows this problem quite well:
* September 18th to October 17th, it was between 0.10 and 0.15
* October 18th (12.10 being released about 6pm), it was 0.28
* October 19th, 0.66
* October 20th, 0.56
* October 21st, 0.45
We can expect the calculation to continue declining until it reaches the "real" daily error rate -- probably between 0.10 and 0.15, like it was before.
This theory is supported by looking at the error reports themselves. There is no spike in relative frequency of any individual problem that pre-release testers weren't seeing (for example, a problem with the Ubuntu installer). There are just more reports of every problem.
This would also explain the smaller spike seen at beta 1, from an error rate of 0.08 on release day September 6th to 0.15 on September 7th.
This bug will be fixed once our calculation no longer produces an error rate that spikes after release days. A mathematician could probably give us a good way to do this.
Some possible approaches, most hackish first:
(a) Take a machine into account only once it has reported at least two errors.
(b) Take a machine into account only once it has been known for at least 1/x′ days, where x′ = the error rate for the previous day.
(c) Weight a machine's individual error count by the number of days since its first error report -- ignore it altogether on the first day, multiply it by 1/2 the next day, 2/3 the day after, 3/4 the day after that, etc. |
|
