Duplicate oops reports logged for each checkwatches oops

[Diogo Matsubara]

A duplicate OOPS was logged (OOPS-1748C1004) when the checkwatches script logged OOPS-1748CCW994

This is likely due to the change on r11637: "Generate OOPS reports when cronscripts log WARN or higher, and log unhandled exceptions in scripts. Improve logging."

[Deryck Hodge (deryck)]

Marking this HIGH per zero-oops policy, but AIUI, we're waiting on info from stub about the possibility of disabling the logging he added when a script implements it's own OOPS logging.

[Robert Collins (lifeless)]

The answer should be : don't log a high severity message if you also generate an OOPS.

[Robert Collins (lifeless)]

@deryck this doesn't need to be high, its not adding new oopses - if
you have no oopses, you'll still have none without whatever change is
needed here.

I agree it would be nice to fix promptly to get good signal out of the
oops system, but I'd probably put fixing checkwatches oopses higher
than addressing this issue.

[Stuart Bishop (stub)]

If the new OOPSes are good enough, tear out the custom OOPS logging code so we don't duplicate work.

If the new OOPSes are not good enough, add a flag to LaunchpadCronScript that disables adding the OopsHandler to the root Logger. Open a bug on what is missing, because we are duplicating work. Or just fix the OopsHandler.

Alternatively, we can just filter the duplicates in the reports - the new OOPSes are unique enough to filter with no false positives.

Note that if the global OOPS reporting is filtered or disabled we will not get OOPSes for unhandled exceptions (such as lockfile errors or other error conditions in the infrastructure).

[Diogo Matsubara (matsubara)]

Robert, I disagree. I think this should be high because this is making the lpnet oops report useless with all the noise from the checkwatches OOPSes.

Stuart, how are the new OOPSes unique enough to filter them in the reports? I'm not in favor of filtering them in the reports as we'd basically be hiding the problem which is the same oops recorded twice.

[Deryck Hodge (deryck)]

We can do an easy fix in checkwatches to make sure we don't have the duplicate OOPS. I'll do a branch sometime next week and fix this. Maybe even on the plane to UDS, if I get all my other hacking done. :-)

[Robert Collins (lifeless)]

@Diogo useless is relative - we have oops, we have a zero oops policy;
I agree the report could be more useful without the 2-times distortion
here, but useless means that looking at the report has no value - and
thats simply not true.

[Diogo Matsubara (matsubara)]

@Robert, sorry, useless was a too strong word. The current situation makes the daily report very difficult to read with all the noise.

[Stuart Bishop (stub)]

The duplicate OOPS reports are certainly unique enough to filter them. The request variables are the most obvious difference.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r11889 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/11889>.

[Deryck Hodge (deryck)]

I'm marking this qa-untestable. There's no easy way to test it and issues with staging/qastaging would prevent any run of this script currently. The script runs fine locally, tests pass, and it's low impact just changing the logging level. We can confirm after deploy if the double oops is fixed and errors are still logged.

Cheers,
deryck