Ensure whoopsie sends complete crash files to the error tracker

[Evan]

[Impact]
Apport was modifed in bug 989698 to stop showing the information collection process if the crash database (launchpad) doesn't accept a type of report. Data would be collected in the background and whoopsie would just upload the crash. This led to a situation where whoopsie might upload the .crash file to daisy before the information collection was done. Subsequently, there are error reports missing valuable data like the package version and release of Ubuntu.

[Test Case]

[Regression Potential]

We do not appear to be getting many crash reports for the current development release, according to the API:

https://errors.ubuntu.com/api/most-common-problems/Ubuntu%2012.10/today

The DistroRelease field is definitely set to "Ubuntu 12.10", so this is likely not a problem of it showing up under "Ubuntu 12.10 (development release)" or "Ubuntu Quantal".

[Brian Murray (brian-murray)]

Would it be possible that if they have the same signature the crash report would be bucketed with the 12.04 version of it? So we are only seeing new crashes (ones that didn't occur in 12.04) in the above url.

[Brian Murray (brian-murray)]

I've also found an oops without DistroRelease field:

https://errors.ubuntu.com/oops/00834b04-b1a2-11e1-85ef-e4115b0f8a4a

That'd probably cause them not to appear.

[Brian Murray (brian-murray)]

And a bunch of errors show up here:

https://errors.ubuntu.com/api/most-common-problems//today

[Brian Murray (brian-murray)]

Looking at the data from now we can see the following:

{"count": 21, "function": "/usr/bin/unattended-upgrade:IOError", "package": "unattended-upgrades", "first-seen": "0.76", "last-seen": "0.76", "report": "1004792"},
{"count": 25, "function": "/usr/bin/unattended-upgrade:SystemError", "package": "unattended-upgrades", "first-seen": "0.76", "last-seen": "0.76", "report": "908346"},

Given that this version of unattended-upgrades is in precise it may be an error with reports from both precise and quantal.

[Brian Murray (brian-murray)]

I was wondering if this might be an issue with the crash report file creation process so searched all the bug descriptions of foundation's packages and did not find any bug reports with ProblemType and not a DistroRelease field in them. So it seems like it may be something with daisy or errors.

[Evan (ev)]

> Would it be possible that if they have the same signature the crash report
> would be bucketed with the 12.04 version of it? So we are only seeing new
> crashes (ones that didn't occur in 12.04) in the above url.

The bucketing is done based on the crash signature and does not factor in the
Ubuntu version. This is by design as the same exception causing code can live
in multiple versions in multiple operating system releases.

The most common problems report shows the buckets that have the highest totals
for a given combination of fields. That is, given any combination of the
Ubuntu release, source package name, and period (day, month, year), what
buckets have seen instances of a problem with that combination of fields more
than 20 times.

This is taken from the DayBucketCount Column Family, which is keyed on a set
of the aforementioned fields, separated by colons, with the individual column
names being the crash signature (bucket ID) and their values being their
current count:

http://bazaar.launchpad.net/~ev/oops-repository/whoopsie-daisy/view/head:/oopsrepository/oopses.py#L150

[Evan (ev)]

> I've also found an oops without DistroRelease field:
>
> https://errors.ubuntu.com/oops/00834b04-b1a2-11e1-85ef-e4115b0f8a4a
>
> That'd probably cause them not to appear.

Yes, there are a number of crashes in the database that for whatever reason do
not have release fields. The code is mostly careful to step around this by not
trying to use the release field as part of the previously mentioned field
combination:

http://bazaar.launchpad.net/~ev/daisy/trunk/view/head:/utils.py#L24

[Brian Murray (brian-murray)]

Looking at the one error I found without DistroRelease again, https://errors.ubuntu.com/oops/00834b04-b1a2-11e1-85ef-e4115b0f8a4a, we'll notice that is also missing the Package, Dependencies among other things.

[Brian Murray (brian-murray)]

Investigating this a bit I remebered that some applications, like aptdaemon and ubiquity, write their own .crash files in /var/crash. Apport's collect_info function would then gather things like DistroRelease and Package.

Having aptd crash I've verified that the .crash file is missing information and apport does try to add this information but in some cases I receive:

                except Exception as e:
                    apport.error(repr(e))
                    self.ui_error_message(_('Invalid problem report'),
                        _('An error occurred while attempting to process this'
                          ' problem report:') + '\n\n' + str(e))
                    self.ui_shutdown()

where 'e' is empty.

[Brian Murray (brian-murray)]

Another corrupt one can be found here:

https://errors.ubuntu.com/oops/13e9b854-d961-11e1-9159-2c768aafd08c

This one is interesting because it is missing DistroRelease and SourcePackage but has ApportVersion and most others. Another thing to note is that I have found corrupt problems for update-manager, ubiquity, and audacious. Audacious doesn't have a source package hook and update-manager doesn't create its own crash reports so it seems there is something generally wrong.

[Brian Murray (brian-murray)]

What is happening is that a .upload file is created by apport before the collect_info function is complete. Subsequently, it is possible for whoopsie to upload a crash that doesn't have all the information in it. In whoopsie.c there a TODO about dealing with crash files being modified:

    /* TODO handle the file being modified underneath us. */

So while that should be fixed the least invasive thing to do would be to ensure that apport creates the .upload file after all the information has been collected. Luckily, collect_info has an on_finished argument that will do the trick. I'll attach the patch to apport that I successfully tested.

I was able to recreate the issue by interrupting the 'collect info' stage of apport by clicking the close window button when a package hook was running. You can see my incomplete problem report at https://errors.ubuntu.com/oops/e139df74-e6f8-11e1-b1e6-2c768aafd08c. Depending on when the collect process is interrupted there probably would not be a package version which would prevent the problem from being bucketed (at least as I understand it).

[Launchpad Janitor (janitor)]

This bug was fixed in the package apport - 2.4-0ubuntu7

---------------
apport (2.4-0ubuntu7) quantal; urgency=low

  * apport/ui.py: only create the .upload file after collect_info is complete
    that way whoopsie will upload crash files with all the information it
    needs (LP: #1020994)
 -- Brian Murray <email address hidden> Wed, 15 Aug 2012 16:07:34 -0700

[Evan (ev)]

Excellent work! Thanks for tackling this.

We should keep a close eye on future Quantal reports to see if we spot any reports with apport 2.4-0ubuntu7 that are missing fields.

[Evan (ev)]

Marking as Invalid for Daisy and Errors. The problem exists in apport or whoopsie, before the report it sent to Daisy.

[Steve Langasek (vorlon)]

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release