'apport' test should provide crash logs as attachments

[Ronald McCollam]

The 'apport' test checks to see if /var/crash is empty. If it is not, the test should provide any crash dumps from this directory as attachments so that bugs can be filed from the crash report.

[Ronald McCollam (fader)]

Attaching everything in /var/crash is a straightforward solution to this; unfortunately, many of the files are rw------- root:root and since checkbox does not run as root, they cannot be read to be attached. This will require some mechanism for escalating privilege to attach these files.

[Matt Zimmerman (mdz)]

There may be a subtle "gotcha" here in the way apport works.

1. When a program crash (or other event is recorded by apport, it creates a .crash file in /var/crash with the details of the crash, such as the process stats, core dump, etc.

2. When the user requests that the crash report be submitted to Launchpad, apport adds more information to it, such as the package version information, and runs the hooks.

This means that if you only obtain the original .crash file, it will not contain enough information to diagnose the crash. We need the additional information added by apport in the second step. If you look at the .crash files you see in certification, you'll notice that there is a lot missing from them.

This means that you will want to do some pre-processing of the .crash files *on the system where the crash occurred* before shipping the crash file off somewhere else, in order for it to be useful. I'm not sure if there is an apport command currently available to do this (it's done by apport-cli and apport-gtk as a preliminary step before reporting the bug).

Martin Pitt should be able to advise as to how best to deal with this.

[Ronald McCollam (fader)]

This would also be very useful for cases where Checkbox is run on a machine without Internet access, either because of hardware issues (bad or missing card or drivers) or environmental issues (no network access or firewalls preventing connection to launchpad).

In these cases, having apport be able to do this processing and save the data on the local machine would allow us to copy the data to an external medium and file the bug from another system or another location.

[Martin Pitt (pitti)]

Does your code take care to only pick crashes which are generated while checkbox was running? It seems a little greedy to me to attach all existing ones, especially since attaching .crash files directly to a bug which is going to be public might inadvertedly expose a lot of private data from the user (and many crash reports might be entirely unrelated to the checkbox run). Also, please note that we have no way to auomatically reprocess .crash file bug attachments.

> This means that you will want to do some pre-processing of the .crash files *on the system where the crash occurred* before shipping the crash file off somewhere else, in order for it to be useful.

Right, you want to at least call add_package_info() and add_hooks_info() on them. But before I elaborate on that, I want to stress that it is _really_ not a good idea to attach /var/crash/*.crash to reported bugs. What is your actual goal on those? With the limited bug description here I don't feel that this approach is the right solution.

[Ronald McCollam (fader)]

Martin,

checkbox-certification (against which this bug is filed) is run only against systems in our certification lab, not by end-users. It is run in the daily testing process which includes a wipe and reinstall of each system and has no end-user interaction. Any crash reports gathered would be ones that were created immediately after a clean install and startup or from running the automated tests.

The goal is to be able to take crash dumps from freshly installed systems that have had some automated test coverage and file bugs based on those dumps. As we have access to the systems and can reproduce the steps that led to the crash (including reinstalls as needed), we have a great opportunity to find and fix bugs. Since this is run internally only, we will not be gathering any information from end-users or annoying them unnecessarily.

If there is additional processing that needs to take place before creating a bug based on this crash data, we'd be happy to do this. Is there a process we can automate to perform this, or is this new functionality that we need to look into?

[Ronald McCollam (fader)]

After discussion and reflection, it sounds like gathering the crash logs, even only for systems that we have in-house, is not useful until/unless we have a way of handling the data from them after they are gathered and converting them to bugs.

With that said, it seems that it would still be quite valuable to get at least a list of the crashes. It is still valuable information to know if it is the same application crashing each time, or if there are multiple applications with problems. We can at least do an 'ls' of /var/crash and attach the results of that.

[Martin Pitt (pitti)]

Hello Roland,

Ronald McCollam [2009-09-02 13:54 -0000]:
> If there is additional processing that needs to take place before
> creating a bug based on this crash data

Yes, when the initial .crash file is written, it only contains the
bare minimum of information which needs to be collected when the
crashing process is still in limbo. All expensive bits (such as
querying the package database) are done later, to avoid taking even
more time until the crashed process can rest in peace.

> Is there a process we can automate to perform this, or is this new
> functionality that we need to look into?

The least interactive way to do it with the existing programs is

  apport-cli -c /path/to/crashfile

and press "K", then the .crash file is fully ready to be transferred
to a different system and be submitted to Launchpad (with
double-clicking in nautilus or using ubuntu-bug foo.crash).

Zero interaction is only possible with using the python Apport API and
don't collect hook information, since hooks can also ask interactive
questions.

Is one of these options suitable for you? I can give you the details
about the API if you need this.

--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[Jeff Lane  (bladernr)]

Old report... but: checkbox does fire apport when tests crash, and perhaps that could lead to adding the ability to fulfill this as a new feature request at some point in the future. Thus, I'm setting this as a Wishlist item.

[Ara Pulido (ara)]

Moving to checkbox project, as it is where this test is now

[Jeff Lane  (bladernr)]

I'd really like to see this go away. We should either change the apport test to fire off apport for any crash files it finds (or offer to at least) or close this as we at least partly answer this by having apport offer to file bugs when a test crashes.

I think it WOULD, however, be good to allow apport to fire when a test crashes, but have it open the bug against ubuntu-certification then let us triage it to the right package from there. That should avoid the issue where we open too many bugs against individual packages.

Otherwise, we could look at just dropping the apport test all together, as it really doesn't do anything useful.

[Jeff Lane  (bladernr)]

This needs to die, and now it has... won't be implemented and I'm fine with that.