crash digger can't discover, apport-retrace cannot retrace some crahes

[RabbitBot]

crash-digger doesn't seem to be able to discover some crash reports, for example, bug 1054602 even though the relevant .crashdb file is installed. The relevant Apport config files for this crash can be viewed at http://bazaar.launchpad.net/~docky-core/plank/trunk/files/head:/data/apport/

Retracing the same bug with apport-retrace fails:
PYTHONPATH=. bin/apport-retrace --auth=~/.cache/apport/launchpad.credentials --sandbox-dir=/home/serioga/Code/apport-retrace-sandbox/ 1054602
Traceback (most recent call last):
  File "bin/apport-retrace", line 464, in <module>
    apt_root = os.path.join(options.cache, report['DistroRelease'], 'apt')
  File "/usr/lib/python2.7/posixpath.py", line 68, in join
    elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'

The config dir in use is https://code.launchpad.net/~elementary-os/elementaryos/apport-retrace-sandbox

I'm running a trunk checkout, revision 2508 (not latest 2509 because it introduces bug 1061856)

[RabbitBot (rabbitbot-a)]

This happens for other projects too, with Apport config files from https://code.launchpad.net/~elementary-os/elementaryos/apport-hooks

[RabbitBot (rabbitbot-a)]

Happens to bug 1061853 too

[Sergey "Shnatsel" Davidoff (shnatsel)]

It was me who reported the bug. I was logged in as RabbitBot, sorry.

[RabbitBot (rabbitbot-a)]

In elementary there are 62 crash reports affected by this bug: 50 amd64 reports and 12 i386 ones. They're all visible to the bot and have either "need-i386-retrace" or "need-amd64-retrace" tag.

[Sergey "Shnatsel" Davidoff (shnatsel)]

Still happens as of revision 2515.

[Martin Pitt (pitti)]

I fixed the "AttributeError: 'NoneType' object has no attribute 'endswith'" in trunk r2516. NB that you are not using sandbox mode there, so the --sandbox-dir argument won't have any effect. Keeping bug open for the first issue (not discovering some crash bugs).

[Martin Pitt (pitti)]

Please note that bug 1061856 has been fixed a while ago, so you should be fine again to run latest trunk.

[Martin Pitt (pitti)]

Here is what I tried:

- I created a crashdb.conf for gala:

$ cat etc/apport/crashdb.conf.d/gala-crashdb.conf
gala = {
 'impl' : 'launchpad',
 'project' : 'gala',
 'bug_pattern_base' : None,
}

- Create empty config dir (it doesn't matter for discovery):

  mkdir -p /tmp/config/Ubuntu\ 12.10

- Run crash-digger on gala:

$ APPORT_CRASHDB_CONF=etc/apport/crashdb.conf PYTHONPATH=. bin/crash-digger -C /tmp/cache -a ~/.cache/apport/launchpad.credentials --crash-db=gala -v --config-dir=/tmp/config
10/18/12 17:06:16: Available releases: ['Ubuntu 12.10']
10/18/12 17:06:17: fill_pool: retrace pool now: set([1061853])
10/18/12 17:06:17: retracing #1061853 (left in pool: 0)
10/18/12 17:06:17: crash is release elementary 0.2 which does not have a config available, skipping

This does discover bug 1061853, which indeed matches what I can see on https://bugs.launchpad.net/gala/+bugs?field.tag=need-amd64-retrace (there might be others which are private and not visible to me, of course).

bug 1054602 is a duplicate, so crash-digger won't consider that.

Can you please be more specific which bugs should be discovered? Do you confirm that you can see this bug with crash-digger? Can you ensure that your bot has the privileges to see private bugs when they come in?

[Sergey "Shnatsel" Davidoff (shnatsel)]

Found it, turns out it was my mistake after all. I've been invoking crash-digger like this:
$ PYTHONPATH=. bin/crash-digger -c --crash-db="gala" --config-dir=/path/to/config/dir ...
while I should have been doing this:
$ PYTHONPATH=. bin/crash-digger --crash-db="gala" --config-dir=/path/to/config/dir ...
The extra "-c" caused this problem. It would be nice if crash-digger could throw an error in such cases instead of undefined behavior.

On an unrelated note, trunk revision 2516 throws a syntax error on python 2.7.3-0ubuntu2, see attached console output.

[Martin Pitt (pitti)]

Fixed the syntax error in Python 2.7 in r2517, sorry about that.

"-c --crash-db="gala" --config-dir=/path" is indeed a really unfortunate combination of options. It means that it first configures the config directory to be '--crash-db="gala"', and then overwrites it to "/path...". If you had used a different order, or "--crash-db gala" (without the =), or not specified --config-dir a second time, you would have gotten an appropriate error message.

I'm not sure how to improve that. argparse/optparse parse don't throw a warning if you specify an option twice, and once it's done you don't see the implausible '--crash-db="gala"' config path value any more.

So I think I fixed the two side issues, and can't do much about the option syntax (If you have a proposal how to improve it, I'm glad to hear).

Thanks!