crash-digger cannot discover bugs reported against projects

[Sergey "Shnatsel" Davidoff]

crash-digger discovers only bugs filed against the distribution on which it is running (output of self.launchpad.distributions[self.distro]), not against projects. E.g. it's unable to discover bug #1001297 even though I have permissions to view it and it (accidentially) still has "need-amd64-retrace" tag.

[Martin Pitt (pitti)]

Hey Sergey,

this is an untested patch. Could you give it a try?

[Sergey "Shnatsel" Davidoff (shnatsel)]

I've tried the patch, nothing changed. It still discovers only two Maverick bugs and fails to retrace them.

[Martin Pitt (pitti)]

Sergey "Shnatsel" Davidoff [2012-05-24 8:40 -0000]:
> I've tried the patch, nothing changed. It still discovers only two
> Maverick bugs and fails to retrace them.

Did you set the "project" option in your crash database conf?

[Sergey "Shnatsel" Davidoff (shnatsel)]

Here's the crashdb file for Scratch: http://bazaar.launchpad.net/~elementary-os/elementaryos/apport-hooks/view/head:/apport/scratch-crashdb.conf
And source_scratch.py, just in case: http://bazaar.launchpad.net/~elementary-os/elementaryos/apport-hooks/view/head:/apport/source_scratch.py

[Martin Pitt (pitti)]

Ah, thanks; it might not consider the report specific crashdb, I'll look at the code a bit harder.

  report['ThirdParty'] = 'True'

This is obsolete, BTW. You can just drop it.

[Sergey "Shnatsel" Davidoff (shnatsel)]

Thanks, I've dropped it from generator script and updated https://wiki.ubuntu.com/Apport/DeveloperHowTo accordingly

[Martin Pitt (pitti)]

This patch is now in trunk together with a test: http://bazaar.launchpad.net/~apport-hackers/apport/trunk/revision/2410

So this is necessary to make stuff like get_unretraced() work for project DBs at all. However, it seems it is not sufficient yet, so not setting to "fix committed" yet.

[Martin Pitt (pitti)]

It should really work with this patch now. How do you call crash-digger? In particular, do you supply a correct crashdb.conf with the "project" option, in either /etc/apport/crashdb.conf or by pointing to your custom one in $APPORT_CRASHDB_CONF ? Is the remaining problem that crash-digger should grow an option for specifying a custom crashdb.conf without the need of setting the environment variable?

[Sergey "Shnatsel" Davidoff (shnatsel)]

Here's how I call crash-digger for testing:
 $ crash-digger --config-dir=/home/precise/apport-sandbox/ --auth=~/.cache/apport/launchpad.credentials
I did not alter any environment variables.

I use per-package hooks in /etc/apport/crashdb.conf.d/ (for reference, they're in lp:~elementary-os/elementaryos/apport-hooks)

So if I set $APPORT_CRASHDB_CONF to the path to a custom file in /etc/apport/crashdb.conf.d/, it should work now?
Can I specify several crashdb files there? Probably it's not really needed since I can call it in a shell loop fairly easily.

The command-line option is not really needed either as long as the environment variable is documented in "crash-digger --help" (since crash-digger doesn't have a manpage).

[Martin Pitt (pitti)]

Hello Sergey,

Sergey "Shnatsel" Davidoff [2012-06-28 12:40 -0000]:
> I use per-package hooks in /etc/apport/crashdb.conf.d/ (for reference,
> they're in lp:~elementary-os/elementaryos/apport-hooks)

Right, package hooks are only being used for reporting bugs, not for
retracing.

> So if I set $APPORT_CRASHDB_CONF to the path to a custom file in /etc/apport/crashdb.conf.d/, it should work now?

Not quite, as these are not full configuration files, just parts of
it.

> Can I specify several crashdb files there? Probably it's not really
> needed since I can call it in a shell loop fairly easily.

That's indeed one difficulty. Right now crash-digger was designed to
just work on the default crash db (the one that
/etc/apport/crashdb.conf defines as "default =...").

So from what I can see, it should help you already to add a --crash-db
command line option, so that you can e. g. call crash-digger with
"--crash-db cerbere" or "--crash-db feedler". Then you would need to
call crash-digger in a shell loop for all projects. From there it is
only a small step towards supporting to specify the option multiple
times and iterating over all of them in crash-digger itself.

Are you running your retracers from trunk, i. e. lp:apport? Or do you
need this packaged for testing?

Thanks,

Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[Martin Pitt (pitti)]

I added a --crash-db option in http://bazaar.launchpad.net/~apport-hackers/apport/trunk/revision/2417 , can you please update your trunk checkout and try it out?

I also requested another package build from trunk in https://code.launchpad.net/~pitti/+recipe/apport-test, in case you need a package for precise.

Supporting multiple --crash-db options is not as easy as I thought initially, so I did not add that yet. Let's confirm first that it works for you at all now.

[Sergey "Shnatsel" Davidoff (shnatsel)]

Well, using a package is not possible because python3-kde4 is not available in Precise. With that dependency hacked out and apport-kde not installed, all I get is this:
"ERROR: The launchpadlib Python module is not installed. This functionality is not available."
I cannot find a Python 3 module for launchpadlib anywhere, and Python 2 module is already installed, so I'm not sure what causes this.

[Martin Pitt (pitti)]

You don't need to install apport-kde for the retracing bits. There is indeed no Python 3 version of launchpadlib, so apport-retrace still uses Python 2.

However, it is actually easier to run the retracers right from a trunk checkout. I mostly build packages as well for people who need to test the client-side bits on precise.

[Sergey "Shnatsel" Davidoff (shnatsel)]

So, what steps shall I take to test it?
Maybe it's easier to boot Quantal from LiveCD and try it from repos there?

[Martin Pitt (pitti)]

Just use it from trunk:

bzr branch lp:apport
cd apport
./setup.py build

Now you can run:

PYTHONPATH=. bin/crash-digger [...] # this will use apport-retrace from the checkout, too
PYTHONPATH=. bin/apport-retrace [...]

[Sergey "Shnatsel" Davidoff (shnatsel)]

It works now. Thanks!

[Martin Pitt (pitti)]

Great!