not every crash file in the test-crashes package has a StacktraceAddressSignature

[Brian Murray]

At least these two crash files are missing StacktraceAddressSignature information:

trusty/armhf/_usr_lib_dconf_dconf-service.2001.crash
quantal/armhf/_usr_lib_dconf_dconf-service.2001.crash

Is this intended or did something go awry when generating the .crash files?

[Martin Pitt (pitti)]

What do you mean with "these two crash files", i. e. where are they? The gdb-ish fields (StacktraceAddressSignature, Stacktrace, Disassembly, etc.) nor all the package fields (Package:, Dependencies:, etc.) won't be present for an intial .crash file when the crash happens. They are only added to the report when post-processing a report, i. e. when the user agreed to submit it to the crash db.

[Martin Pitt (pitti)]

Oh sorry, you meant http://bazaar.launchpad.net/~daisy-pluckers/error-tracker-deployment/test-crashes/view/head:/generate-sigsegv-crash.py presumably, which does call add_gdb_info(). Yes, this is supposed to have a StacktraceAddressSignature field.

[Brian Murray (brian-murray)]

I was examining crash files in the apport-test-crashes package from the daisy pluckers PPA - https://code.launchpad.net/~daisy-pluckers/+archive/daisy-seeds.

[Martin Pitt (pitti)]

So e. g. quantal-amd64's cat crash has a proper Stacktrace with enough addresses/data to create an SAS from:

Stacktrace:
 #0 0x00002aaaaadb6fe0 in read () from /lib/x86_64-linux-gnu/libc.so.6
 #1 0x0000000000404c76 in ?? ()
 #2 0x00000000004023c3 in ?? ()
 #3 0x00002aaaaacf176d in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6
 #4 0x0000000000402565 in ?? ()
 #5 0x00007fffffffe668 in ?? ()
 #6 0x000000000000001c in ?? ()
 #7 0x0000000000000001 in ?? ()
 #8 0x00007fffffffe96e in ?? ()
 #9 0x0000000000000000 in ?? ()

But on armhf it is rather short; there are only two valid frames, and we require at least three (without errors) or five (with at most one resolution error):

Stacktrace:
 #0 0x400ad4f8 in read () from /lib/arm-linux-gnueabihf/libc.so.6
 #1 0x0000b4a4 in ?? ()
 #2 0x0000b4a4 in ?? ()
 Backtrace stopped: previous frame identical to this frame (corrupt stack?)

However, the current logic in crash_signature_addresses() doesn't actually check for duplicate addresses. Also, in the corresponding amd64/armhf pair for dconf-service, the arm trace has four frames, so it ought to work. Curiously calling it manually with those very files actually works:

$ python3 -c 'import apport; r = apport.Report(); r.load(open("quantal-armhf/usr/share/apport-test-crashes/_bin_cat.2001.crash", "rb")); print(r.crash_signature_addresses())'
/bin/cat:11:x86_64:/lib/arm-linux-gnueabihf/libc-2.15.so+874f8:/bin/cat+34a4:/bin/cat+34a4

$ python3 -c 'import apport; r = apport.Report(); r.load(open("quantal-armhf/usr/share/apport-test-crashes/_usr_lib_dconf_dconf-service.2001.crash", "rb")); print(r.crash_signature_addresses())'
/usr/lib/dconf/dconf-service:11:x86_64:/lib/arm-linux-gnueabihf/libc-2.15.so+171e6:/lib/arm-linux-gnueabihf/libc-2.15.so+887c8:/lib/arm-linux-gnueabihf/libglib-2.0.so.0.3400.1+35702:/lib/arm-linux-gnueabihf/libglib-2.0.so.0.3400.1+35702

So out of the 14 expected .crash files (7 for each quantal and trusty), only 9 amd64 ones have a S-A-S, and only 1 armhf one has. I don't think that's a bug in crash_signature_addresses() itself, but somewhere in between how these test crashes get built and processed.

[Martin Pitt (pitti)]

I committed http://bazaar.launchpad.net/~daisy-pluckers/error-tracker-deployment/test-crashes/revision/38 (amonst some other updates) to ensure that all crash files have a proper Stacktrace and StacktraceSignature field, and fail the build otherwise. That will make this failure more obvious at least.

[Martin Pitt (pitti)]

This is now exposed in https://launchpadlibrarian.net/163830272/buildlog_ubuntu-trusty-armhf.apport-test-crashes_0.20140127-1~39~ubuntu14.04.1_FAILEDTOBUILD.txt.gz :

Ensure that *.crash files have expected keys...
StacktraceAddressSignature missing in _bin_cat.2001.crash!

[Martin Pitt (pitti)]

The amd64 and i386 builds in the recipe worked fine, so their *.crash files are complete. I built the test-crashes package about five times on my amd64/trusty workstation, and the reports were always complete.

I am now buildin in LXC container on a Calxeda node (armhf), and can reproduce the bug there. However, I think our ARM PPA builders are not real ARM machines but Qemu emulators, but let's check those later.

[Martin Pitt (pitti)]

When I try this manually in current trusty on ARM, I indeed only get two frames in gdb:

$ gdb /usr/lib/dconf/dconf-service /tmp/tmpmm_fur/my.core
(gdb) bt
#0 0xb6d746a0 in poll () at ../sysdeps/unix/syscall-template.S:81
#1 0xb6e2326e in ?? () from /lib/arm-linux-gnueabihf/libglib-2.0.so.0
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Similarly bad with /bin/cat. I'm afraid I don't know why gdb can extract so little information about these on armhf.

Downgrading gdb to the saucy version (https://launchpad.net/ubuntu/+source/gdb/7.6.1-0ubuntu3/+build/5101544) does seem to help; I get better stack traces with that. They are still not up to par with x86, but at least they are long enough to get a StacktraceSignatureAddress.