libcairo2-dbg CRC doesn't match symbol (I suspect this is more widespread)

[Alex Bennée]

While attempting to oprofile something on my Hardy system I discovered the following:

vnms@vnms1:~/ajb$ opreport -V bfd -l /usr/lib/libcairo.so.2.17.3
CPU: Core 2, speed 2500 MHz (estimated)
Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (Unhalted core cycles) count 100000
op_bfd ctor for /usr/lib/libcairo.so.2.17.3
bfd_info::get_symbols() for /usr/lib/libcairo.so.2.17.3
bfd_get_symtab_upper_bound: 8
bfd_canonicalize_symtab: 0
fetching .gnu_debuglink section
.gnu_debuglink section has size 18
.gnu_debuglink filename is libcairo.so.2.17.3
looking for debugging file libcairo.so.2.17.3 with crc32 = 2cc7fcff
found /usr/lib/debug/usr/lib/libcairo.so.2.17.3 with crc32 = 9a3e25be
found /usr/lib/libcairo.so.2.17.3 with crc32 = 3ec5255e
failed to process separate debug file
number of symbols before filtering 1
number of symbols now 1
samples % image name app name symbol name
9199 44.4482 libcairo.so.2.17.3 vsnet /usr/lib/libcairo.so.2.17.3
5724 27.6575 libcairo.so.2.17.3 gnome-terminal /usr/lib/libcairo.so.2.17.3
4879 23.5746 libcairo.so.2.17.3 gdmgreeter /usr/lib/libcairo.so.2.17.3
466 2.2516 libcairo.so.2.17.3 vsview /usr/lib/libcairo.so.2.17.3
241 1.1645 libcairo.so.2.17.3 gnome-panel /usr/lib/libcairo.so.2.17.3
185 0.8939 libcairo.so.2.17.3 metacity /usr/lib/libcairo.so.2.17.3
2 0.0097 libcairo.so.2.17.3 nautilus /usr/lib/libcairo.so.2.17.3

I suspect other packages may well have odd CRC's but I have no idea if this is a feature of the way dbg packages are created.

To recreate you'll probably want to rebuild the Karmic oprofile package for Hardy. The current Hardy package seems broken.

[Sebastien Bacher (seb128)]

the debug symbols work correctly in gdb, not sure what ooprofile is doing or the version of the packages installed

[Alex Bennée (ajbennee)]

The problem is that when the library was built the CRC of the debug symbols file gets planted in the main library file. OProfile very sensibly checks that the CRC of the debug file matches that stored in the actual library and ignores it if it doesn't.

GDB may well work and having hacked oprofile to ignore the embedded CRC in library it *seems* to give correct results. Of course it's hard to be sure because the -dbg symbols look as though they were created at a different time to the actual library. If I look at the opreport -V bfd output on the same library on my Gentoo system (where I know the striped symbol-file was created at the same time as the library) I can see the CRC's match and everything is fine.

Picking a random example (libc) I see the following when running "opreport -V bfd":

looking for debugging file libc-2.7.so with crc32 = 9587b62d
separate_debug_file_exists: /lib/.debug/libc-2.7.so give img_path/lib/.debug/libc-2.7.so
separate_debug_file_exists: /usr/lib/debug/lib/libc-2.7.so give img_path/usr/lib/debug/lib/libc-2.7.so
found /usr/lib/debug/lib/libc-2.7.so with crc32 = 9587b62d
now loading: /usr/lib/debug/lib/libc-2.7.so

And you can see the CRC of the debug libc matches so we can be confident of the results offered by oprofile (and gdb for that matter).

As I stated in the original report this may be an artifact of the build process that creates packages however I don't really have an understanding of the Ubuntu build farm. Perhaps there is someone on the infrastructure team that could give some input.

The versions of libcairo2 involved are:

11:05 root@vnms/x86_64 [vnms] >dpkg --status libcairo2 libcairo2-dbg
Package: libcairo2
Status: deinstall ok installed
Priority: optional
Section: libs
Installed-Size: 828
Maintainer: Ubuntu Core Developers <email address hidden>
Architecture: amd64
Source: cairo
Version: 1.6.0-0ubuntu2
Replaces: libcairo0.5.1, libcairo0.6.0, libcairo0.9.0, libcairo1
Provides: libcairo
Depends: libc6 (>= 2.4), libfontconfig1 (>= 2.4.0), libfreetype6 (>= 2.3.5), libgcc1 (>= 1:4.1.1-21), libpixman-1-0 (>= 0.10.0), libpng12-0 (>= 1.2.13-4), libstdc++6, libx11-6, libxrender1, zlib1g (>= 1:1.2.3.3.dfsg-1)
Conflicts: libcairo1
Description: The Cairo 2D vector graphics library
 Cairo is a multi-platform library providing anti-aliased
 vector-based rendering for multiple target backends. Paths consist
 of line segments and cubic splines and can be rendered at any width
 with various join and cap styles. All colors may be specified with
 optional translucence (opacity/alpha) and combined using the
 extended Porter/Duff compositing algebra as found in the X Render
 Extension.
 .
 Cairo exports a stateful rendering API similar in spirit to the path
 construction, text, and painting operators of PostScript, (with the
 significant addition of translucence in the imaging model). When
 complete, the API is intended to support the complete imaging model of
 PDF 1.4.
 .
 This package contains the shared libraries.
Homepage: http://cairographics.org/
Original-Maintainer: Dave Beckett <email address hidden>

Package: libcairo2-dbg
Status: install ok installed
Priority: extra
Section: libdevel
Installed-...

[Sebastien Bacher (seb128)]

the dbg is built using dh_strip which calls objcopy --only-keep-debug I'm not running hardy there but gdb would work if the sum was incorrect between those so not sure what is wrong there

[Sebastien Bacher (seb128)]

gdb wouldn't work if the control sum was incorrect rather

[Alex Bennée (ajbennee)]

It could be this only affects the Hardy packages? Or maybe something more subtle?

As you say GDB should complain but it certainly doesn't seem fussy:

vnms@vnms:~$ gdb /usr/bin/gnome-terminal
GNU gdb 6.8-debian
Copyright (C) 2008 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu"...
(no debugging symbols found)
(gdb) symbol-file /usr/lib/debug/usr/lib/libcairo.so.2.17.3
Reading symbols from /usr/lib/debug/usr/lib/libcairo.so.2.17.3...done.

I can manually examine the CRC32's of libcairo and libc debugs and compare with the debug_link sections in the elf files and see the same discrepancy oprofile does (ruling out oprofile getting it wrong).

vnms@vnms:~$ cfv -v -C -t csv /usr/lib/debug/usr/lib/libcairo.so.2.17.3 /usr/lib/debug/lib/libc-2.7.so
/usr/lib/debug/lib/libc-2.7.so : OK (407169,9587b62d)
/usr/lib/debug/usr/lib/libcairo.so.2.17.3 : OK (1275263,9a3e25be)

vnms@vnms:~$ readelf -x 25 /usr/lib/libcairo.so.2.17.3

Hex dump of section '.gnu_debuglink':
  0x00000000 6c696263 6169726f 2e736f2e 322e3137 libcairo.so.2.17
  0x00000010 2e330000 fffcc72c .3.....,

vnms@vnms:~$ readelf -x 68 /lib/libc.so.6

Hex dump of section '.gnu_debuglink':
  0x00000000 6c696263 2d322e37 2e736f00 2db68795 libc-2.7.so.-...

I can only assume GDB is checking the build ID but I'm not sure how to check for that. Certainly there don't seem to any "notes" sections in the ELF file according to readelf. I've mailed the GDB mailing list to see how I can extract it.

If the build IDs match between the library and it's debug component I can only assume there is some weird bug in binutils that created an odd split debug section.

[Sebastien Bacher (seb128)]

thanks for the details about crc checking, indeed the dbg is broken in hardy, you can use the dbgsym which doesn't have this issue in hardy apparently, closing the bug since the karmic version is fixed and that bug doesn't justify a stable update for hardy especially that it's recommended to use the dbgsym builds for ubuntu

[Alex Bennée (ajbennee)]

Just for completeness are there any pointers to using the "pkg-create-dbgsym" package? Installing it and rebuilding the package doesn't seem to have worked.

[Alex Bennée (ajbennee)]

It looks like this is a bug in the tools. I installed "pkg-create-dbgsym" and rebuilt cairo from scratch and installed the new packages and I still get failures due to bad CRCs. I shall raise a new bug as I doubt this is the libcairo package explicitly.

[Sebastien Bacher (seb128)]

did you install the dbgsym after the bug?

you can also add "deb http://ddebs.ubuntu.com hardy-updates main restricted universe multiverse" to your sources.list (https://wiki.ubuntu.com/DebuggingProgramCrash) and install the official build one

[Sebastien Bacher (seb128)]

could you check if other libs have the same issue? the bug is probably one in the cairo rules than one in the tools, we do lot of retracing on crashes sent and we would have noticed if debug symbols were broken around the board in hardy

[Alex Bennée (ajbennee)]

I was rebuilding from source after install the pkg-create-dbgsyms package. Although curiously installing a dbgsym from the repo you referenced:

apt-get install libcairo2-dbgsym

followed by opreport:

<snip>
fetching .gnu_debuglink section
.gnu_debuglink section has size 18
.gnu_debuglink filename is libcairo.so.2.17.3
looking for debugging file libcairo.so.2.17.3 with crc32 = 2cc7fcff
found /usr/lib/debug/usr/lib/libcairo.so.2.17.3 with crc32 = 2cc7fcff
now loading: /usr/lib/debug/usr/lib/libcairo.so.2.17.3

So that package seems OK. Possibly my build broke due to some sort environmental failure. I'll see if I can find any other broken debug packages.

[Sebastien Bacher (seb128)]

you rebuilt from source but did you install the dbgsym ddeb locally or the dbg deb?

[Alex Bennée (ajbennee)]

Two cases:

1. Built from source (with pkg-create-dbgsyms installed). Installed my locally built package with the locally built dbgsym ddeb:

sudo dpkg -i libcairo2-dbgsym_1.6.0-0ubuntu2_amd64.ddeb libcairo2_1.6.0-0ubuntu2_amd64.deb

The CRCs didn't match leading me to raise #423748.

2. Installed from apt repo, with the libcairo2-dbgsym from the ddeb repo and it worked fine.

As far as this bug is concerned I think it's fine with the ddeb provided dbgsym packages (which I shall use in future). Once I have written a script to check the debug images I'll see if there is a more general problem and append that info to #423748.

[Alex Bennée (ajbennee)]

I wrote a quick script and found that the only debug symbols I'm sure are suspect were the origin -dbg version of the libcairo ones. There are a bunch of files in the top level of /usr/lib/debug which don't match but I was kinda guessing, e.g.:

14:31 ajb@pitcairn/x86_64 [dbgsym.git] >./dbgsym.py -a -f /usr/lib/debug/ld-linux-x86-64.so.2
couldn't find paired library for: /usr/lib/debug/ld-linux-x86-64.so.2
trying alt: /lib/ld-linux-x86-64.so.2
debug_file: /usr/lib/debug/ld-linux-x86-64.so.2 has CRC of 77A3FBF0
actual_file: /lib/ld-linux-x86-64.so.2 thinks CRC should be 17CC7B66

But some don't make sense. For example:

/lib/libnss_nis-2.7.so
/lib32/libnss_nis-2.7.so
/usr/lib/debug/libnss_nis-2.7.so
/usr/lib/debug/lib/libnss_nis-2.7.so
/usr/lib/debug/lib32/libnss_nis-2.7.so

3 debug files but only two actual libraries. The odd one out doesn't match as you'd expect:

14:35 ajb@pitcairn/x86_64 [dbgsym.git] >./dbgsym.py -a -f /usr/lib/debug/libnss_nis-2.7.so
couldn't find paired library for: /usr/lib/debug/libnss_nis-2.7.so
trying alt: /lib/libnss_nis-2.7.so
debug_file: /usr/lib/debug/libnss_nis-2.7.so has CRC of DE32A5A5
actual_file: /lib/libnss_nis-2.7.so thinks CRC should be 430800A
14:36 ajb@pitcairn/x86_64 [dbgsym.git] >./dbgsym.py -a -f /usr/lib/debug/lib/libnss_nis-2.7.so
14:37 ajb@pitcairn/x86_64 [dbgsym.git] >./dbgsym.py -a -f /usr/lib/debug/lib32/libnss_nis-2.7.so
14:37 ajb@pitcairn/x86_64 [dbgsym.git] >

You can find the script at github:

http://github.com/stsquad/dbgsym/tree/master