[SRU] valgrind fails to use debug symbols from glib/gtk

[Sebastien Bacher]

[Impact]

 * With dwz the .gnu_debuglink section may appear duplicated in the debug file referenced originally in the .gnu_debuglink section. In that case Valgrind fails to load the debug symbols.

[Test Case]

 * In Ubuntu releases earlier than 19.10 use gio from a package rebuilt with dwz.
 * Install debug symbols for gio following the guide at https://wiki.ubuntu.com/Debug%20Symbol%20Packages
 * Observe no relevant error when running gio with valgrind:
   $ valgrind /usr/bin/gio ~

[Regression Potential]

 * The fix is very simple, just ignoring the second .gnu_debuglink found. There are no likely regressions.

[Original Bug Text]

The debug symbols from libglib2.0-0-dbgsym and libgtk-3-0-dbgsym don't get properly picked up by valgrind in eoan for some reason. The symbols correctly work under gdb.

There is an warning printed from valgrind
'debuginfo section duplicates a section in the main ELF file'

Laney tried to rebuild without dh_dwz and it makes the warning go away and the symbol work.
Downgrading valgrind to the disco version doesn't fix the issue.

[Karthik-datt (karthik-datt)]

Created attachment 114009
Callgrind.out file is also attached

==967== Callgrind, a call-graph generating cache profiler

==967== Copyright (C) 2002-2015, and GNU GPL'd, by Josef Weidendorfer et al.

==967== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info

==967== Command: /usr/share/WashingMachine/WashingMachineApp --platform eglfs

==967==

==967== For interactive control, run 'callgrind_control -h'.

-967- WARNING: Serious error when reading debug info

-967- When reading debug info from /lib/ld-2.24.so:

-967- debuginfo section duplicates a section in the main ELF file

-967- WARNING: Serious error when reading debug info

-967- When reading debug info from /lib/libgcc_s.so.1:

-967- debuginfo section duplicates a section in the main ELF file

-967- WARNING: Serious error when reading debug info

-967- When reading debug info from /lib/libc-2.24.so:

-967- debuginfo section duplicates a section in the main ELF file

-967- WARNING: Serious error when reading debug info

-967- When reading debug info from /lib/libpthread-2.24.so:

-967- debuginfo section duplicates a section in the main ELF file

-967- WARNING: Serious error when reading debug info

-967- When reading debug info from /lib/libm-2.24.so:

-967- debuginfo section duplicates a section in the main ELF file

-967- WARNING: Serious error when reading debug info

-967- When reading debug info from /lib/libdl-2.24.so:

-967- debuginfo section duplicates a section in the main ELF file

-967- WARNING: Serious error when reading debug info

-967- When reading debug info from /lib/librt-2.24.so:

-967- debuginfo section duplicates a section in the main ELF file

^C==967==

==967== Process terminating with default action of signal 2 (SIGINT)

==967== at 0x4019678: ??? (in /lib/ld-2.24.so)

==967==

==967== Events : Ir

==967== Collected : 12295033

==967==

==967== I refs: 12,295,033

We are trying it on a Linux Arm platform , Imx6 Boundary devices Boards.

Tries it with 3.11 and also 3.13 version. Same warnings are thrown

[Iain Lane (laney)]

Here's how I reproduce (I just picked a simple glib-using program, could be anything AFAICT):

laney@raleigh> valgrind /usr/bin/gio ~
==2653== Memcheck, a memory error detector
==2653== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2653== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==2653== Command: /usr/bin/gio
==2653==
--2653-- WARNING: Serious error when reading debug info
--2653-- When reading debug info from /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0.6200.1:
--2653-- debuginfo section duplicates a section in the main ELF file
--2653-- WARNING: Serious error when reading debug info
--2653-- When reading debug info from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.1:
--2653-- debuginfo section duplicates a section in the main ELF file
--2653-- WARNING: Serious error when reading debug info
--2653-- When reading debug info from /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6200.1:
--2653-- debuginfo section duplicates a section in the main ELF file
[...]
==3711==
==3711== HEAP SUMMARY:
==3711== in use at exit: 36,900 bytes in 236 blocks
==3711== total heap usage: 469 allocs, 233 frees, 68,995 bytes allocated
==3711==
==3711== 16 bytes in 1 blocks are possibly lost in loss record 10 of 85
==3711== at 0x483A723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3711== by 0x483D017: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3711== by 0x4ABC58F: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.1)
==3711==
==3711== 16 bytes in 1 blocks are possibly lost in loss record 11 of 85
==3711== at 0x483CD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3711== by 0x4ABC540: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.1)
==3711== by 0xF: ???
==3711== by 0x4BC0ECE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6200.1)
==3711==
==3711== 16 bytes in 1 blocks are possibly lost in loss record 12 of 85
==3711== at 0x483A723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3711== by 0x483D017: realloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3711== by 0x4ABC58F: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.1)
==3711== by 0xF: ???
==3711== by 0x4BC0E47: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6200.1)
==3711==
==3711== 32 bytes in 2 blocks are possibly lost in loss record 16 of 85
==3711== at 0x483CD99: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3711== by 0x4ABC540: ??? (in /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.1)
==3711== by 0x2F: ???
==3711== by 0x4BC0ECE: ??? (in /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6200.1)
==3711==
==3711== 32 bytes in 2 blocks are possibly lost in loss record 17 of 85
==3711== at 0x483A723: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memc...

[Balint Reczey (rbalint)]

This seems similar to LP: #1843479 , maybe strip is broken, will check.

[Balint Reczey (rbalint)]

The problematic debug symbol files triggering the valgrind issue.

root@ee:~# readelf --debug-dump=links /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0.6200.1
Contents of the .gnu_debugaltlink section:

  Separate debug info file: /usr/lib/debug/.dwz/x86_64-linux-gnu/libglib2.0-0.debug
  Build-ID (0x14 bytes):
 cb 51 16 ac 07 e2 cb 3d 10 5e 39 f1 63 ba b7 b1 45 94 a1 49

Contents of the .gnu_debuglink section:

  Separate debug info file: 73a7789b7bf820c8546a35c13938e61fac4de0.debug
  CRC value: 0xc19e667a

root@ee:~# readelf --debug-dump=links /usr/lib/debug/.build-id/0f/73a7789b7bf820c8546a35c13938e61fac4de0.debug
readelf: Warning: Unsupported form (DW_FORM_GNU_strp_alt) for attribute DW_AT_comp_dir
Contents of the .gnu_debugaltlink section:

  Separate debug info file: /usr/lib/debug/.dwz/x86_64-linux-gnu/libglib2.0-0.debug
  Build-ID (0x14 bytes):
 cb 51 16 ac 07 e2 cb 3d 10 5e 39 f1 63 ba b7 b1 45 94 a1 49

root@ee:~# readelf --debug-dump=links /usr/lib/debug/.dwz/x86_64-linux-gnu/libglib2.0-0.debug

.gnu_debuglink should not be present in 73a7789b7bf820c8546a35c13938e61fac4de0.debug.

[Balint Reczey (rbalint)]

Valgrind is typically used with binaries from the same Ubuntu release or locally built ones.

Since dwz was enabled by default for Debhelper compat level 12 in 19.04 I suggest not backporting the fix to 18.04. Since the issue was not reported against 19.04 and 19.04 goes EOL soon I suggest not backporting the fix to Disco either.

[Balint Reczey (rbalint)]

Created attachment 124162
Don't look for debug alt file in debug image if it is already found

In a case observed in Ubuntu the duplicated section was .gnu_debuglink and it is safe to ignore it like it is done for similar sections.

[Balint Reczey (rbalint)]

I've sent the fix upstream at: https://bugs.kde.org/show_bug.cgi?id=396656

[Launchpad Janitor (janitor)]

This bug was fixed in the package valgrind - 1:3.15.0-1ubuntu5

---------------
valgrind (1:3.15.0-1ubuntu5) focal; urgency=medium

  * Drop MPI 1 support to fix FTBFS with openmpi 4.0

valgrind (1:3.15.0-1ubuntu4) focal; urgency=medium

  * Don't look for debug alt file in debug image if it is already found
    (LP: #1848211)

 -- Balint Reczey <email address hidden> Thu, 28 Nov 2019 19:35:25 +0100

[Balint Reczey (rbalint)]

I have uploaded the SRU for 19.10.

[Timo Aaltonen (tjaalton)]

Hello Sebastien, or anyone else affected,

Accepted valgrind into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/valgrind/1:3.15.0-1ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted valgrind (1:3.15.0-1ubuntu3.1) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

apport/2.20.11-0ubuntu8.2 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#valgrind

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Balint Reczey (rbalint)]

The apport failure is unrelated and is caused and tracked by LP: #1854237.

[Balint Reczey (rbalint)]

Verified with 1:3.15.0-1ubuntu3.1 on Eoan:

root@ee-valgrind:~# echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse
> deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
> deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
> sudo tee -a /etc/apt/sources.list.d/ddebs.list
deb http://ddebs.ubuntu.com eoan main restricted universe multiverse
deb http://ddebs.ubuntu.com eoan-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com eoan-proposed main restricted universe multiverse
root@ee-valgrind:~# sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F2EDC64DC5AEE1F6B9C621F0C8CAB6595FDFF622
Executing: /tmp/apt-key-gpghome.p3fDDHpLDX/gpg.1.sh --keyserver keyserver.ubuntu.com --recv-keys F2EDC64DC5AEE1F6B9C621F0C8CAB6595FDFF622
gpg: key C8CAB6595FDFF622: 4 signatures not checked due to missing keys
gpg: key C8CAB6595FDFF622: public key "Ubuntu Debug Symbol Archive Automatic Signing Key (2016) <email address hidden>" imported
gpg: Total number processed: 1
gpg: imported: 1
root@ee-valgrind:~# apt update -qq
28 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@ee-valgrind:~# apt install libglib2.0-0-dbgsym=2.62.1-1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  libfreetype6
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  libglib2.0-0-dbgsym
0 upgraded, 1 newly installed, 0 to remove and 28 not upgraded.
Need to get 3738 kB of archives.
After this operation, 4418 kB of additional disk space will be used.
Get:1 http://ddebs.ubuntu.com eoan/main amd64 libglib2.0-0-dbgsym amd64 2.62.1-1 [3738 kB]
Fetched 3738 kB in 0s (60.2 MB/s)
Selecting previously unselected package libglib2.0-0-dbgsym:amd64.
(Reading database ... 30827 files and directories currently installed.)
Preparing to unpack .../libglib2.0-0-dbgsym_2.62.1-1_amd64.ddeb ...
Unpacking libglib2.0-0-dbgsym:amd64 (2.62.1-1) ...
Setting up libglib2.0-0-dbgsym:amd64 (2.62.1-1) ...
root@ee-valgrind:~# valgrind gio ~
==2560== Memcheck, a memory error detector
==2560== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==2560== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==2560== Command: gio /root
==2560==
--2560-- WARNING: Serious error when reading debug info
--2560-- When reading debug info from /usr/lib/x86_64-linux-gnu/libgmodule-2.0.so.0.6200.1:
--2560-- debuginfo section duplicates a section in the main ELF file
--2560-- WARNING: Serious error when reading debug info
--2560-- When reading debug info from /usr/lib/x86_64-linux-gnu/libglib-2.0.so.0.6200.1:
--2560-- debuginfo section duplicates a section in the main ELF file
--2560-- WARNING: Serious error when reading debug info
--2560-- When reading debug info from /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0.6200.1:
--2560-- debuginfo section duplicates a section in the main ELF file
Usage:
  gio COMMAND [ARGS…]

Commands:
  help Pri...

[Launchpad Janitor (janitor)]

This bug was fixed in the package valgrind - 1:3.15.0-1ubuntu3.1

---------------
valgrind (1:3.15.0-1ubuntu3.1) eoan; urgency=medium

  * Don't look for debug alt file in debug image if it is already found
    (LP: #1848211)

 -- Balint Reczey <email address hidden> Wed, 27 Nov 2019 13:52:54 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for valgrind has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Funous (funous)]

This issue still exists in Valgrind 3.16.1 compiled from source on Ubuntu 20.04. The attached patch still applies and fixes the issue.

[Mark J. Wielaard (3y9m2vcw-ll9d-fkzsxrqg)]

Is this still a real problem?

I am not sure I understand how this can happen:

    With dwz the .gnu_debuglink section may appear duplicated in the
    debug file referenced originally in the .gnu_debuglink section.

I assume that is meant to say .gnu_debugaltlink. But if you somehow got a .gnu_debugaltlink in both the main ELF file and the separate .debug image then something must have gone wrong in either the dwz step or the strip step.

Does this mean the main ELF file also still has some .debug_info section? Could you show the section headers of the main ELF file, the .debug file and the alt file?

[Balint Reczey (rbalint)]

Please see the linked original bug for the reproducer.

[Mark J. Wielaard (3y9m2vcw-ll9d-fkzsxrqg)]

(In reply to Balint Reczey from comment #4)
> Please see the linked original bug for the reproducer.

I looked at https://bugs.launchpad.net/ubuntu/+source/valgrind/+bug/1848211 but I still don't fully understand how/why/when this happens.

There is some confusion because the patch is for .gnu_debugaltlink but the bug talks about .gnu_debuglink (but I assume it really is the altlink).

I think what would be helpful was an explanation of how dh_dwz works. It looks like it runs dwz before the debuginfo is stripped into separate files. Which might explain why things look garbled (it should be run on the split .debug files of the package).

It looks like the original bug was on arm32 with ld.so. But later in the bug report a x86_64 issue with glib2 library.

[Mark J. Wielaard (3y9m2vcw-ll9d-fkzsxrqg)]

It was a bit of a struggle to get this reproduced. Installing debuginfo (dbgsym) packages is not the simplest thing. But once you finally manage then you can indeed replicate it simply with: valgrind/vg-in-place -v /usr/bin/gio ~

--3475-- Reading syms from /usr/bin/gio
--3475-- Considering /usr/lib/debug/.build-id/50/d6f5b1c2450bf79554ee66724c911a73f1f635.debug ..
--3475-- .. build-id is valid
--3475-- WARNING: Serious error when reading debug info
--3475-- When reading debug info from /usr/bin/gio:
--3475-- debuginfo section duplicates a section in the main ELF file

And indeed /usr/bin/gio has both a .gnu_debugaltlink and .gnu_debuglink section.

The .gnu_debugaltlink points to /usr/lib/debug/.dwz/x86_64-linux-gnu/libglib2.0-bin.debug

The .gnu_debuglink points to "d6f5b1c2450bf79554ee66724c911a73f1f635.debug" (odd imho that is named after part of the build-id), luckily we look it up through the build-id and end up at /usr/lib/debug/.build-id/50/d6f5b1c2450bf79554ee66724c911a73f1f635.debug (which turns out to be a real file, not a symlink?)

That .debug file also has a .debug_gnualtlink section which points to the same /usr/lib/debug/.dwz/x86_64-linux-gnu/libglib2.0-bin.debug (side-note, given that this is an absolute path instead of a relative path, this makes things even harder to resolve unless everything is installed in the correct place, or you resolve everything through ids).

Since they are the same we can indeed just pick one. I believe this is a really odd thing to do, but we shouldn't really crash and burn.

The proposed patch seems to be a simplest way to do that.

[Mark J. Wielaard (3y9m2vcw-ll9d-fkzsxrqg)]

commit 8b1961511c93962ea2a9b918af8e9c32e3c24d71
Author: Balint Reczey <email address hidden>
Date: Thu Nov 28 13:34:21 2019 +0100

    Don't look for debug alt file in debug image if it is already found

    With dwz the .gnu_debuglink section may appear duplicated in the
    debug file referenced originally in the .gnu_debuglink section.

    https://bugs.launchpad.net/ubuntu/+source/valgrind/+bug/1848211

    https://bugs.kde.org/show_bug.cgi?id=396656
    https://bugs.kde.org/show_bug.cgi?id=427969

    Signed-off-by: Balint Reczey <email address hidden>