gdb doesn't search in debug-file-directory for .gnu_debugaltlink

Bug #1818918 reported by Brian Murray on 2019-03-06
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Apport
High
Brian Murray
apport (Ubuntu)
Undecided
Unassigned
gdb (Ubuntu)
Undecided
Unassigned

Bug Description

As far as I can tell gdb version 8.2.90 isn't searching the debug-file-directory, which I set, for the '.gnu_debugaltlink' which is in the debug symbols. Here's the error I'm seeing:

Type "apropos word" to search for commands related to "word".
Reading symbols from /srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox//usr/bin/gnome-calculator...
Reading symbols from /srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox/usr/lib/debug/.build-id/95/59c4c5ee30eb66d47bb9bd64784a69c9a6de6b.debug...
could not find '.gnu_debugaltlink' file for /srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox/usr/lib/debug/.build-id/95/59c4c5ee30eb66d47bb9bd64784a69c9a6de6b.debug
(No debugging symbols found in /srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox/usr/lib/debug/.build-id/95/59c4c5ee30eb66d47bb9bd64784a69c9a6de6b.debug)

Here's part of an strace of what's going on behind the scenes:

lstat("/srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox/usr/lib/debug/.build-id/95/59c4c5ee30eb66d47bb9bd64784a69c9a6de6b.debug", {st_mode=S_IFREG|0644, st_size=839744, ...}) = 0
openat(AT_FDCWD, "/usr/lib/debug/.dwz/x86_64-linux-gnu/gnome-calculator.debug", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

This is the only time "/usr/lib/debug" is searched, generally "/srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox/usr/lib/debug/" is used. I'll attach the full strace though.

For completeness here's the gdb command I'm using:

Calling gdb command: '/srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2' '/srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox/usr/bin/gdb' --ex 'set debug-file-directory /srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox/usr/lib/debug' --ex 'set solib-absolute-prefix /srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox' --ex 'add-auto-load-safe-path /srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox' --ex 'set solib-search-path /srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox/lib/x86_64-linux-gnu' --ex 'set data-directory /srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox/usr/share/gdb' --ex 'file "/srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox//usr/bin/gnome-calculator"' --ex 'core-file /tmp/apport_core_1b6dn6np'

Related branches

Brian Murray (brian-murray) wrote :
tags: added: disco
Brian Murray (brian-murray) wrote :

This is still an issue with gdb version 8.2.90.20190311-0ubuntu1 in disco.

Matthias Klose (doko) wrote :

please can you can up with a test case not involving snaps? Asking because I can't.

Changed in gdb (Ubuntu):
status: New → Incomplete
Brian Murray (brian-murray) wrote :

This test case doesn't involve snaps, notice the file switch to gdb:

--ex 'file "/srv/vms/apport-sandbox-dir/Ubuntu 19.04/amd64/report-sandbox//usr/bin/gnome-calculator"'

This is still an issue with the version of gdb from eoan, 8.3-0ubuntu1, and eoan's gnome-calculator package.

Calling gdb command: '/srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2' '/srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/bin/gdb' --ex 'set debug-file-directory /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug' --ex 'set solib-absolute-prefix /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox' --ex 'add-auto-load-safe-path /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox' --ex 'set solib-search-path /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/lib/x86_64-linux-gnu:/srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/x86_64-linux-gnu' --ex 'set data-directory /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/share/gdb' --ex 'file "/srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox//usr/bin/gnome-calculator"' --ex 'core-file /tmp/apport_core_b_z3909f'
/srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/bin/gdb: warning: Couldn't determine a path for the index cache directory.
GNU gdb (Ubuntu 8.3-0ubuntu1) 8.3
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox//usr/bin/gnome-calculator...
Reading symbols from /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.build-id/42/410fc27971e3eb1d581449a1687495a876ab5b.debug...
could not find '.gnu_debugaltlink' file for /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.build-id/42/410fc27971e3eb1d581449a1687495a876ab5b.debug
(No debugging symbols found in /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.build-id/42/410fc27971e3eb1d581449a1687495a876ab5b.debug)
[New LWP 14762]
[New LWP 14766]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/gnome-calculator'.

Attached you'll find the core file for gnome-calculator. Is there anything more I can give you?

Launchpad Janitor (janitor) wrote :

[Expired for gdb (Ubuntu) because there has been no activity for 60 days.]

Changed in gdb (Ubuntu):
status: Incomplete → Expired
Changed in gdb (Ubuntu):
status: Expired → New
Brian Murray (brian-murray) wrote :

I recently started seeing issues with some other applications too e.g.:

Reading symbols from /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox//usr/lib/gdm3/gdm-session-worker...
Reading symbols from /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.build-id/d8/4d9e76a9073d1195af86ff6620382437e41977.debug...
could not find '.gnu_debugaltlink' file for /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.build-id/d8/4d9e76a9073d1195af86ff6620382437e41977.debug
(No debugging symbols found in /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.build-id/d8/4d9e76a9073d1195af86ff6620382437e41977.debug)
...
Reading symbols from /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox//usr/bin/gnome-shell...
Reading symbols from /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.build-id/2c/28a1bf7a132275c4fd2f9f44011c043d4424e5.debug...
could not find '.gnu_debugaltlink' file for /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.build-id/2c/28a1bf7a132275c4fd2f9f44011c043d4424e5.debug
(No debugging symbols found in /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.build-id/2c/28a1bf7a132275c4fd2f9f44011c043d4424e5.debug)

Matthias Klose (doko) wrote :

looking up the .gnu_debugaltlink in a normal debug session works perfectly fine. Please can you show this behavior without any sandbox, or try to diagnose why it doesn't work in this sandbox? I am not familiar with this sandbox environment.

Changed in gdb (Ubuntu):
status: New → Incomplete
Brian Murray (brian-murray) wrote :

I've discovered that I can workaround this issue by modifying the argument passed to the 'set debug-file-directory' option of gdb e.g. by using the following:

'set debug-file-directory /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.dwz/'

I no longer see the error message regarding "could not find '.gnu_debugaltlink' file for...'. Originally the 'set debug-file-directory' argument was:

'set debug-file-directory /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/'

Additionally its worth noting that if I use both paths e.g.:

'set debug-file-directory /srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/.dwz/:/srv/vms/apport-sandbox-dir/Ubuntu 19.10/amd64/report-sandbox/usr/lib/debug/'

I receive the error message again.

Launchpad Janitor (janitor) wrote :

[Expired for gdb (Ubuntu) because there has been no activity for 60 days.]

Changed in gdb (Ubuntu):
status: Incomplete → Expired
Changed in gdb (Ubuntu):
status: Expired → New
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in gdb (Ubuntu):
status: New → Confirmed
tags: added: fr-823
Brian Murray (brian-murray) wrote :

I've managed to workaround this in apport by creating a symlink from the host system's /usr/share/debug/.dwz to the gdb sandbox's /usr/share/debug/.dwz folder. That being said I still think there is an issue with gdb not checking the debug-file-directory for the "Separate debug info file" in the .gnu_debugaltlink section.

Changed in apport:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Brian Murray (brian-murray)
Changed in apport:
status: In Progress → Fix Released
Brian Murray (brian-murray) wrote :

I've come up with a simpler test case on an Ubuntu 20.10 system which follows:

1) Execute 'gdb --args cat'
2) in gdb type 'run'
3) press Ctrl-Z
4) in gdb type 'generate-core-file /tmp/cat.core'
5) download the dbgsym for coreutils
6) Extract them 'dbg-deb -x coreutils-dbgsym_8.32-3ubuntu1_amd64.ddeb /tmp/dbgsym'
7) Execute "gdb --ex 'file /bin/cat' --ex 'core-file /tmp/cat.core' --ex 'set debug-file-directory /tmp/dbgsym/usr/lib/debug'"

In gdb you'll see the following error message:

Reading symbols from /bin/cat...
(No debugging symbols found in /bin/cat)

Running "objdump -g /bin/cat" we can see the location for the separate debug info:

Contents of the .gnu_debugaltlink section (loaded from /bin/cat):

  Separate debug info file: /usr/lib/debug/.dwz/x86_64-linux-gnu/coreutils.debug
  Build-ID (0x14 bytes):
 cb 5b e4 8a 6d b2 52 6e 9c 80 8d 64 ec 4b 1f b7 7f 0c ca 9e

Contents of the .gnu_debuglink section (loaded from /bin/cat):

  Separate debug info file: a7cee6aca864b8f79dfaa8a267855333b445c1.debug
  CRC value: 0x5e28a31d

The separate debug info file exists in /tmp/dbgsym:

[ 11:07AM 10857 ] [ bdmurray@impulse:/tmp/dbgsym ]
 $ find . -name a7cee\*.debug
./usr/lib/debug/.build-id/fb/a7cee6aca864b8f79dfaa8a267855333b445c1.debug

However despite setting debug-file-directory an strace reveals that /tmp/dbgsym is not searched:

 $ grep a7cee /tmp/gdb-bin-cat.trace
access("/usr/lib/debug/.build-id/fb/a7cee6aca864b8f79dfaa8a267855333b445c1.debug", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/bin/a7cee6aca864b8f79dfaa8a267855333b445c1.debug", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/bin/.debug/a7cee6aca864b8f79dfaa8a267855333b445c1.debug", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/debug//bin/a7cee6aca864b8f79dfaa8a267855333b445c1.debug", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Brian Murray (brian-murray) wrote :

Here's the full strace for the /bin/cat test case.

Brian Murray (brian-murray) wrote :

Oh and using either of '/tmp/dbgsym' or /tmp/dbgsym/usr/lib/debug' for the debug-file-directory works.

Brian Murray (brian-murray) wrote :

This is also true with gdb 10.1 from hirsute-proposed.

Tiago Stürmer Daitx (tdaitx) wrote :

> 7) Execute "gdb --ex 'file /bin/cat' --ex 'core-file /tmp/cat.core' --ex 'set debug-file-directory /tmp/dbgsym/usr/lib/debug'"

Try setting the debug-file-directory first, it should work.
gdb --ex 'set debug-file-directory /tmp/dbgsym/usr/lib/debug:/usr/lib/debug' --ex 'file /bin/cat' --ex 'core-file /tmp/cat.core'

Still, this does not seem to invalidate the original bug report or the one in comment #4, as they are setting 'file' and 'core' last. It is just that comment #12 does not seem to indicate a valid reproducer.

Brian Murray (brian-murray) wrote :

Ah, thanks for pointing that out but there is still an issue with the test case in comment #12 as the .gnu_debugaltlink file is still not loaded.

Reading symbols from /bin/cat...
Reading symbols from /tmp/dbgsym/usr/lib/debug/.build-id/fb/a7cee6aca864b8f79dfaa8a267855333b445c1.debug...
could not find '.gnu_debugaltlink' file for /tmp/dbgsym/usr/lib/debug/.build-id/fb/a7cee6aca864b8f79dfaa8a267855333b445c1.debug
(No debugging symbols found in /tmp/dbgsym/usr/lib/debug/.build-id/fb/a7cee6aca864b8f79dfaa8a267855333b445c1.debug)
[New LWP 302900]
Core was generated by `/bin/cat'.
Program terminated with signal SIGTSTP, Stopped (user).

Tiago Stürmer Daitx (tdaitx) wrote :

I tried the testcase from comment #12 in Focal and Hirsute and it worked just fine:

$ gdb --ex 'set debug-file-directory /tmp/dbgsym/usr/lib/debug:/usr/lib/debug' --ex 'file /bin/cat' --ex 'core-file /tmp/cat.core'
Type "apropos word" to search for commands related to "word".
Reading symbols from /bin/cat...
Reading symbols from /tmp/dbgsym/usr/lib/debug/.build-id/b3/57ed53c8c9cb1a312f83b28982304effae0135.debug...
[New LWP 2094475]
Core was generated by `/bin/cat'.
Program terminated with signal SIGTSTP, Stopped (user).
#0 0x00007ffff7eb6142 in __GI___libc_read (fd=fd@entry=0, buf=buf@entry=0x7ffff7faa000, nbytes=nbytes@entry=131072) at ../sysdeps/unix/sysv/linux/read.c:26

# hirsute
$ gdb --ex 'set debug-file-directory /tmp/dbgsym/usr/lib/debug:/usr/lib/debug' --ex 'file /bin/cat' --ex 'core-file /tmp/cat.core'
Reading symbols from /bin/cat...
Reading symbols from /tmp/dbgsym/usr/lib/debug/.build-id/fb/a7cee6aca864b8f79dfaa8a267855333b445c1.debug...
[New LWP 14651]
Core was generated by `/usr/bin/cat'.
Program terminated with signal SIGTSTP, Stopped (user).
#0 0x00007ffff7edccb2 in __GI___libc_read (fd=fd@entry=0, buf=buf@entry=0x7ffff791e000, nbytes=nbytes@entry=131072) at ../sysdeps/unix/sysv/linux/read.c:26
26 ../sysdeps/unix/sysv/linux/read.c: No such file or directory.

If the dbgsym is installed and I use (with intentional typo):
set debug-file-directory /tmp/dbgsym/usr/lib/debugnot:/usr/lib/debug
it then loads symbols correctly from /usr/lib/debug

Note, this was tested under Focal (host) and Hirsute (lxd container).

Brian Murray (brian-murray) wrote :

This is still an issue with hirsute afaict.

(hirsute-amd64)root@impulse:/tmp# gdb --ex 'set debug-file-directory /tmp/dbgsym/usr/lib/debug:/usr/lib/debug' --ex 'file /bin/cat' --ex 'cor
e-file /tmp/cat.core'
GNU gdb (Ubuntu 10.1-0ubuntu1) 10.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word".
Reading symbols from /bin/cat...
Reading symbols from /tmp/dbgsym/usr/lib/debug/.build-id/bc/06d3bee37b8c7e67b31cb2689cb351102ae73b.debug...
could not find '.gnu_debugaltlink' file for /tmp/dbgsym/usr/lib/debug/.build-id/bc/06d3bee37b8c7e67b31cb2689cb351102ae73b.debug
(No debugging symbols found in /tmp/dbgsym/usr/lib/debug/.build-id/bc/06d3bee37b8c7e67b31cb2689cb351102ae73b.debug)
[New LWP 1146067]
Core was generated by `/bin/cat'.
Program terminated with signal SIGTSTP, Stopped (user).
#0 0x00007ffff7edca92 in read () from /lib/x86_64-linux-gnu/libc.so.6

Tiago Stürmer Daitx (tdaitx) wrote :

I was able to reproduce what Brian is reporting here.

If the -dbgsym for the package is installed, gdb works and reports that it is reading from the /tmp/dbgsym path. When -dbgsym package is not installed then it fails with the
'could not find '.gnu_debugaltlink' file for ...' message

Tiago Stürmer Daitx (tdaitx) wrote :

So the issue is: if there is a .gnu_debugaltlink GDB will try to load that file and throw an error if it can't. That path is absolute and GDB does _not_ look for that path/file anywhere else, not even inside 'debug-file-directory'.

GDB seems to only look at section .gnu_debugaltlink in debug/.build-id/nn/nnnnnn.debug, it does not seem to use that section from the binary at all.

Due to that, another workaround is to modify that section to point to the 'right' place:

1) use objcopy to dump .gnu_debugaltlink from debug/.build-id/nn/nnnnnn.debug into a file
2) use sed to modify the path in the dump file
3) use objcopy to update .gnu_debugaltlink section in debug/.build-id/nn/nnnnnn.debug

As in:
$ objcopy --dump-section .gnu_debugaltlink=altlink /tmp/dbgsym/usr/lib/debug/.build-id/76/e9f820204912084fd156c593b2c92f1a4b51f1.debug
$ sed -i 's:^:/tmp/dbgsym:' altlink
$ objcopy --update-section .gnu_debugaltlink=altlink /tmp/dbgsym/usr/lib/debug/.build-id/76/e9f820204912084fd156c593b2c92f1a4b51f1.debug

@tdaitx asked me to take a look at this problem, and here is what I found.

1) As he said above, the problem is that Debian/Ubuntu generate .gnu_debugaltlink sections containing full pathnames to the DWZ alt debug files. IMO, we should be using dwz's "--relative" option when invoking it via dh_dwz. I will send a merge request on Salsa and propose that we adopt this flag as a distro.

2) While we could use the workaround described in the last comment, I think it is better if GDB is adjusted to cope with this scenario (i.e., having a full pathname on .gnu_debugaltlink, but having the actual DWZ file in another directory that is also provided as the debug-file-directory to GDB). I went ahead and submitted a patch to GDB to do just that:

  https://sourceware.org/pipermail/gdb-patches/2020-November/173276.html

I think searching for the DWZ files using the debug-file-directory provided by the user is a sensible approach, and is also something that other projects (namely elfutils) seem to do.

All in all, as I said in (1), I think the best long-term solution is to adjust dh_dwz to put relative pathnames in the .gnu_debugaltlink. Arguably, we could also create a symlink in /usr/lib/debug/.build-id/ and make it point to the corresponding file inside /usr/lib/debug/.dwz/, but that is an orthogonal issue and would not help with this specific bug.

Tiago Stürmer Daitx (tdaitx) wrote :

@sergiodj many thanks for the help here

It was a dwz man page somewhere that made me think we might be misusing .gnu_debugaltlink somehow (didn't even know about dh_dwz), but on a quick search I couldn't understand how the path was supposed to look like so I could test that idea with my replace-altlink-section-hack.

And if GDB can support looking for .dwz on the debug-file-drectory even with hardcoded paths, all the better.

Many, many thanks for taking the time to look at this and come up with these 2 solutions. We really appreciate it.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers