gdb crashes when trying to start a debugging session

[Peter Bennett]

This bug log applies to Ubuntu Mate for raspberry PI.

I cannot use gdb to debug a program. See below. This happens on any program, even programs that come with ubuntu, for example try gdb /bin/less. It always crashes in d-linux-armhf.so.3

gdb mythfrontend
GNU gdb (Ubuntu 7.11-0ubuntu1) 7.11
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from mythfrontend...done.
(gdb) start
Temporary breakpoint 1 at 0x62c22: file main.cpp, line 1674.
Starting program: /srv/storage/homexenial/peter/proj/mythtv-build/myth-p28/usr/bin/mythfrontend

Program received signal SIGSEGV, Segmentation fault.
0x76fd9822 in ?? () from /lib/ld-linux-armhf.so.3
(gdb)

[Peter Bennett (pgbennett)]

I did not mean to change it from Ubuntu Mate to Ubuntu. I am not able to change it back to Ubuntu Mate since that is not in the list.

[Paul White (paulw2u)]

Probably best to reassign to gdb project rather than to Ubuntu MATE.

[matteo (matteoids)]

Hello,

I have the same problem using Ubutu Mate for Raspberry Pi (ubuntu version 16.04), gdb v.7.11.1, and Eclipse CDT v.8.6.

Simply running in debug an "Hello world" project, I found the same segmentation fault:

Program received signal SIGSEGV, Segmentation fault.
0x76fd9822 in ?? () from /lib/ld-linux-armhf.so.3

Someone can help me? How can I solve this problem and debug my project from eclipse perspective (or from gdb using command prompt)?

Thanks in advance.
Matteo

[Peter Bennett (pgbennett)]

There is a workaround. Start the program, then from another command prompt or from an ssh remote login, use gdp -p xxxxx where xxxxx is the process number. This works without crashing. If you need to debug something that happens before you can get in from another command prompt, add to the program a command that stops process at the beginning of main, for example a call to gets, which will wait for you to press enter before continuing.

[Max Barry (max-maxbarry)]

Same here on Raspberry Pi 3 Model B 1.2. Any idea on a cause?

[Frikkie Thirion (frixbuntu)]

Good day,

I'm developing on the Allwinner A20 & A33 CPUs, which contains ARM cores.
I've first noticed the segfault during remote debugging, using gdbserver,
from my host PC to the target. Debugging the the application directly on
the target did not cause this problem. Performing the remote debugging from
a 32-bit host to the 32-bit target made no difference. That seemed to
eliminate the possibility of a 64/32 bit compatibility issue.

My setup:
Host PC:
  Kubuntu 16.04 64-bit
  Toolchain: gcc-linaro-6.1.1-2016.08
           : gdb version (Linaro_GDB-2016.08) 7.11.1.20160801-git
            Obtained from: https://releases.linaro.org/components/toolchain/binaries
Target:
  Ubuntu 16.04 ARMHF
  : GDB version: (Ubuntu 7.11.1-0ubuntu1~16.04) 7.11.1

Illustrating the problem
------------------------
Program: main.c
  #include <stdio.h>
  int main (void)
  {
    printf("Hallo world!\n");
    return 0;
  }
Compile on host PC:
  $ arm-linux-gnueabihf-gcc -g main.c -o main_32
File signature:
  $ file main_32
    main_32: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked,
          interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 2.6.32,
          BuildID[sha1]=cc1a4a1ff17da63f9f8362c84dffe54ee7fff4ff, not stripped

Remote target:
The executable (main_32) needs to be copied to the remote target
$ gdbserver :5555 main_32
  Process main_32 created; pid = 904
  Listening on port 5555
  Remote debugging from host 192.168.7.2

Host PC:
$ arm-linux-gnueabihf-gdb ./main_32
  (gdb) target remote 192.168.7.1:5555
  Remote debugging using 192.168.7.1:5555
  Reading /lib/ld-linux-armhf.so.3 from remote target...
  warning: File transfers from remote targets can be slow. Use "set sysroot" to access files
           locally instead.
  Reading /lib/ld-linux-armhf.so.3 from remote target...
  Reading symbols from target:/lib/ld-linux-armhf.so.3...Reading /lib/ld-2.23.so from remote
          target...
  Reading /lib/.debug/ld-2.23.so from remote target...
  (no debugging symbols found)...done.
  0xb6fd7a40 in ?? () from target:/lib/ld-linux-armhf.so.3

(gdb) continue
  Continuing.
  Program received signal SIGSEGV, Segmentation fault.
  0xb6fd9822 in ?? () from target:/lib/ld-linux-armhf.so.3

To try and figure out where the segfault occurred the debug symbols need to be loaded.
The gdb command 'set debug-file-directory' can be used to accomplish this. It seems that this command is only relative to the host PC filesystem and not according to the remote target file system.
With 'set sysroot' the debugger can also be instructed to load all data locally instead of transferring it runtime over the network.

Make sure the debug symbols are on the remote file system:
# apt-get install libc6-dbg
Copy the full contents of the SD card residing on the Allwinner A20 rootfs to a local directory on
the host PC: /opt/gnueabihf_rootfs

Illustrating the solution
-------------------------
Remote target:
$ gdbserver :5555 main_32
  Process main_32 created; pid = 912
  Listening on port 5555
  Remote debugging from host 192.168.7.2

  Hallo world!
  Child exited with status 0

Host PC:
$ arm-linux-gnueabihf-gdb ./main_32
  (gdb) set deb...

[Shervin Emami (shervin-emami)]

I've been having the same SEGV crash when debugging an ARM device remotely using gdbserver. I can confirm that Frikkie's workaround of attaching to the process from a separate console is working for me. But it's by using "--attach localhost:XXXX" in gdbserver on the remote ARM device, instead of using "-p XXXX", since gdbserver doesn't support "-p".

[Jasem Mutlaq (mutlaqja)]

Anyone working on this?

[Denys Yurchenko (magnetronus)]

Workaround of Frikkie Thirion works fine.

What about fix?

[gust (gustzhang)]

as Frikkie Thirion (frixbuntu) #6 figure ,to install libc6-dbg solved it .

sudo apt-get install libc6-dbg

[Chris Coulson (chrisccoulson)]

This occurs when gdb sets breakpoints on various probe events in the dynamic loader. The probe event locations are exported from ld.so as SDT markers, but gdb needs to know whether ARM or Thumb instructions are being exported at each marker so that it can insert the appropriate breakpoint instruction sequence. It does this by mapping the probe location to a function symbol (see arm_pc_is_thumb in gdb/arm-tdep.c), and using the target address of the symbol to determine if the function is called in Thumb or ARM more (bit 0 of the target address will be set for Thumb mode).

The problem here is that gdb can't map any of the probes to a symbol if the debug symbols aren't installed, and arm_pc_is_thumb returns false in this case (indicating ARM instructions).

But the probes that gdb sets breakpoints on are in dl_main, and dl_main is meant to be called in Thumb mode. See the output of readelf, noting that bit 0 of the target address is set:
$ readelf -s /usr/lib/debug/lib/arm-linux-gnueabihf/ld-2.27.so | grep dl_main
    42: 0000172d 10516 FUNC LOCAL DEFAULT 11 dl_main

So without the debug symbols installed, gdb inserts the ARM breakpoint instruction sequence for the dynamic loader probes, when it should be inserting the Thumb instruction sequence. This is why it crashes.

[Chris Coulson (chrisccoulson)]

(Note, I looked at this because it was breaking rustc tests on armhf. The workaround is the same - install libc6-dbg to make the crash go away)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.