gdbserver inside qemu does not stop on breakpoints

[agonified]

When debugging a kernel, gdb can connect to the remote target running in the qemu but the server does not stop execution on breakpoints set before. This also applies to kvm.

Also, looking at the stack trace that I see when I manually stop execution, gdbserver seems to be very confused about where the current execution frame is: no matter what is happening in the VM, gdbserver reports the same (somewhat irrelevant) stack trace.

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: qemu (not installed)
ProcVersionSignature: Ubuntu 3.0.0-13.22-generic 3.0.6
Uname: Linux 3.0.0-13-generic x86_64
NonfreeKernelModules: wl
ApportVersion: 1.23-0ubuntu4
Architecture: amd64
Date: Thu Dec 8 17:33:28 2011
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Beta amd64 (20110921.2)
KvmCmdLine: Error: command ['ps', '-C', 'kvm', '-F'] failed with exit code 1: UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
MachineType: Acer Aspire 3820
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.0.0-13-generic root=UUID=64cecdc0-10e1-410e-b9f6-c74238a53651 ro quiet splash acpi_osi=Linux vt.handoff=7
SourcePackage: qemu-kvm
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/04/2010
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: V1.13
dmi.board.asset.tag: No Asset Tag
dmi.board.name: Aspire 3820
dmi.board.vendor: Acer
dmi.board.version: V1.13
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 10
dmi.chassis.vendor: Acer
dmi.chassis.version: V1.13
dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvrV1.13:bd05/04/2010:svnAcer:pnAspire3820:pvrV1.13:rvnAcer:rnAspire3820:rvrV1.13:cvnAcer:ct10:cvrV1.13:
dmi.product.name: Aspire 3820
dmi.product.version: V1.13
dmi.sys.vendor: Acer

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug.

Could you show us exactly how to reproduce your failure and what to look for? I see you are on oneiric. We have somewhat recently, successfully, used gdb remotely on oneiric, but it was with very early boot which may affect this. Could you tell us which release+kernel were running in the guest?

[agonified (hakanakkan)]

Guest is a 2.6.38-10-server x86_64 kernel with ubuntu patches applied (configured to have some set of modules built-in + debugging symbols + non-preemptive + HZ=100). I can provide the config file if necessary. Root fs contains a minimal 10.04 server.

Here is how I launch the VM:
$ qemu-system-x86_64 -s -m 512 -drive file="root1.qcow2" -kernel ../../linux-2.6.38/arch/x86/boot/bzImage -append " console=tty1 root=/dev/sda1 ro "

VM launches fine and drops to a login shell. Then I start gdb and connect to the server:
$ gdb ../../linux-2.6.38/vmlinux
{Init messages...}
Reading symbols from ../../linux-2.6.38/vmlinux...done.
(gdb) target remote :1234
Remote debugging using :1234
0xffffffff81076a0b in pvclock_clocksource_read (src=0xffffffff82201fd8) at arch/x86/kernel/pvclock.c:106
106 if ((valid_flags & PVCLOCK_TSC_STABLE_BIT) &&
(gdb)

At this point the VM is unresponsive. Then I set a breakpoint at schedule() (which should be called 100 times per second) and let the VM go.

(gdb) b schedule
Breakpoint 1 at 0xffffffff81b2b3f0: file kernel/sched.c, line 3946.
(gdb) c
Continuing.

I would expect the debugger to hit the breakpoint but it never does. VM is fully functional at this point. Then I pause the VM again with ctrl+c and print the stack trace:

^C
Program received signal SIGINT, Interrupt.
0xffffffff81076a0b in pvclock_clocksource_read (src=0xffffffff82201fd8) at arch/x86/kernel/pvclock.c:106
106 if ((valid_flags & PVCLOCK_TSC_STABLE_BIT) &&
(gdb) where
#0 0xffffffff81076a0b in pvclock_clocksource_read (src=0xffffffff82201fd8) at arch/x86/kernel/pvclock.c:106
#1 0xffffffff81053255 in arch_safe_halt () at /DOCS/nmc/linux-2.6.38/arch/x86/include/asm/paravirt.h:110
#2 default_idle () at arch/x86/kernel/process.c:391
#3 0xffffffff8104921b in cpu_idle () at arch/x86/kernel/process_64.c:139
#4 0xffffffff81ad9cce in device_irq_init (pdata=0xffffffff824cdfa0, chip=0xffffffff824c99c0) at drivers/mfd/88pm860x-core.c:508
#5 device_8607_init (chip=0xffffffff824c99c0, i2c=<optimized out>, pdata=0xffffffff824cdfa0) at drivers/mfd/88pm860x-core.c:682
#6 0xffffffff8247bc6f in start_kernel () at init/main.c:734
#7 0xffffffff8247b347 in x86_64_start_reservations (real_mode_data=<optimized out>) at arch/x86/kernel/head64.c:124
#8 0xffffffff8247b44d in x86_64_start_kernel (real_mode_data=0xffff88001fffb880 " console=tty1") at arch/x86/kernel/head64.c:94
#9 0x0000000000000000 in ?? ()
(gdb)

I see the same stack trace every single time I stop the VM; while sitting idle on the console, while running a busy-loop application, while doing net/disk IO. I also set other breakpoints at frequently executed kernel functions like do_timer() but the VM never stops at these breakpoints.

Please let me know if more information is necessary.

[Serge Hallyn (serge-hallyn)]

Thanks - please do attach your .config, and I'll try to reproduce.

[agonified (hakanakkan)]

I reconfigured the kernel from the default configuration with the debugging symbols and it now seems to be working okay. There is something in the config I previously posted that makes things weird. The question is how can we track it down?

[Serge Hallyn (serge-hallyn)]

Do you mind posting the new, working, config? There may (likely will) be too much noise in the diff, but it's worth looking for clues.

[agonified (hakanakkan)]

This config lets me debug the kernel. However, I can not start debugging with qemu's -S option. With -S, qemu waits for a gdb connection to start the VM but gdb can't connect to the default port (:1234). Connection times out.

If I start with -s, then everything is good.

[Launchpad Janitor (janitor)]

[Expired for qemu-kvm (Ubuntu) because there has been no activity for 60 days.]

[Horst Schirmeier (horst)]

I'm seeing the very same problem, please reopen.

Ubuntu 12.10 amd64 (Linux 3.5.0-25-generic x86_64), qemu-kvm 1.2.0+noroms-0ubuntu2.12.10.3. Guest is an i386 eCos test kernel (I can attach it if necessary; I guess you can use anything bootable here), started either via
 qemu-system-i386 -kernel ecos.elf -s -S
or via an ISO image with grub2:
 qemu-system-i386 -cdrom ecos.iso -s -S

I'm attaching GDB with:
 gdb ecos.elf
(gdb) target remote localhost:1234
(gdb) b main
(gdb) c
The breakpoint never hits, although this perfectly works on my 32-bit Debian Squeeze box.

If I start qemu-system-i386 with "-machine accel=tcg", the breakpoint matches very quickly, kvm halts, and the GDB prompt allows me to inspect the state. If I start qemu-system-i386 with "-machine accel=kvm" (the default), I observe the aforementioned behaviour.

[Serge Hallyn (serge-hallyn)]

 status confirmed

[Horst Schirmeier (horst)]

More details:
 - It works if I use hbreak (GDB hardware-assisted breakpoints) instead of break for the very first breakpoint.
 - After this first breakpoint, normal "break" breakpoints work fine, too.
 - Accoording to http://chemnitzer.linux-tage.de/2012/vortraege/folien/1061-VirtualDebugging.pdf slide 18 this might be due to GDB not properly dealing with protocol changes. In this case, this is a GDB bug/issue, not a qemu-kvm one.

[Ciro Santilli 六四事件 法轮功 (cirosantilli)]

As mentioned by Horst, hbreak is needed.

If you are getting "Remote 'g' packet reply is too long", see: https://sourceware.org/bugzilla/show_bug.cgi?id=13984