crash at qemu_iohandler_poll (iohandler.c:124) on macos 10.8.2

[Christopher Mason]

I'm seeing consistent hangs / crashes on MacOS 10.8.2 with 1.3.0. I've tried both gcc-4.2 and clang. I've tried a half a dozen different images/kernels.

I configured qemu like this:

./configure --disable-sdl --disable-kvm --enable-cocoa --cc=gcc-4.2 --host-cc=gcc-4.2 --enable-debug --extra-cflags=-g --extra-ldflags=-g

And ran it like this:

qemu-system-arm -nographic -M versatilepb -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.img-2.6.32-5-versatile -hda debian_squeeze_armel_standard.qcow2 -append "root=/dev/sda1 console=ttyAMA0"

With images, kernel, and initrd described here:

http://psellos.com/2012/08/2012.08.qemu-arm-osx.html

And I get:

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x000000010142f2d0
0x000000010142f2d0 in ?? ()

(gdb) bt
#0 0x000000010142f2d0 in ?? ()
#1 0x000000010016e209 in qemu_iohandler_poll (readfds=0x10097ca00, writefds=0x10097ca80, xfds=0x10097cb00, ret=4) at iohandler.c:124
#2 0x0000000100172acf in main_loop_wait (nonblocking=0) at main-loop.c:418
#3 0x0000000100207bbf in main_loop () at vl.c:1765
#4 0x000000010020e7b0 in qemu_main (argc=12, argv=0x7fff5fbff360, envp=0x7fff5fbff3c8) at vl.c:3992
#5 0x00000001001d6013 in main (argc=12, argv=0x7fff5fbff360) at ui/cocoa.m:884
(gdb) frame 1
#1 0x000000010016e209 in qemu_iohandler_poll (readfds=0x10097ca00, writefds=0x10097ca80, xfds=0x10097cb00, ret=4) at iohandler.c:124
124 ioh->fd_read(ioh->opaque);
Current language: auto; currently c
(gdb) p ioh
$1 = (IOHandlerRecord *) 0x10142f110
(gdb) p *ioh
$2 = {
  fd_read_poll = 0,
  fd_read = 0x10017212b <sigfd_handler>,
  fd_write = 0,
  opaque = 0x3,
  next = {
    le_next = 0x0,
    le_prev = 0x105d00bc0
  },
  fd = 3,
  deleted = false
}

[Stefan Hajnoczi (stefanha)]

On Mon, Dec 31, 2012 at 08:46:45PM -0000, Christopher Mason wrote:
> Public bug reported:
>
> I'm seeing consistent hangs / crashes on MacOS 10.8.2 with 1.3.0. I've
> tried both gcc-4.2 and clang. I've tried a half a dozen different
> images/kernels.

Which QEMU version are you building? Have you tried qemu.git/master?

> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_PROTECTION_FAILURE at address: 0x000000010142f2d0
> 0x000000010142f2d0 in ?? ()
>
> (gdb) bt
> #0 0x000000010142f2d0 in ?? ()
> #1 0x000000010016e209 in qemu_iohandler_poll (readfds=0x10097ca00, writefds=0x10097ca80, xfds=0x10097cb00, ret=4) at iohandler.c:124
> #2 0x0000000100172acf in main_loop_wait (nonblocking=0) at main-loop.c:418
> #3 0x0000000100207bbf in main_loop () at vl.c:1765
> #4 0x000000010020e7b0 in qemu_main (argc=12, argv=0x7fff5fbff360, envp=0x7fff5fbff3c8) at vl.c:3992
> #5 0x00000001001d6013 in main (argc=12, argv=0x7fff5fbff360) at ui/cocoa.m:884
> (gdb) frame 1
> #1 0x000000010016e209 in qemu_iohandler_poll (readfds=0x10097ca00, writefds=0x10097ca80, xfds=0x10097cb00, ret=4) at iohandler.c:124
> 124 ioh->fd_read(ioh->opaque);
> Current language: auto; currently c
> (gdb) p ioh
> $1 = (IOHandlerRecord *) 0x10142f110
> (gdb) p *ioh
> $2 = {
> fd_read_poll = 0,
> fd_read = 0x10017212b <sigfd_handler>,

The fd_read() function pointer should be called here. But somehow we
end up with 0x000000010142f2d0, which is awefully close to the
IOHandlerRecord (0x10142f110).

Perhaps printing out the entire io_handlers list would be interesting
too.

Does this happen at an unspecified point or is it always when the
fd_read sigfd_handler() callback is invoked? (You could put a
breakpoint on sigfd_handler() and continue the first time it is hit to
check this.)

Stefan

[Christopher Mason (a-cmason)]

Using qemu master rev dbd99ae..25bbf61 configured with:

./configure --disable-sdl --disable-kvm --enable-cocoa --enable-debug --extra-cflags=-g --extra-ldflags=-g

(I'm using clang 4.1 now. Should I be using clang or gcc 4.2? Are these the right config args?)

(gdb) b sigfd_handler
Breakpoint 1 at 0x1001c098d: file main-loop.c, line 41.

(gdb) r -nographic -M versatilepb -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.img-2.6.32-5-versatile -hda debian_squeeze_armel_standard.qcow2 -append "root=/dev/sda1 console=ttyAMA0"
...
Breakpoint 1, sigfd_handler (opaque=0x3) at main-loop.c:41
41 int fd = (intptr_t)opaque;
(gdb) bt
#0 sigfd_handler (opaque=0x3) at main-loop.c:41
#1 0x00000001001baaee in qemu_iohandler_poll (readfds=0x100a0938c, writefds=0x100a0940c, xfds=0x100a0948c, ret=3) at iohandler.c:124
#2 0x00000001001c00bb in main_loop_wait (nonblocking=0) at main-loop.c:418
#3 0x000000010027bde4 in main_loop () at vl.c:1765
#4 0x00000001002765c2 in qemu_main (argc=12, argv=0x7fff5fbff340, envp=0x7fff5fbff3a8) at vl.c:4014
#5 0x0000000100239a13 in main (argc=12, argv=0x7fff5fbff340) at ui/cocoa.m:884
Current language: auto; currently minimal
(gdb) p io_handlers
$1 = {
  lh_first = 0x102102ab0
}
(gdb) p * io_handlers.lh_first
$2 = {
  fd_read_poll = 0x1001fad60 <stdio_read_poll>,
  fd_read = 0x1001fae20 <stdio_read>,
  fd_write = 0,
  opaque = 0x1021029c0,
  next = {
    le_next = 0x102100000,
    le_prev = 0x100a09368
  },
  fd = 0,
  deleted = false
}
(gdb) p * io_handlers.lh_first->next.le_prev
$3 = (struct IOHandlerRecord *) 0x102102ab0
(gdb) p * io_handlers.lh_first->next.le_next
$4 = {
  fd_read_poll = 0,
  fd_read = 0x1001c0970 <sigfd_handler>,
  fd_write = 0,
  opaque = 0x3,
  next = {
    le_next = 0x0,
    le_prev = 0x102102ad0
  },
  fd = 3,
  deleted = false
}

(gdb) c

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x0000000102100040
0x0000000102100040 in ?? ()
(gdb) bt
#0 0x0000000102100040 in ?? ()
#1 0x00000001001baaee in qemu_iohandler_poll (readfds=0x100a0938c, writefds=0x100a0940c, xfds=0x100a0948c, ret=3) at iohandler.c:124
#2 0x00000001001c00bb in main_loop_wait (nonblocking=0) at main-loop.c:418
#3 0x000000010027bde4 in main_loop () at vl.c:1765
#4 0x00000001002765c2 in qemu_main (argc=12, argv=0x7fff5fbff340, envp=0x7fff5fbff3a8) at vl.c:4014
#5 0x0000000100239a13 in main (argc=12, argv=0x7fff5fbff340) at ui/cocoa.m:884

(gdb) p io_handlers
$5 = {
  lh_first = 0x102102ab0
}
(gdb) p * io_handlers.lh_first
$6 = {
  fd_read_poll = 0x1001fad60 <stdio_read_poll>,
  fd_read = 0x1001fae20 <stdio_read>,
  fd_write = 0,
  opaque = 0x1021029c0,
  next = {
    le_next = 0x102100000,
    le_prev = 0x100a09368
  },
  fd = 0,
  deleted = false
}
(gdb) p * io_handlers.lh_first->next.le_next
$8 = {
  fd_read_poll = 0,
  fd_read = 0x1001c0970 <sigfd_handler>,
  fd_write = 0,
  opaque = 0x3,
  next = {
    le_next = 0x0,
    le_prev = 0x102102ad0
  },
  fd = 3,
  deleted = false
}
(gdb) p * io_handlers.lh_first->next.le_prev
$9 = (struct IOHandlerRecord *) 0x102102ab0

[Stefan Hajnoczi (stefanha)]

On Fri, Jan 04, 2013 at 06:09:30PM -0000, Christopher Mason wrote:
> Using qemu master rev dbd99ae..25bbf61 configured with:
>
> ./configure --disable-sdl --disable-kvm --enable-cocoa --enable-debug
> --extra-cflags=-g --extra-ldflags=-g
>
> (I'm using clang 4.1 now. Should I be using clang or gcc 4.2? Are these
> the right config args?)

I have never used QEMU on Mac myself, sorry. Maybe someone else can
help.

> (gdb) b sigfd_handler
> Breakpoint 1 at 0x1001c098d: file main-loop.c, line 41.
>
> (gdb) r -nographic -M versatilepb -kernel vmlinuz-2.6.32-5-versatile -initrd initrd.img-2.6.32-5-versatile -hda debian_squeeze_armel_standard.qcow2 -append "root=/dev/sda1 console=ttyAMA0"
> ...
> Breakpoint 1, sigfd_handler (opaque=0x3) at main-loop.c:41
> 41 int fd = (intptr_t)opaque;
> (gdb) bt
> #0 sigfd_handler (opaque=0x3) at main-loop.c:41
> #1 0x00000001001baaee in qemu_iohandler_poll (readfds=0x100a0938c, writefds=0x100a0940c, xfds=0x100a0948c, ret=3) at iohandler.c:124
> #2 0x00000001001c00bb in main_loop_wait (nonblocking=0) at main-loop.c:418
> #3 0x000000010027bde4 in main_loop () at vl.c:1765
> #4 0x00000001002765c2 in qemu_main (argc=12, argv=0x7fff5fbff340, envp=0x7fff5fbff3a8) at vl.c:4014
> #5 0x0000000100239a13 in main (argc=12, argv=0x7fff5fbff340) at ui/cocoa.m:884
> Current language: auto; currently minimal
> (gdb) p io_handlers
> $1 = {
> lh_first = 0x102102ab0
> }
> (gdb) p * io_handlers.lh_first
> $2 = {
> fd_read_poll = 0x1001fad60 <stdio_read_poll>,
> fd_read = 0x1001fae20 <stdio_read>,
> fd_write = 0,
> opaque = 0x1021029c0,
> next = {
> le_next = 0x102100000,
> le_prev = 0x100a09368
> },
> fd = 0,
> deleted = false
> }
> (gdb) p * io_handlers.lh_first->next.le_prev
> $3 = (struct IOHandlerRecord *) 0x102102ab0
> (gdb) p * io_handlers.lh_first->next.le_next
> $4 = {
> fd_read_poll = 0,
> fd_read = 0x1001c0970 <sigfd_handler>,
> fd_write = 0,
> opaque = 0x3,
> next = {
> le_next = 0x0,
> le_prev = 0x102102ad0
> },
> fd = 3,
> deleted = false
> }
>
> (gdb) c
>
> Program received signal EXC_BAD_ACCESS, Could not access memory.
> Reason: KERN_PROTECTION_FAILURE at address: 0x0000000102100040
> 0x0000000102100040 in ?? ()
> (gdb) bt
> #0 0x0000000102100040 in ?? ()
> #1 0x00000001001baaee in qemu_iohandler_poll (readfds=0x100a0938c, writefds=0x100a0940c, xfds=0x100a0948c, ret=3) at iohandler.c:124
> #2 0x00000001001c00bb in main_loop_wait (nonblocking=0) at main-loop.c:418
> #3 0x000000010027bde4 in main_loop () at vl.c:1765
> #4 0x00000001002765c2 in qemu_main (argc=12, argv=0x7fff5fbff340, envp=0x7fff5fbff3a8) at vl.c:4014
> #5 0x0000000100239a13 in main (argc=12, argv=0x7fff5fbff340) at ui/cocoa.m:884
>
> (gdb) p io_handlers
> $5 = {
> lh_first = 0x102102ab0
> }
> (gdb) p * io_handlers.lh_first
> $6 = {
> fd_read_poll = 0x1001fad60 <stdio_read_poll>,
> fd_read = 0x1001fae20 <stdio_read>,
> fd_write = 0,
> opaque = 0x1021029c0,
> next = {
> le_next = 0x102100000,
> le_prev = 0x100a09368
> },
> fd = 0,
> deleted = false
> }
> (gdb) p * io_handlers.lh_first->next.le_next
> $8 = {
> fd_read_poll = 0,
> ...

[Peter Maydell (pmaydell)]

Just a note that IME trying to debug QEMU under gdb on MacOS doesn't work very well. In particular as far as I can tell gdb breaks sigwait() such that the sigwait() in sigwait_compat() can return 0 without setting the int* sig. This causes QEMU to write an uninitialized value into the qemu_signalfd_siginfo struct it sends down the pipe, and then sigfd_handler() calls sigaction() with this bogus data as the signal number. Since sigfd_handler() doesn't check the return value from sigaction() we then proceed to leap off into nowhere.

sigfd_handler() should probably be checking the return value from sigaction() but the underlying problem is MacOS and/or its gdb breaking sigwait() behaviour somehow.

[Thomas Huth (th-huth)]

Can you still reproduce this problem wit the latest release of QEMU (currently version 2.9.0) and macOS, or could we close this bug nowadays?

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]