segfault in qemu-system-x86_64

[Brian Candler]

[Ubuntu 14.04 amd64 server, fully patched, xenial HWE kernel, on a 16GB Mac Mini]

I am using packer (www.packer.io) to create a VM image. Packer starts a qemu-system-x86_64 process; inside it's running an ubuntu 16.04 image doing a bunch of work including running ansible to create a bunch of lxd containers all running mysql. And then the qemu process itself segfaults :-(

I have caught a coredump but it doesn't seem all that useful:

$ gdb -c /tmp/core_qemu-system-x86.24041 /usr/bin/qemu-system-x86_64
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/qemu-system-x86_64...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 24041]
[New LWP 26214]
[New LWP 24045]
[New LWP 26215]
[New LWP 24043]
[New LWP 26321]
[New LWP 26326]
[New LWP 26017]
[New LWP 26325]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/qemu-system-x86_64 -netdev user,id=user.0,hostfwd=tcp::3234-:22 -devic'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00005648c536ad20 in ?? ()
(gdb) bt
#0 0x00005648c536ad20 in ?? ()
#1 0x00005648c536b96a in ?? ()
#2 0x00005648c536cc92 in ?? ()
#3 0x00005648c5367828 in ?? ()
#4 0x00005648c5317e77 in ?? ()
#5 0x00005648c51bfbd6 in ?? ()
#6 0x00007f4b0e1a9f45 in __libc_start_main (main=0x5648c51be640, argc=17,
    argv=0x7ffc2c0cd578, init=<optimised out>, fini=<optimised out>,
    rtld_fini=<optimised out>, stack_end=0x7ffc2c0cd568) at libc-start.c:287
#7 0x00005648c51c412c in ?? ()
(gdb) info threads
  Id Target Id Frame
  9 Thread 0x7f47777fe700 (LWP 26325) sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
  8 Thread 0x7f47597fa700 (LWP 26017) sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
  7 Thread 0x7f4b04acd700 (LWP 26326) sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
  6 Thread 0x7f4776ffd700 (LWP 26321) sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
  5 Thread 0x7f4affe1d700 (LWP 24043) 0x00007f4b0e2791e7 in ioctl ()
    at ../sysdeps/unix/syscall-template.S:81
  4 Thread 0x7f475bfff700 (LWP 26215) sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
  3 Thread 0x7f4afe5ff700 (LWP 24045) pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
  2 Thread 0x7f4759ffb700 (LWP 26214) sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
* 1 Thread 0x7f4b13f24980 (LWP 24041) 0x00005648c536ad20 in ?? ()
(gdb) thread apply all bt

Thread 9 (Thread 0x7f47777fe700 (LWP 26325)):
#0 sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
#1 0x00005648c54ad007 in ?? ()
#2 0x00005648c536effc in ?? ()
#3 0x00007f4b0e555184 in start_thread (arg=0x7f47777fe700)
    at pthread_create.c:312
#4 0x00007f4b0e28237d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 8 (Thread 0x7f47597fa700 (LWP 26017)):
#0 sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
#1 0x00005648c54ad007 in ?? ()
#2 0x00005648c536effc in ?? ()
#3 0x00007f4b0e555184 in start_thread (arg=0x7f47597fa700)
    at pthread_create.c:312
#4 0x00007f4b0e28237d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 7 (Thread 0x7f4b04acd700 (LWP 26326)):
#0 sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
#1 0x00005648c54ad007 in ?? ()
#2 0x00005648c536effc in ?? ()
#3 0x00007f4b0e555184 in start_thread (arg=0x7f4b04acd700)
    at pthread_create.c:312
#4 0x00007f4b0e28237d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 6 (Thread 0x7f4776ffd700 (LWP 26321)):
#0 sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
#1 0x00005648c54ad007 in ?? ()
#2 0x00005648c536effc in ?? ()
#3 0x00007f4b0e555184 in start_thread (arg=0x7f4776ffd700)
    at pthread_create.c:312
#4 0x00007f4b0e28237d in clone ()
---Type <return> to continue, or q <return> to quit---
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 5 (Thread 0x7f4affe1d700 (LWP 24043)):
#0 0x00007f4b0e2791e7 in ioctl () at ../sysdeps/unix/syscall-template.S:81
#1 0x00005648c53fe584 in ?? ()
#2 0x00005648c53fe664 in ?? ()
#3 0x00005648c539e612 in ?? ()
#4 0x00007f4b0e555184 in start_thread (arg=0x7f4affe1d700)
    at pthread_create.c:312
#5 0x00007f4b0e28237d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 4 (Thread 0x7f475bfff700 (LWP 26215)):
#0 sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
#1 0x00005648c54ad007 in ?? ()
#2 0x00005648c536effc in ?? ()
#3 0x00007f4b0e555184 in start_thread (arg=0x7f475bfff700)
    at pthread_create.c:312
#4 0x00007f4b0e28237d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 3 (Thread 0x7f4afe5ff700 (LWP 24045)):
#0 pthread_cond_wait@@GLIBC_2.3.2 ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00005648c54ace39 in ?? ()
#2 0x00005648c538c2c3 in ?? ()
#3 0x00005648c538c6c0 in ?? ()
#4 0x00007f4b0e555184 in start_thread (arg=0x7f4afe5ff700)
    at pthread_create.c:312
#5 0x00007f4b0e28237d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 2 (Thread 0x7f4759ffb700 (LWP 26214)):
#0 sem_timedwait ()
    at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_timedwait.S:101
#1 0x00005648c54ad007 in ?? ()
#2 0x00005648c536effc in ?? ()
#3 0x00007f4b0e555184 in start_thread (arg=0x7f4759ffb700)
---Type <return> to continue, or q <return> to quit---
    at pthread_create.c:312
#4 0x00007f4b0e28237d in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Thread 1 (Thread 0x7f4b13f24980 (LWP 24041)):
#0 0x00005648c536ad20 in ?? ()
#1 0x00005648c536b96a in ?? ()
#2 0x00005648c536cc92 in ?? ()
#3 0x00005648c5367828 in ?? ()
#4 0x00005648c5317e77 in ?? ()
#5 0x00005648c51bfbd6 in ?? ()
#6 0x00007f4b0e1a9f45 in __libc_start_main (main=0x5648c51be640, argc=17,
    argv=0x7ffc2c0cd578, init=<optimised out>, fini=<optimised out>,
    rtld_fini=<optimised out>, stack_end=0x7ffc2c0cd568) at libc-start.c:287
#7 0x00005648c51c412c in ?? ()
(gdb)

I am afraid my gdb foo ends there.

Note: I *do* have the libc6-dbg package installed, so I don't know why the libc symbols aren't resolved.

The full qemu command line would be something like this (this is from a subsequent run):

/usr/bin/qemu-system-x86_64 -m 14G -drive file=output-qemu-nmm/vtp-nmm.qcow2,if=virtio,cache=writeback,discard=unmap -boot c -vnc 0.0.0.0:83 -name vtp-nmm.qcow2 -machine type=pc,accel=kvm -netdev user,id=user.0,hostfwd=tcp::2628-:22 -device virtio-net,netdev=user.0

Given the relatively old version of qemu which is included in trusty, I may just have to update this machine to xenial. There doesn't seem to be any newer qemu in trusty-backports.

=== Additional system info ===

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"

Linux brian 4.4.0-38-generic #57~14.04.1-Ubuntu SMP Tue Sep 6 17:20:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

ii ipxe-qemu 1.0.0+git-20131111.c3d1e78-2ubuntu1.1 all PXE boot firmware - ROM images for qemu
ii qemu-keymaps 2.0.0+dfsg-2ubuntu1.27 all QEMU keyboard maps
ii qemu-kvm 2.0.0+dfsg-2ubuntu1.27 amd64 QEMU Full virtualization
ii qemu-system-common 2.0.0+dfsg-2ubuntu1.27 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-x86 2.0.0+dfsg-2ubuntu1.27 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 2.0.0+dfsg-2ubuntu1.27 amd64 QEMU utilities

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: qemu-system-x86 2.0.0+dfsg-2ubuntu1.27
ProcVersionSignature: Ubuntu 4.4.0-38.57~14.04.1-generic 4.4.19
Uname: Linux 4.4.0-38-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.21
Architecture: amd64
Date: Tue Oct 4 11:59:00 2016
InstallationDate: Installed on 2014-07-16 (810 days ago)
InstallationMedia: Ubuntu-Server 14.04 LTS "Trusty Tahr" - Release amd64 (20140416.2)
SourcePackage: qemu
UpgradeStatus: No upgrade log present (probably fresh install)

[Brian Candler (b-candler)]

Hmm, a different malloc-type error on next run:

Program received signal SIGABRT, Aborted.
0x00007f7b20acbc37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007f7b20acbc37 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f7b20acf028 in __GI_abort () at abort.c:89
#2 0x00007f7b20b082a4 in __libc_message (do_abort=do_abort@entry=1,
    fmt=fmt@entry=0x7f7b20c166b0 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007f7b20b1455e in malloc_printerr (ptr=<optimised out>,
    str=0x7f7b20c12801 "free(): invalid pointer", action=1) at malloc.c:4996
#4 _int_free (av=<optimised out>, p=<optimised out>, have_lock=0)
    at malloc.c:3840
#5 0x0000563c539742ea in coroutine_trampoline (i0=<optimised out>,
    i1=<optimised out>) at util/coroutine-ucontext.c:78
#6 0x00007f7b20ade800 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x00007fffc12967b0 in ?? ()
#8 0x0000000000000000 in ?? ()
(gdb)

I am going to see if I can run this build on some different hardware.