qemu segfaults when creating an armhf container on an amd64 host

[Jean-Baptiste Lallement]

On Quantal

TEST CASE:
1. From a terminal run:
  $ sudo lxc-create -n quantal-armhf -t ubuntu -- -r quantal -a armhf

ACTUAL RESULT:
Output:
"""
No config file specified, using the default config
debootstrap is /usr/sbin/debootstrap
[...]
Hit http://ports.ubuntu.com quantal-security/universe Translation-en
Fetched 17.6 MB in 45s (383 kB/s)
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
/usr/share/lxc/templates/lxc-ubuntu: line 447: 14607 Segmentation fault (core dumped) chroot $rootfs apt-get update
failed to execute template 'ubuntu'
aborted

"""

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: qemu (not installed)
ProcVersionSignature: Ubuntu 3.5.0-1.1-generic 3.5.0-rc3
Uname: Linux 3.5.0-1-generic x86_64
ApportVersion: 2.2.5-0ubuntu1
Architecture: amd64
Date: Tue Jun 26 09:50:48 2012
KvmCmdLine: Error: command ['ps', '-C', 'kvm', '-F'] failed with exit code 1: UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
MachineType: Gigabyte Technology Co., Ltd. GA-890GPA-UD3H
ProcEnviron:
 TERM=xterm
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.5.0-1-generic root=UUID=cf89ba34-108b-404d-9804-32d54a1df2ea ro quiet splash
SourcePackage: qemu-kvm
UpgradeStatus: Upgraded to quantal on 2012-01-31 (146 days ago)
dmi.bios.date: 07/23/2010
dmi.bios.vendor: Award Software International, Inc.
dmi.bios.version: FD
dmi.board.name: GA-890GPA-UD3H
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.type: 3
dmi.chassis.vendor: Gigabyte Technology Co., Ltd.
dmi.modalias: dmi:bvnAwardSoftwareInternational,Inc.:bvrFD:bd07/23/2010:svnGigabyteTechnologyCo.,Ltd.:pnGA-890GPA-UD3H:pvr:rvnGigabyteTechnologyCo.,Ltd.:rnGA-890GPA-UD3H:rvrx.x:cvnGigabyteTechnologyCo.,Ltd.:ct3:cvr:
dmi.product.name: GA-890GPA-UD3H
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[Serge Hallyn (serge-hallyn)]

Thanks for submitting this bug. The qemu used by lxc is actually qemu-arm-static which comes from the qemu-linaro package, so I will re-target the bug. I'll also mark it as affecting lxc.

[Stéphane Graber (stgraber)]

This looks like a regression in qemu from Precise to Quantal. I just rebuilt my test containers here on Precise and both precise and quantal armhf containers built successfully.

[Serge Hallyn (serge-hallyn)]

Hm, I can't reproduce this exactly, rather I got:

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 iproute:amd64 : Depends: libc6:amd64 (>= 2.14) but it is not going to be installed
                 Depends: libdb5.1:amd64 but it is not going to be installed
 isc-dhcp-client:amd64 : Depends: debianutils:amd64 (>= 2.8.2) but it is not going to be installed
                         Depends: isc-dhcp-common:amd64 (= 4.1.ESV-R4-0ubuntu8) but it is not going to be installed
                         Depends: libc6:amd64 (>= 2.15) but it is not going to be installed
 mountall:amd64 : Depends: makedev:amd64 but it is not installable
                  Depends: udev:amd64 but it is not going to be installed
                  Depends: plymouth:amd64 but it is not going to be installed
                  Depends: coreutils:amd64 (>= 7.1) but it is not going to be installed
                  Depends: libc6:amd64 (>= 2.9) but it is not going to be installed
                  Depends: libdbus-1-3:amd64 (>= 1.2.16) but it is not going to be installed
                  Depends: libnih-dbus1:amd64 (>= 1.0.0) but it is not going to be installed
                  Depends: libnih1:amd64 (>= 1.0.0) but it is not going to be installed
                  Depends: libplymouth2:amd64 (>= 0.8.1-3) but it is not going to be installed
                  Depends: libudev0:amd64 (>= 151) but it is not going to be installed
                  PreDepends: dpkg:amd64 (>= 1.15.7.2) but it is not going to be installed
 upstart:amd64 : Depends: libc6:amd64 (>= 2.9) but it is not going to be installed
                 Depends: libdbus-1-3:amd64 (>= 1.2.16) but it is not going to be installed
                 Depends: libnih-dbus1:amd64 (>= 1.0.0) but it is not going to be installed
                 Depends: libnih1:amd64 (>= 1.0.0) but it is not going to be installed
                 Depends: libudev0:amd64 (>= 151) but it is not going to be installed
                 Depends: sysvinit-utils:amd64 but it is not going to be installed
                 Depends: sysv-rc:amd64 but it is not installable
                 Depends: initscripts:amd64 but it is not going to be installed
                 Depends: ifupdown:amd64 (>= 0.6.10ubuntu5) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
failed to execute template 'ubuntu'
aborted

[Serge Hallyn (serge-hallyn)]

(actually it just occurred to me that machine was not quite pristine, re-trying elsewhere)

[Jean-Baptiste Lallement (jibel)]

This is bug 1017862

[Stéphane Graber (stgraber)]

Right, with quantal-proposed enable you shouldn't be getting that bug anymore (1017862), I'll SRU it next week.

[Serge Hallyn (serge-hallyn)]

Could you re-try this on a clean machine? I can create a precise container on quantal with no problems. I'm having trouble with a quantal container, but it seems to be a problem in lxc (which should be fixed) or transient archive problem. I see no qemu crashes here.

[Serge Hallyn (serge-hallyn)]

Makes sense, thanks. Will re-try one more time with -proposed

[Serge Hallyn (serge-hallyn)]

couldn't reproduce.

[Jean-Baptiste Lallement (jibel)]

I get it systematically on my system with latest updates applied. Do you have any advice how to troubleshoot it ?

[Serge Hallyn (serge-hallyn)]

Quoting Jean-Baptiste Lallement (<email address hidden>):
> I get it systematically on my system with latest updates applied. Do you

By systematically do you mean every time, or every N times?

> have any advice how to troubleshoot it ?

A few things may help,

1. install the qemu-user-static-dbg package (may make the coredump more meaningful)
2. set your core limit (ulimit -c 9999999)
3. re-run
4. post the resulting coredump here
5. look in syslog for the segfault information (two lines) and post them here

I should be able to use that info to at least find where in the qemu source the
fault is happening.

[Serge Hallyn (serge-hallyn)]

ping - is this still happening for you? Can we mark it fix released?

[Jean-Baptiste Lallement (jibel)]

I cannot tell for the moment, it is not possible to install an armhf container on an amd64 host due to gcc 4.7.1-7ubuntu1 failing to build on armhf.

[Jean-Baptiste Lallement (jibel)]

The problem persists on Quantal with all updates applied:

$ sudo lxc-create -n quantal-armhf -t ubuntu -- -r quantal -a armhf
[...]
Fetched 22.1 MB in 60s (365 kB/s)
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
/usr/share/lxc/templates/lxc-ubuntu: line 460: 29480 Segmentation fault (core dumped) chroot $rootfs apt-get update
failed to execute template 'ubuntu'
aborted

[Serge Hallyn (serge-hallyn)]

I can't currently reproduce this as it seems the armhf archives are out of sync

[Serge Hallyn (serge-hallyn)]

Ok - I'm able to reproduce this when qemu-user-static is not installed. When qemu-user-static is installed, container creation succeeds.

Looking at the lxc-ubuntu template, I'm not sure what we want to do in the failing case. I would guess we want to announce that qemu-armhf-static is not installed before trying to run mount from that architecture?

I'm assigning to stgraber to make sure I understand right.

[Serge Hallyn (serge-hallyn)]

@jibel,

to be clear, you should NOT get this if you have qemu-user-static installed. If you do, please shout to have the priority raised.

The question for now is what we should do about this. Should we detect foreign arch and insist qemu-user-static be installed?

In the meantime, since there is a workaround, I am setting the priority to low.

[Serge Hallyn (serge-hallyn)]

(my guess fwiw is that calling qemu-arm in a chroot fails because the libraries it links against are not available; hence the need for qemu-arm-static)

[Jean-Baptiste Lallement (jibel)]

qemu-user-static was installed on the system when I have been able to reproduce this error (installed on 2012-06-06 from apt log files)

I tried again this morning, and it works correctly. The main difference is that qemu-user-static has been upgraded from 1.0.50-2012.03-0ubuntu3 to 1.2.0-2012.09-0ubuntu1 on 2012-09-23 (~1 week after my last test) So probably *something* has been fixed in this new upstream version.

In any case, I tried without qemu-user-static installed and I get the error
    chroot: failed to run command `useradd': Exec format error
which is expected since the emulator is not installed.

I don't think it make sense to run lxc for an architecture without the right emulator installed but it is another bug than the one I initially reported and it can be closed as 'fix released'

[Serge Hallyn (serge-hallyn)]

I've been creating several armhf containers on raring with no problems. Can we call this fix released in raring?