tcg.c:1693: tcg fatal error

[Gianfranco Costamagna]

this started happening after the launchpad buildd trusty deploy
https://code.launchpad.net/~costamagnagianfranco/+archive/ubuntu/firefox/+build/6224439

debconf-updatepo
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)
/build/buildd/qemu-2.0.0+dfsg/tcg/tcg.c:1693: tcg fatal error
/build/buildd/qemu-2.0.0+dfsg/tcg/tcg.c:1693: tcg fatal error

this seems to be the patch needed
https://patches.linaro.org/32473/

[Serge Hallyn (serge-hallyn)]

Hi,

have already tested with that patch to verify that it fixes the
issue? If I put qemu + that patch into a ppa, will your infrastructure
allow you to test that way?

I'm a bit concerned about this patch, as it appears to be one which has
been in the Suse tree for quite some time, begging the question why it is
not upstream. It would be nice to hear from agraf or pmaydell whether
this patch is safe.

[Peter Maydell (pmaydell)]

That patch is not in mainline because it's an appalling hack. If we care about multi-threaded guests we need to fix them properly, not paper over the issues by constraining multiple threads to one CPU in the hopes the race conditions don't bite us so often.

[Gianfranco Costamagna (costamagnagianfranco)]

No, I did not test the patch, I don't think that I can push that patch in lp infrastructure, and I don't have my own one.

Anyway Peter is right, we don't need to hide bugs, but to fix them ;)

[Colin Watson (cjwatson)]

Peter: While this is true, is it actually likely to happen? I thought in our conversations in the past (when I previously attempted to escalate this class of problems via Linaro) it had been fairly clear that this was a very difficult task that isn't likely to be scheduled for the foreseeable future.

[Peter Maydell (pmaydell)]

I think it's likely to happen eventually; it depends rather on the balance between this and other work priorities (at least if it's going to be Linaro doing the work). Regardless, I'm not taking hacky workarounds like this into mainline (hacks are hard to get out once you let them in, and they remove any motivation anybody might have had for fixing things properly).

[Colin Watson (cjwatson)]

Right, I can absolutely understand that. The question would I suppose be whether you think this is a completely unreasonable thing to put in a distro patch; I think SUSE are doing so for basically the same reason we would be, that is, a setup such as PPAs where virtualisation isn't available directly on the architecture in question but we can use qemu-user to run foreign-arch binaries reasonably transparently. My understanding is that this patch would make a pretty huge dent in our failure rate for that setup.

[Peter Maydell (pmaydell)]

Well, it won't make anything any worse, so it's your call based on how much it actually improves your failure rate I guess.

[Serge Hallyn (serge-hallyn)]

https://launchpad.net/~serge-hallyn/+archive/ubuntu/qemu-user-thread contains a package with this patch applied (built for trusty). Please let us know how much it helps.

[Gianfranco Costamagna (costamagnagianfranco)]

I got this morning a new FTBFS in a package that have been always built successfully in the past, just FYI
https://launchpadlibrarian.net/181879742/buildlog_ubuntu-trusty-armhf.boinc_7.4.0~nightly1~~git20140809%2Br21874~r184~ubuntu14.04.1_FAILEDTOBUILD.txt.gz

[Gianfranco Costamagna (costamagnagianfranco)]

Interesting enough the utopic build was successful!

[Gianfranco Costamagna (costamagnagianfranco)]

And when I retried the failed build it succeeded but a segmentation fault (core dumped) is in the logs
https://code.launchpad.net/~costamagnagianfranco/+archive/ubuntu/firefox/+build/6254532
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)
Segmentation fault (core dumped)

[Serge Hallyn (serge-hallyn)]

Are these new failures using the qem upackage in ppa:serge-hallyn/qemu-user-thread, or using the stock trusty qemu package?

[Gianfranco Costamagna (costamagnagianfranco)]

How can I use your one? I have no possibility to tweak the host system, I'm not an lp admin, do you think I can try to add your ppa as dependency and it will work?

[Serge Hallyn (serge-hallyn)]

No, that won't work as it's the emulator in which the build is running.

You should however be able to install the ppa's qemu on a local host,
install a standard ubuntu server vm, apt-get install ubuntu-dev-tools,
copy the package sources over to the vm, and build the packages inside
the vm.

[Gianfranco Costamagna (costamagnagianfranco)]

Hi Serge, sorry for the delay,
the problem is that if I run pbuilder-dist utopic armhf create
pbuilder-dist utopic armhf build boinc_whatever.dsc the build never fails also with the trusty qemu, so I cannot test your package with the patch.
(I'm already on trusty, should I really install a server machine?)

[Serge Hallyn (serge-hallyn)]

Perhaps i don't understand what you are doing.

My understanding was that a package is being built (in the buildds) under qemu. Qemu is failing due to tcg failure. We want to tes twhether a qemu patch fixes it.

That's why I suggest installing the new qemu package on your host, using it to run a new server VM, and building the problematic package inside that VM. That way we can see whether the proposed qemu package fixes the build error.

[Gianfranco Costamagna (costamagnagianfranco)]

I didn't install a VM because:
-I'm on ubuntu 14.04 on my laptop
-I use pbuilder-dist (from ubuntu-dev-tools) armhf that uses qemu as underlying virtualization system.

the fact is:
why on my machine it doesn't show this error?

possible solution:
-because qemu+pbuilder-dist is not multithreaded?

[Serge Hallyn (serge-hallyn)]

This patch is still not applied upstream.

As there has been no discussion in over a year, I assume it is no longer a problem and I'm going to mark it invalid. Please rep;ly if that is not the case.

If it *is* still a problem, then we should go discuss on oftc#qemu.

[Michael Tokarev (mjt+launchpad-tls)]

Well. This is definitely wrong. It is a valid bug, but it needs quite serious work to fix, which requires major refactoring of the tcg code. Upstream is working on it, see http://wiki.qemu.org/Features/tcg-multithread

[Serge Hallyn (serge-hallyn)]

Ah, thanks for setting me straight.

[Colin Watson (cjwatson)]

We're now building armhf/arm64 packages on real arm64 hardware in Launchpad, so, while this problem may well still exist in qemu, it no longer applies to Launchpad builds.

[Peter Maydell (pmaydell)]

We think we've fixed our linux-user threading issues, so if there are still problems as of qemu-2.11 or later, please raise a fresh bug report with repro instructions.

[Christian Ehrhardt  (paelzer)]

per former comments, in context qemu 2.11 is in proposed

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu1

---------------
qemu (1:2.11+dfsg-1ubuntu1) bionic; urgency=medium

  * Merge with Debian testing, among other fixes this includes
    - fix fatal error on negative maxcpus (LP: #1722495)
    - fix segfault on dump-guest-memory on guests without memory (LP: #1723381)
    - linux user threading issues (LP: #1350435)
    - TOD-Clock Epoch Extension Support on s390x (LP: #1732691)
    Remaining changes:
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-kvm.service: systemd unit to call qemu-kvm-init
      - d/qemu-system-common.install: install systemd unit and helper script
      - d/qemu-system-common.maintscript: clean old sysv and upstart scripts
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: install /etc/default/qemu-kvm
    - Enable nesting by default
      - set nested=1 module option on intel. (is default on amd)
      - re-load kvm_intel.ko if it was loaded without nested=1
      - d/p/ubuntu/expose-vmx_qemu64cpu.patch: expose nested kvm by default
        in qemu64 cpu type.
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
    - libvirt/qemu user/group support
      - qemu-system-common.postinst: remove acl placed by udev, and add udevadm
        trigger.
      - qemu-system-common.preinst: add kvm group if needed
    - Distribution specific machine type
      - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
        types to ease future live vm migration.
      - d/qemu-system-x86.NEWS Info on fixed machine type defintions
    - improved dependencies
      - Make qemu-system-common depend on qemu-block-extra
      - Make qemu-utils depend on qemu-block-extra
      - let qemu-utils recommend sharutils
    - s390x support
      - Create qemu-system-s390x package
      - Include s390-ccw.img firmware
      - Enable numa support for s390x
    - ppc64[le] support
      - d/qemu-system-ppc.links provide usr/bin/qemu-system-ppc64le symlink
    - arch aware kvm wrappers
  * Added Changes
    - update VCS-git to match the bionic branch
    - sdl2 is yet too unstable for the LTS Ubuntu release given the reports
      we still see upstream and in Debian - furthermore sdl2 isn't in main yet,
      so we revert related changes to stick with the proven for now:
      - 0fd25810 - do not build-depend on libx11-dev (libsdl2-dev already
                   depends on it)
      - 9594f820 - switch from sdl1.2 to sdl2 (#870025)
    - d/qemu-system-x86.README.Debian: document intention of nested being
      default is comfort, not full support
    - update Ubuntu machine types for qemu 2.11
    - qemu-guest-agent: freeze-hook fixes (LP: #1484990)
      - d/p/guest-agent-freeze-hook-skip-dpkg-artifacts.patch
      - d/qemu-guest-agent.install: provide /etc/qemu/fsfreeze-hook
      - d/qemu-guest-agent.dirs: provide /etc/qemu/fsfreeze-hook.d
    - Create and install pxe netboot images for KVM s390x (LP: #1732094)
      - d/rules enable install s390x-netbo...

[Gianfranco Costamagna (costamagnagianfranco)]

Hello, this seems to be still an issue
W: Failure while unpacking required packages. This will be attempted up to five times.
W: See //debootstrap/debootstrap.log for details (possibly the package /var/cache/apt/archives/bash_4.4.18-1ubuntu1_arm64.deb is at fault)

dpkg -l |grep qemu
ii ipxe-qemu 1.0.0+git-20161027.b991c67+really20150424.a25a16d-1ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu 1:2.11+dfsg-1ubuntu1 amd64 fast processor emulator
ii qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu1 amd64 extra block backend modules for qemu-system and qemu-utils
rc qemu-kvm 1:2.11+dfsg-1ubuntu1 amd64 QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu0.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system 1:2.11+dfsg-1ubuntu1 amd64 QEMU full system emulation binaries
ii qemu-system-arm 1:2.11+dfsg-1ubuntu1 amd64 QEMU full system emulation binaries (arm)
ii qemu-system-common 1:2.11+dfsg-1ubuntu1 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-mips 1:2.11+dfsg-1ubuntu1 amd64 QEMU full system emulation binaries (mips)
ii qemu-system-misc 1:2.11+dfsg-1ubuntu1 amd64 QEMU full system emulation binaries (miscellaneous)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu1 amd64 QEMU full system emulation binaries (ppc)
ii qemu-system-s390x 1:2.11+dfsg-1ubuntu1 amd64 QEMU full system emulation binaries (s390x)
ii qemu-system-sparc 1:2.11+dfsg-1ubuntu1 amd64 QEMU full system emulation binaries (sparc)
ii qemu-system-x86 1:2.11+dfsg-1ubuntu1 amd64 QEMU full system emulation binaries (x86)
ii qemu-user 1:2.11+dfsg-1ubuntu1 amd64 ...

[Gianfranco Costamagna (costamagnagianfranco)]

(wrong bug, sorry!)