Bug #1928075 “Segfault with qemu-aarch64-static doing debootstr...” : Bugs : qemu package : Ubuntu

TJ (tj) on 2021-05-11

description:

updated

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-05-11:

#1

I've seen similar issues due to e.g. a newer glibc (could be any other program or even guest kernel) to use newer instructions and thereby trigger an issue that exists in the emulation.

This usually ends in the need to bisect the host to identify a (hopefully already existing) fix and then consider if it can be backported.

For a start I'd ask if you can reproduce the same on Hirsute with the newer qemu/kernel (those to are the most important components in this - an that is also the reasons why container based tests are not sufficient).

If the new code works then we need to find what fixed it, if the new code does not work we need to get you onto a super-recent kernel/qemu and if that also fails convert this to an upstream bug.

Could you do this initial "check newer versions" for this case?

Revision history for this message

TJ (tj) wrote on 2021-05-11:

#2

Will do - I run builds of the mainline kernel tracking upstream so that part isn't a problem. We are operating on bare-metal on the host.

I suspect it is glibc and/or compilation options since trying to manually call, for example, /bin/dash or other executables hits the same issue.

Revision history for this message

TJ (tj) wrote on 2021-05-11:

#3

Looking closer we've just noticed the bullseye's ldconfig claims to be dynamically linked but buster's is statically linked.

This reminds of bug #1908331 we solved earlier this year in qemu itself due to the build flag -static-pie

$ file bullseye-arm64/sbin/ldconfig
bullseye-arm64/sbin/ldconfig: ELF 64-bit LSB shared object, ARM aarch64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=6cca288a8f9e39114be7ce6d9761f252990f5214, for GNU/Linux 3.7.0, stripped
$ file buster-arm64/sbin/ldconfig
buster-arm64/sbin/ldconfig: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, for GNU/Linux 3.7.0, BuildID[sha1]=078feba3f88616e37d2a77a1997d9de2343df80b, stripped

Revision history for this message

TJ (tj) wrote on 2021-05-11:

#4

Another data point.

We tried installing qemu-aarch64-static from Debian's qemu-user-static amd64 packages from testing (1:5.2+dfsg-10) (bullseye) and experimental (6.0+dfsg-1~exp0) into the chroot. Both result in the same error.

On my PC I have kernel 5.12.0-elloe+ and Callum has 5.10.0. We both see the same seg-fault.

Revision history for this message

TJ (tj) wrote on 2021-05-11:

#5

First simple test with 21.04 amd64 LXD host with kernel 5.12.0 results in the same issue.

I'll need to repeat in a virtual machine next.

Revision history for this message

TJ (tj) wrote on 2021-05-11:

#6

In a 21.04 amd64 desktop virtual machine it fails but with a different error apparently related to bullseye systemd package, not sure if I can untangle this mix of problems:

$ sudo apt install qemu-user-static debootstrap

$ sudo qemu-debootstrap --arch=arm64 bullseye bullseye-arm64/ http://ftp.debian.org/debian
W: qemu-debootstrap is deprecated. Please use regular debootstrap directly
I: Running command: debootstrap --arch=arm64 bullseye bullseye-arm64/ http://ftp.debian.org/debian
...
I: Configuring libc-bin...
I: Unpacking the base system...
W: Failure trying to run: chroot "/home/ubuntu/chroot/bullseye-arm64" dpkg --force-overwrite --force-confold --skip-same-version --install /var/cache/apt/archives/libapparmor1_2.13.6-10_arm64.deb /var/cache/apt/archives/libargon2-1_0~20171227-0.2_arm64.deb /var/cache/apt/archives/libcryptsetup12_2%3a2.3.5-1_arm64.deb /var/cache/apt/archives/libip4tc2_1.8.7-1_arm64.deb /var/cache/apt/archives/libjson-c5_0.15-2_arm64.deb /var/cache/apt/archives/libkmod2_28-1_arm64.deb /var/cache/apt/archives/libcap2_1%3a2.44-1_arm64.deb /var/cache/apt/archives/dmsetup_2%3a1.02.175-2.1_arm64.deb /var/cache/apt/archives/libdevmapper1.02.1_2%3a1.02.175-2.1_arm64.deb /var/cache/apt/archives/systemd_247.3-5_arm64.deb /var/cache/apt/archives/systemd-timesyncd_247.3-5_arm64.deb
W: See /home/ubuntu/chroot/bullseye-arm64/debootstrap/debootstrap.log for details (possibly the package systemd is at fault)

--- log ---
...
Setting up systemd (247.3-5) ...
Created symlink /<email address hidden> -> /lib/systemd/system/getty@.service.
Created symlink /etc/systemd/system/multi-user.target.wants/remote-fs.target -> /lib/systemd/system/remote-fs.target.
Created symlink /etc/systemd/system/sysinit.target.wants/systemd-pstore.service -> /lib/systemd/system/systemd-pstore.service.
Initializing machine ID from random generator.
Detected unsafe path transition / -> /var during canonicalization of /var/log/journal.
Detected unsafe path transition / -> /var during canonicalization of /var/log/journal.
Detected unsafe path transition / -> /var during canonicalization of /var/log/journal.
dpkg: error processing package systemd (--install):
installed systemd package post-installation script subprocess returned error exit status 73
Processing triggers for libc-bin (2.31-11) ...
Errors were encountered while processing:
systemd

In a 21.04 amd64 desktop virtual machine it fails but with a different error apparently related to bullseye systemd package, not sure if I can untangle this mix of problems:

$ sudo apt install qemu-user-static debootstrap

$ sudo qemu-debootstrap --arch=arm64 bullseye bullseye-arm64/ http://ftp.debian.org/debian
W: qemu-debootstrap is deprecated. Please use regular debootstrap directly
I: Running command: debootstrap --arch=arm64 bullseye bullseye-arm64/ http://ftp.debian.org/debian
...
I: Configuring libc-bin...
I: Unpacking the base system...
W: Failure trying to run: chroot "/home/ubuntu/chroot/bullseye-arm64" dpkg --force-overwrite --force-confold --skip-same-version --install /var/cache/apt/archives/libapparmor1_2.13.6-10_arm64.deb /var/cache/apt/archives/libargon2-1_0~20171227-0.2_arm64.deb /var/cache/apt/archives/libcryptsetup12_2%3a2.3.5-1_arm64.deb /var/cache/apt/archives/libip4tc2_1.8.7-1_arm64.deb /var/cache/apt/archives/libjson-c5_0.15-2_arm64.deb /var/cache/apt/archives/libkmod2_28-1_arm64.deb /var/cache/apt/archives/libcap2_1%3a2.44-1_arm64.deb /var/cache/apt/archives/dmsetup_2%3a1.02.175-2.1_arm64.deb /var/cache/apt/archives/libdevmapper1.02.1_2%3a1.02.175-2.1_arm64.deb /var/cache/apt/archives/systemd_247.3-5_arm64.deb /var/cache/apt/archives/systemd-timesyncd_247.3-5_arm64.deb
W: See /home/ubuntu/chroot/bullseye-arm64/debootstrap/debootstrap.log for details (possibly the package systemd is at fault)

--- log ---
...
Setting up systemd (247.3-5) ...
Created symlink /etc/systemd/system/getty.target.wants/getty@tty1.service -> /lib/systemd/system/getty@.service.
Created symlink /etc/systemd/system/multi-user.target.wants/remote-fs.target -> /lib/systemd/system/remote-fs.target.
Created symlink /etc/systemd/system/sysinit.target.wants/systemd-pstore.service -> /lib/systemd/system/systemd-pstore.service.
Initializing machine ID from random generator.
Detected unsafe path transition / -> /var during canonicalization of /var/log/journal.
Detected unsafe path transition / -> /var during canonicalization of /var/log/journal.
Detected unsafe path transition / -> /var during canonicalization of /var/log/journal.
dpkg: error processing package systemd (--install):
 installed systemd package post-installation script subprocess returned error exit status 73
Processing triggers for libc-bin (2.31-11) ...
Errors were encountered while processing:
 systemd

Revision history for this message

TJ (tj) wrote on 2021-05-11:

#7

Ok, that was an easy solve:

$ sudo chown 0:0 bullseye-arm64

And with that installation runs to completion. This is inside a 21.04 amd64 desktop KVM virtual machine.

Revision history for this message

TJ (tj) wrote on 2021-05-12:

#8

Download full text (4.4 KiB)

I stepped back to the chroot and tried qemu-aarch64-static from 21.04's qemu-user-static (5.2+dfsg-9ubuntu2) but *debootstrap* fails in the same way as previously reported.

HOWEVER, this works:

$ sudo chroot bullseye-arm64 /usr/bin/qemu-aarch64-static /sbin/ldconfig -v
/sbin/ldconfig: Can't stat /usr/local/lib/aarch64-linux-gnu: No such file or directory
/sbin/ldconfig: Path `/usr/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Can't stat /usr/local/lib: No such file or directory
/sbin/ldconfig: Path `/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Path `/usr/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Path `/usr/lib' given more than once
/lib/aarch64-linux-gnu:
        libSegFault.so -> libSegFault.so
        libpam_misc.so.0 -> libpam_misc.so.0.82.1
        libpcprofile.so -> libpcprofile.so
        libext2fs.so.2 -> libext2fs.so.2.4
        libattr.so.1 -> libattr.so.1.1.2448
        libssl.so.1.1 -> libssl.so.1.1
        libnsl.so.2 -> libnsl.so.2.0.1
        libc.so.6 -> libc-2.31.so
        libgcc_s.so.1 -> libgcc_s.so.1
        libudev.so.1 -> libudev.so.1.7.0
        libz.so.1 -> libz.so.1.2.11
        libdb-5.3.so -> libdb-5.3.so
        libtic.so.6 -> libtic.so.6.2
        libxxhash.so.0 -> libxxhash.so.0.8.0
        libnss_files.so.2 -> libnss_files-2.31.so
        libnsl.so.1 -> libnsl-2.31.so
        libcrypt.so.1 -> libcrypt.so.1.1.0
        libss.so.2 -> libss.so.2.0
        libp11-kit.so.0 -> libp11-kit.so.0.3.0
        libgpg-error.so.0 -> libgpg-error.so.0.29.0
        libuuid.so.1 -> libuuid.so.1.3.0
        libgmp.so.10 -> libgmp.so.10.4.1
        libnettle.so.8 -> libnettle.so.8.3
        libhogweed.so.6 -> libhogweed.so.6.3
        libpcreposix.so.3 -> libpcreposix.so.3.13.3
        libffi.so.7 -> libffi.so.7.1.0
        libpcre2-8.so.0 -> libpcre2-8.so.0.10.1
        libapt-pkg.so.6.0 -> libapt-pkg.so.6.0.0
        libnss_dns.so.2 -> libnss_dns-2.31.so
        libnss_hesiod.so.2 -> libnss_hesiod-2.31.so
        libblkid.so.1 -> libblkid.so.1.1.0
        libpamc.so.0 -> libpamc.so.0.82.1
        libzstd.so.1 -> libzstd.so.1.4.8
        libkeyutils.so.1 -> libkeyutils.so.1.9
        libmount.so.1 -> libmount.so.1.1.0
        libselinux.so.1 -> libselinux.so.1
        libtasn1.so.6 -> libtasn1.so.6.6.0
        libcap-ng.so.0 -> libcap-ng.so.0.0.0
        libsemanage.so.1 -> libsemanage.so.1
        libmemusage.so -> libmemusage.so
        libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2
        libanl.so.1 -> libanl-2.31.so
        libdl.so.2 -> libdl-2.31.so
        libpthread.so.0 -> libpthread-2.31.so
        libe2p.so.2 -> libe2p.so.2.3
        libpam.so.0 -> libpam.so.0.85.1
        libresolv.so.2 -> libresolv-2.31.so
        libcrypto.so.1.1 -> libcrypto.so.1.1
        libthread_db.so.1 -> libthread_db-1.0.so
        libnss_compat.so.2 -> libnss_compat-2.31.so
        libdebconfclient.so.0 -> libdebconfclient.so.0.0.0
/sbin/ldconfig: /lib/aarch64-linux-gnu/ld-2.31.so is the dynamic linker, ignoring

...

I stepped back to the chroot and tried qemu-aarch64-static from 21.04's qemu-user-static (5.2+dfsg-9ubuntu2) but *debootstrap* fails in the same way as previously reported.

HOWEVER, this works:

$ sudo chroot bullseye-arm64 /usr/bin/qemu-aarch64-static /sbin/ldconfig -v
/sbin/ldconfig: Can't stat /usr/local/lib/aarch64-linux-gnu: No such file or directory
/sbin/ldconfig: Path `/usr/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Can't stat /usr/local/lib: No such file or directory
/sbin/ldconfig: Path `/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Path `/usr/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Path `/usr/lib' given more than once
/lib/aarch64-linux-gnu:                                            
        libSegFault.so -> libSegFault.so       
        libpam_misc.so.0 -> libpam_misc.so.0.82.1
        libpcprofile.so -> libpcprofile.so
        libext2fs.so.2 -> libext2fs.so.2.4        
        libattr.so.1 -> libattr.so.1.1.2448
        libssl.so.1.1 -> libssl.so.1.1            
        libnsl.so.2 -> libnsl.so.2.0.1                  
        libc.so.6 -> libc-2.31.so             
        libgcc_s.so.1 -> libgcc_s.so.1              
        libudev.so.1 -> libudev.so.1.7.0
        libz.so.1 -> libz.so.1.2.11            
        libdb-5.3.so -> libdb-5.3.so         
        libtic.so.6 -> libtic.so.6.2    
        libxxhash.so.0 -> libxxhash.so.0.8.0
        libnss_files.so.2 -> libnss_files-2.31.so      
        libnsl.so.1 -> libnsl-2.31.so         
        libcrypt.so.1 -> libcrypt.so.1.1.0 
        libss.so.2 -> libss.so.2.0                 
        libp11-kit.so.0 -> libp11-kit.so.0.3.0
        libgpg-error.so.0 -> libgpg-error.so.0.29.0
        libuuid.so.1 -> libuuid.so.1.3.0
        libgmp.so.10 -> libgmp.so.10.4.1
        libnettle.so.8 -> libnettle.so.8.3
        libhogweed.so.6 -> libhogweed.so.6.3
        libpcreposix.so.3 -> libpcreposix.so.3.13.3
        libffi.so.7 -> libffi.so.7.1.0
        libpcre2-8.so.0 -> libpcre2-8.so.0.10.1
        libapt-pkg.so.6.0 -> libapt-pkg.so.6.0.0
        libnss_dns.so.2 -> libnss_dns-2.31.so
        libnss_hesiod.so.2 -> libnss_hesiod-2.31.so
        libblkid.so.1 -> libblkid.so.1.1.0
        libpamc.so.0 -> libpamc.so.0.82.1
        libzstd.so.1 -> libzstd.so.1.4.8
        libkeyutils.so.1 -> libkeyutils.so.1.9
        libmount.so.1 -> libmount.so.1.1.0
        libselinux.so.1 -> libselinux.so.1
        libtasn1.so.6 -> libtasn1.so.6.6.0
        libcap-ng.so.0 -> libcap-ng.so.0.0.0
        libsemanage.so.1 -> libsemanage.so.1
        libmemusage.so -> libmemusage.so
        libgssapi_krb5.so.2 -> libgssapi_krb5.so.2.2
        libanl.so.1 -> libanl-2.31.so
        libdl.so.2 -> libdl-2.31.so
        libpthread.so.0 -> libpthread-2.31.so
        libe2p.so.2 -> libe2p.so.2.3
        libpam.so.0 -> libpam.so.0.85.1
        libresolv.so.2 -> libresolv-2.31.so
        libcrypto.so.1.1 -> libcrypto.so.1.1
        libthread_db.so.1 -> libthread_db-1.0.so
        libnss_compat.so.2 -> libnss_compat-2.31.so
        libdebconfclient.so.0 -> libdebconfclient.so.0.0.0
/sbin/ldconfig: /lib/aarch64-linux-gnu/ld-2.31.so is the dynamic linker, ignoring

ld-linux-aarch64.so.1 -> ld-2.31.so
        liblzma.so.5 -> liblzma.so.5.2.5
        libkrb5support.so.0 -> libkrb5support.so.0.1
        libacl.so.1 -> libacl.so.1.1.2253
        libkrb5.so.3 -> libkrb5.so.3.3
        libaudit.so.1 -> libaudit.so.1.0.0
        libsepol.so.1 -> libsepol.so.1
        libbz2.so.1.0 -> libbz2.so.1.0.4
        libtinfo.so.6 -> libtinfo.so.6.2
        libstdc++.so.6 -> libstdc++.so.6.0.28
        libk5crypto.so.3 -> libk5crypto.so.3.1
        libsystemd.so.0 -> libsystemd.so.0.30.0
        libcom_err.so.2 -> libcom_err.so.2.1
        libpcre.so.3 -> libpcre.so.3.13.3
        libsmartcols.so.1 -> libsmartcols.so.1.1.0
        libtirpc.so.3 -> libtirpc.so.3.0.0
        libunistring.so.2 -> libunistring.so.2.1.0
        libapt-private.so.0.0 -> libapt-private.so.0.0.0
        libseccomp.so.2 -> libseccomp.so.2.5.1
        liblz4.so.1 -> liblz4.so.1.9.3
        libm.so.6 -> libm-2.31.so 
        libgnutls.so.30 -> libgnutls.so.30.29.1
        libutil.so.1 -> libutil-2.31.so
        libidn2.so.0 -> libidn2.so.0.3.7
        librt.so.1 -> librt-2.31.so
        libBrokenLocale.so.1 -> libBrokenLocale-2.31.so
        libgcrypt.so.20 -> libgcrypt.so.20.2.8
/lib:

I'm not entirely clear on how to further debug this since executing under gdb in the correct context

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-05-12: Re: [Bug 1928075] Re: Segfault with qemu-aarch64-static doing debootstrap for debian bullseye but not buster

#9

...
> And with that installation runs to completion. This is inside a 21.04
> amd64 desktop KVM virtual machine.

  #1 The 21.04 environment fully works
  #2 The 20.04 environment fails
  #3 The qemu-of-21.04 in 20.04 environment fails

I'm not sure, but this makes me think of the kernel maybe being related.
Well this isn't system-virtualization, it is an emulation so the
chances are much lower than usual, but might be worth a try.

In #3 which kernel did you use and is there a chance you could
try the one of 21.04 for a test? Then we'd have
#4 qemu-and-kernel-of-21.04 in 20.04 environment TBD

Another - more cumbersome, but helpful - test could be
- start in a 20.04 virtual machine
- test should fail
- upgrade components to 21.04 one by one
- iterate the test
- once it works, downgrade the component again
=> Can you this way find a single piece that gets us in/out of the bad case?

Changed in qemu (Ubuntu Hirsute):
status:	New → Fix Released
Changed in qemu (Ubuntu Groovy):
status:	New → Incomplete
Changed in qemu (Ubuntu Focal):
status:	New → Incomplete
Changed in qemu (Ubuntu):
status:	New → Fix Released

Revision history for this message

TJ (tj) wrote on 2021-05-12:

#10

All tests using chroot's on my host are with kernel v5.12.0 with the ubuntu kernel .config as base + 'make olddefconfig'.

I'll try the standard Ubuntu 20.04 kernel, and HWE, just in case.

Revision history for this message

TJ (tj) wrote on 2021-05-12:

#11

Testing by trying to execute /sbin/ldconfig within the aarch64 Debian bullseye chroot on 20.04 amd64 host for all installed kernels is working, including 5.12.0, the version booted to by default !

That is: 5.4.0-73-lowlatency 5.8.0-41-lowlatency 5.10.2 5.11.0+ 5.12.0-elloe+

This is extremely difficult to understand since I only got involved in reproducing and testing this on my PC when my colleague hit this issue (recall his PC is on v5.10.0).

Now, using qemu-debootstrap to build a new Debian bullseye also works (remembering to set ownership on the directory to root first!).

$ uname -a
Linux elloe000 5.12.0-elloe+ #31 SMP PREEMPT Sat May 1 10:46:45 BST 2021 x86_64 x86_64 x86_64 GNU/Linux
$ sudo mkdir bullseye2-arm64

$ sudo qemu-debootstrap --arch=arm64 bullseye bullseye2-arm64 http://ftp.debian.o
rg/debian
I: Running command: debootstrap --arch arm64 --foreign bullseye bullseye2-arm64 http://ftp.debian.org/debian
I: Retrieving InRelease
I: Checking Release signature
I: Valid Release signature (key id 0146DC6D4A0B2914BDED34DB648ACFD622F3D138)
I: Retrieving Packages
I: Validating Packages
I: Resolving dependencies of required packages...
I: Resolving dependencies of base packages...
I: Checking component main on http://ftp.debian.org/debian...
...
I: Running command: chroot bullseye2-arm64 /debootstrap/debootstrap --second-stage
I: Installing core packages...
...
I: Unpacking the base system...
...
I: Configuring libc-bin...
I: Base system installed successfully.

$ sudo chroot bullseye2-arm64/ /sbin/ldconfig -v
/sbin/ldconfig: Can't stat /usr/local/lib/aarch64-linux-gnu: No such file or directory
/sbin/ldconfig: Path `/usr/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Path `/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Path `/usr/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Path `/usr/lib' given more than once
/lib/aarch64-linux-gnu:
libSegFault.so -> libSegFault.so
...
/sbin/ldconfig: /lib/aarch64-linux-gnu/ld-2.31.so is the dynamic linker, ignoring

ld-linux-aarch64.so.1 -> ld-2.31.so
...
libgcrypt.so.20 -> libgcrypt.so.20.2.8
/usr/local/lib:
/lib:

Seemingly now working - so now I am extremely confused :s

I'll retest on my colleagues PC tomorrow !

Testing by trying to execute /sbin/ldconfig within the aarch64 Debian bullseye chroot on 20.04 amd64 host for all installed kernels is working, including 5.12.0, the version booted to by default !

That is: 5.4.0-73-lowlatency 5.8.0-41-lowlatency 5.10.2 5.11.0+ 5.12.0-elloe+

This is extremely difficult to understand since I only got involved in reproducing and testing this on my PC when my colleague hit this issue (recall his PC is on v5.10.0).

Now, using qemu-debootstrap to build a new Debian bullseye also works (remembering to set ownership on the directory to root first!).

$ uname -a
Linux elloe000 5.12.0-elloe+ #31 SMP PREEMPT Sat May 1 10:46:45 BST 2021 x86_64 x86_64 x86_64 GNU/Linux
$ sudo mkdir bullseye2-arm64

$ sudo qemu-debootstrap --arch=arm64 bullseye bullseye2-arm64 http://ftp.debian.o
rg/debian                                                   
I: Running command: debootstrap --arch arm64 --foreign bullseye bullseye2-arm64 http://ftp.debian.org/debian            
I: Retrieving InRelease                                     
I: Checking Release signature                               
I: Valid Release signature (key id 0146DC6D4A0B2914BDED34DB648ACFD622F3D138)                                            
I: Retrieving Packages                                      
I: Validating Packages                                      
I: Resolving dependencies of required packages...           
I: Resolving dependencies of base packages...                                                                           
I: Checking component main on http://ftp.debian.org/debian...
... 
I: Running command: chroot bullseye2-arm64 /debootstrap/debootstrap --second-stage
I: Installing core packages...
...
I: Unpacking the base system...
...
I: Configuring libc-bin...
I: Base system installed successfully.

$ sudo chroot bullseye2-arm64/ /sbin/ldconfig -v
/sbin/ldconfig: Can't stat /usr/local/lib/aarch64-linux-gnu: No such file or directory
/sbin/ldconfig: Path `/usr/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Path `/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Path `/usr/lib/aarch64-linux-gnu' given more than once
/sbin/ldconfig: Path `/usr/lib' given more than once
/lib/aarch64-linux-gnu:                                     
        libSegFault.so -> libSegFault.so
...
/sbin/ldconfig: /lib/aarch64-linux-gnu/ld-2.31.so is the dynamic linker, ignoring

ld-linux-aarch64.so.1 -> ld-2.31.so
...
        libgcrypt.so.20 -> libgcrypt.so.20.2.8
/usr/local/lib:
/lib:

Seemingly now working - so now I am extremely confused :s

I'll retest on my colleagues PC tomorrow !

Revision history for this message

TJ (tj) wrote on 2021-05-12:

#12

Before I forget and in case others find it useful I wrote a simple Python tool to check binfmt magic bytes registered with the kernel with particular files.

It can be found here:

https://iam.tj/projects/misc/binfmt-check

Revision history for this message

TJ (tj) wrote on 2021-05-13:

#13

We've installed 5.12.0 on Callum's PC and the latest qemu-user-static (5.2+dfsg-9ubuntu3) from 21.04 and so far it also appears to be working. We're not sure what has been happening but we'll chalk this one down to user-error for now unless it re-occurs!

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-05-17:

#14

Thanks TJ!

I agree to "I am extremely confused" - /e as well :-)

I mean I'm glad it was resolved by kernel upgrades, but there isn't much attack surface left to solve anything on the distro-packaging level.

Let me know if you find the underlying root cause so we can have a look at it.

Revision history for this message

Randy Fay (randyfay) wrote on 2021-10-15:

#15

I was able to solve this on Ubuntu 20.04 by moving to latest qemu 5 using
```
sudo add-apt-repository ppa:jacob/virtualisation (for Ubuntu 20.04)
sudo apt-get update && sudo apt-get install qemu qemu-user qemu-user-static
```

Revision history for this message

Bryce Harrington (bryce) wrote on 2021-10-18:

#16

[Groovy is past its end of support period]

Changed in qemu (Ubuntu Groovy):
status:	Incomplete → Won't Fix
Changed in qemu (Ubuntu Focal):
status:	Incomplete → New

Revision history for this message

Bryce Harrington (bryce) wrote on 2021-10-18:

#17

That is an interesting data point that the newer qemu version from a PPA seems to resolve it.
TJ, can you confirm this is true for you as well?

Randy, do you have any insights on which fix from the newer qemu might be worth looking at for SRU?

Changed in qemu (Ubuntu Focal):
status:	New → Incomplete

Revision history for this message

frank (frankwu) wrote on 2021-10-27:

#18

In my test,arm64 works with ppa qemu-user-static,but it breaks armhf in similar manner as arm64 with ubuntu 20.4 version. Seems to affect only bullseye chroot for me. Buster can be created without problems.

Imho this confirms problem with linkage type of ldconfig (dynamically vs. Static) like tj mentioned in #3.i guess this can be only fixed by replacing binary (maybe libs) inside chroot and not the qemu-user-static

Revision history for this message

frank (frankwu) wrote on 2021-11-28 (last edit on 2021-12-23):

#19

sorry, did not get an email about answers and then saw i posted twice

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-11-29:

#20

Hi Frank,
I was trying to grasp the exact details that you are describing and giving this another repro run. You are saying
bullseye armhf chroot @ focal on an x86 host
is failing for you, correct?

Note: We have just fixed a similar (only on the surface of it) issue in Impish (bug 1947860).

Repro on Focal:

Note: As Host I was using a new VM based on the current daily focal cloud image which is based on 5.4.0-90-generic.

$ apt install qemu-user-static debootstrap
$ sudo qemu-debootstrap --arch=arm64 bullseye bullseye-arm64 http://ftp.debian.org/debian
# and in another terminal
$ sudo qemu-debootstrap --arch=armhf bullseye bullseye-armhf http://ftp.debian.org/debian

For me as of the test today armhf worked and arm64 failed.

$ tail -n 2 bullseye-arm64/debootstrap/debootstrap.log
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)

I was fetching TJs test script from https://iam.tj/projects/misc/binfmt-check and running it in the chroots. In the partially failing arm64 chroot all that is a bit tricky and might need reruns.

The magic in both cases matches:

ubuntu@f-emuarmhf-1928075:~$ python3 binfmt-check qemu-aarch64 bullseye-arm64/sbin/ldconfig
target_file=bullseye-arm64/sbin/ldconfig
octets=20
mask=ffffffffffffff00fffffffffffffffffeffffff
target=7f454c460201010300000000000000000300b700
result=7f454c460201010000000000000000000200b700
magic=7f454c460201010000000000000000000200b700
Matches magic
ubuntu@f-emuarmhf-1928075:~$ python3 binfmt-check qemu-arm bullseye-armhf/sbin/ldconfig
target_file=bullseye-armhf/sbin/ldconfig
octets=20
mask=ffffffffffffff00fffffffffffffffffeffffff
target=7f454c4601010103000000000000000002002800
result=7f454c4601010100000000000000000002002800
magic=7f454c4601010100000000000000000002002800
Matches magic

Then I tried 1:6.0+dfsg-2expubuntu1.1~backport20.04-202111060014~ubuntu20.04.1 from https://launchpad.net/~canonical-server/+archive/ubuntu/server-backports

With the qemu-user-static package from that indeed armhf and arm64 works.
Magic handling stays the same

We had quite some binfmt changes in 6.1 and 5.2, but quite maybe as reported in comment #15 even qemu 5.0 would have been enough. After all 4.2-3 (just after focal) did switch the registration of binfmts.

This case certainly can be considered confirmed now, the questions for now is if that was the actual qemu code, or the surrounding changes/fixups to binfmt handling.

P.S. /me hopes this problem will not again become as eluding as it was before ...

Unless someone can directly point to a fix I think I need to build a series of PPAs to bisect through that a bit until we have an idea what the fix really was.

Hi Frank,
I was trying to grasp the exact details that you are describing and giving this another repro run. You are saying
  bullseye armhf chroot @ focal on an x86 host
is failing for you, correct?

Note: We have just fixed a similar (only on the surface of it) issue in Impish (bug 1947860).

Repro on Focal:

Note: As Host I was using a new VM based on the current daily focal cloud image which is based on 5.4.0-90-generic.

$ apt install qemu-user-static debootstrap
$ sudo qemu-debootstrap --arch=arm64 bullseye bullseye-arm64 http://ftp.debian.org/debian
# and in another terminal
$ sudo qemu-debootstrap --arch=armhf bullseye bullseye-armhf http://ftp.debian.org/debian

For me as of the test today armhf worked and arm64 failed.

$ tail -n 2  bullseye-arm64/debootstrap/debootstrap.log
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)

I was fetching TJs test script from https://iam.tj/projects/misc/binfmt-check and running it in the chroots. In the partially failing arm64 chroot all that is a bit tricky and might need reruns.

The magic in both cases matches:

ubuntu@f-emuarmhf-1928075:~$ python3 binfmt-check qemu-aarch64 bullseye-arm64/sbin/ldconfig 
target_file=bullseye-arm64/sbin/ldconfig
octets=20
  mask=ffffffffffffff00fffffffffffffffffeffffff
target=7f454c460201010300000000000000000300b700
result=7f454c460201010000000000000000000200b700
 magic=7f454c460201010000000000000000000200b700
Matches magic
ubuntu@f-emuarmhf-1928075:~$ python3 binfmt-check qemu-arm bullseye-armhf/sbin/ldconfig 
target_file=bullseye-armhf/sbin/ldconfig
octets=20
  mask=ffffffffffffff00fffffffffffffffffeffffff
target=7f454c4601010103000000000000000002002800
result=7f454c4601010100000000000000000002002800
 magic=7f454c4601010100000000000000000002002800
Matches magic

Then I tried 1:6.0+dfsg-2expubuntu1.1~backport20.04-202111060014~ubuntu20.04.1 from https://launchpad.net/~canonical-server/+archive/ubuntu/server-backports

With the qemu-user-static package from that indeed armhf and arm64 works.
Magic handling stays the same

We had quite some binfmt changes in 6.1 and 5.2, but quite maybe as reported in comment #15 even qemu 5.0 would have been enough. After all 4.2-3 (just after focal) did switch the registration of binfmts.

This case certainly can be considered confirmed now, the questions for now is if that was the actual qemu code, or the surrounding changes/fixups to binfmt handling.

P.S. /me hopes this problem will not again become as eluding as it was before ...

Unless someone can directly point to a fix I think I need to build a series of PPAs to bisect through that a bit until we have an idea what the fix really was.

Changed in qemu (Ubuntu Focal):
status:	Incomplete → Confirmed
tags:	added: server-next

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-11-29:

#21

Using PPAs:
https://launchpad.net/~paelzer/+archive/ubuntu/lp-1928075-qemu-static-builds-h
https://launchpad.net/~paelzer/+archive/ubuntu/lp-1928075-qemu-static-builds-g

Test results:
works 1:6.0+dfsg-2expubuntu1.1~backport20.04-202111060014~ubuntu20.04.1
works 1:5.2+dfsg-9ubuntu3.3~focalppa1
works 1:5.0-5ubuntu9.10~focalppa1
fails 1:4.2-3ubuntu6.18

This is confirming that what we had in 5.0 is already enough

Next step (all in regard to binfmt):
- 4.2 with the debian/* content of 5.0
- 5.0 but with binfmt of debian/* as it was in 4.2
^ this takes some time to work, I hope to have more PPAs in a bit

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-11-29:

#22

already in 1:4.2-3ubuntu5:
commit 8075687f92b6e177f646a9a7db8cfd63262d033a switch binfmt registration to use update-binfmts --[un]import (#866756)

Remaining fixes
commit 2a3a3f66c07e3ccae80115d994c627fe9df19880 more binfmt-install updates
commit 9ce886cd819b1f9900b84c55f416977e8f418d27 fix improper quoting in d/binfmt-install (#959222)
commit f0caededd5a47ed97276253ea2348688f6cda87d check systemd-detect-virt before running update-binfmt
commit 87965f019e1101037d0b9134dbc090275238ff5c x32 arch is in the same family as i386 & x86_64, comit binfmt registration

TL;DR this should only fix to not fail registration in containers (still a reasonable fix) but maybe not what we need here.

Two more PPAs one if Focal plus those fixes, the other one Groovy minus those fixes as identified above.

PPAs (just started building):
https://launchpad.net/~paelzer/+archive/ubuntu/lp-1928075-qemu-static-builds-g-minusfix
https://launchpad.net/~paelzer/+archive/ubuntu/lp-1928075-qemu-static-builds-f-plusfix

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-11-29:

#23

Works - 1:5.0-5ubuntu9.10~focalppa2 = lp-1928075-qemu-static-builds-g-minusfix

Since the debian/* is the same in regard to binfmt this might indicate a fix in qemu git 4.2 .. 5.0

The other PPA still builds

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-11-30:

#24

To complete the former tests:
fail - 1:4.2-3ubuntu6.19~focalppa2 = lp-1928075-qemu-static-builds-f-plusfix

Which confirms the former thought that it isn't about the binfmt changes in debian/* but an upstream change.

I'll need some builds for that ...

P.S. somewhere in the back of my mind forms a memory about a similar issue, and we did identify the change and found that it was not backportable for potentially causing more regressions than fixes. But sadly I can't find the commit/notes/debug. So another bisect it will have to be :-/

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-11-30:

#25

After being set up in the broken way we can run the test more quickly

$ sudo chroot /home/ubuntu/bullseye-arm64 /bin/sh /debootstrap/debootstrap --second-stage
$ tail -n 1 /home/ubuntu/bullseye-arm64/debootstrap/debootstrap.log

This is repeatable and will as last line contain:
good case:
duplicate value for 'Package' field
bad case:
Segmentation fault (core dumped)

That can be used in a git-bisect script,

Due to "fix_binary" in binfmt we need to replace the interpreter after build.
Otherwise it will use the installed one even if overwriting the file in /usr.
#once
$ sudo update-binfmts --unimport qemu-aarch64

Then on each loop:
$ sudo update-binfmts --remove qemu-aarch64 /home/ubuntu/qemu-aarch64-static.test
$ sudo update-binfmts --install qemu-aarch64 /home/ubuntu/qemu-aarch64-static.test --magic '\x7f\x45\x4c\x46\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00' --mask '\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff' --fix-binary yes --credentials yes
$ sudo update-binfmts --display qemu-aarch64

Checking if v5.0 and v4.2 behave as expected - with the above they do (one works, one fails)

Started bisect and will come back later ...

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2021-11-30:

#26

Download full text (6.1 KiB)

Bisect completed.
I deleted the PPAs as they are no more needed.

$ grep -e '^DEBUG' -e '^Com' bisect.log
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-1511-g381063d778'
DEBUG: latest result is ' duplicate value for 'Package' field'
Completed - RC=1
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-755-ge65cec5e5d'
DEBUG: latest result is 'Segmentation fault (core dumped)'
Completed - RC=0
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-1133-gfa7c8e92cb'
DEBUG: latest result is ' duplicate value for 'Package' field'
Completed - RC=1
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-944-gfe4c15798a'
DEBUG: latest result is ' duplicate value for 'Package' field'
Completed - RC=1
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-849-g8c3fe75e03'
DEBUG: latest result is ' duplicate value for 'Package' field'
Completed - RC=1
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-802-gbe9612e8cb'
DEBUG: latest result is 'Segmentation fault (core dumped)'
Completed - RC=0
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-821-g7cea426c1d'
DEBUG: latest result is 'Segmentation fault (core dumped)'
Completed - RC=0
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-799-gfa857eb570'
DEBUG: latest result is ' duplicate value for 'Package' field'
Completed - RC=1
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-792-g08e3ce59fc'
DEBUG: latest result is ' duplicate value for 'Package' field'
Completed - RC=1
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-788-g5ae774a91f'
DEBUG: latest result is ' duplicate value for 'Package' field'
Completed - RC=1
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-787-g6fd5944980'
DEBUG: latest result is ' duplicate value for 'Package' field'
Completed - RC=1
DEBUG: saved as '/home/ubuntu/qemu-aarch64-static.bisect.v4.2.0-786-g91c8bdb1e6'
DEBUG: latest result is 'Segmentation fault (core dumped)'
Completed - RC=0

$ git bisect log
git bisect start
# old: [b0ca999a43a22b38158a222233d3f5881648bb4f] Update version for v4.2.0 release
git bisect old b0ca999a43a22b38158a222233d3f5881648bb4f
# new: [fdd76fecdde1ad444ff4deb7f1c4f7e4a1ef97d6] Update version for v5.0.0 release
git bisect new fdd76fecdde1ad444ff4deb7f1c4f7e4a1ef97d6
# new: [381063d778a5aa9dcf84a2284a192d84746b2e0f] linux-user: microblaze: Update syscall numbers to kernel 5.5 level
git bisect new 381063d778a5aa9dcf84a2284a192d84746b2e0f
# old: [e65cec5e5d97927d22b39167d3e8edeffc771788] migration/ram: Yield periodically to the main loop
git bisect old e65cec5e5d97927d22b39167d3e8edeffc771788
# new: [fa7c8e92cb9bb004359926497675a9b7d0099dfc] hw/arm/virt: Add missing 5.0 options call to 4.2 options
git bisect new fa7c8e92cb9bb004359926497675a9b7d0099dfc
# new: [fe4c15798a48143dd6b1f58d2d3cad12206ce211] virtiofsd: Convert lo_destroy to take the lo->mutex lock itself
git bisect new fe4c15798a48143dd6b1f58d2d3cad12206ce211
# new: [8c3fe75e0308ba2f01d160ace534b7e386cea808] virtiofsd: Remove unused enum fuse_buf_copy_flags
git bisect new 8c3fe75e0308ba2f01d160ace534b7e386cea808
# old: [be9612e8cbb4b5e5d4c5f66551db2b4d6...

	Status	Importance	Assigned to
qemu (Ubuntu)	Fix Released	Undecided	Unassigned
Focal	Confirmed	Undecided	Unassigned
Groovy	Won't Fix	Undecided	Unassigned
Hirsute	Fix Released	Undecided	Unassigned

Ubuntu
qemu package

Segfault with qemu-aarch64-static doing debootstrap for debian bullseye but not buster

Bug Description

Other bug subscribers

Remote bug watches

Ubuntuqemu package

Segfault with qemu-aarch64-static doing debootstrap for debian bullseye but not buster

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
qemu package