Regression: qemu-user-static binaries are dynamically linked

Bug #1908331 reported by TJ on 2020-12-16
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
qemu (Ubuntu)
High
Unassigned
Groovy
Medium
Unassigned

Bug Description

[ Impact ]

 * An upstream change [1] had undesired impact of making qemu-user-static
   no more fully static. In some cases e.g. cross arch debootstrapping
   that breaks the use case.

 * The root cause is the use of ld with an unsupported pie related flag.
   We have fixed this upstream and this backports that change.

[Test Case]

 * $ file $(dpkg -L qemu-user-static | grep bin\/qemu-.*-static)

  Should show "statically linked", but not "dynamically linked" (bad case)

[Where problems could occur]

 * I can't think of an immediate expected regression that comes to mind,
   but the following at least limits the area to look for issues.
   The change it limited to qemu-user-static. Therefore impacts would be
   seen there but not in e.g. system emulation (other binaries, those
   didn't change except for being rebuilt).
   Also the new behavior matches what we had pre-groovy and therfore what
   users expect.

[Other Info]

 * n/a

[1]: https://git.qemu.org/?p=qemu.git;a=commit;h=127814629b32d5e0de2873d742e08cb9bd412af7

---

On 20.04 (qemu 4.2) the binaries built for qemu-user-static - specifically in the case I've hit /usr/bin/qemu-aarch64-static - are completely static executables.

The same binaries from a qemu 5.x build are not. Although they don't link to other shared libraries they are dynamically linked to glibc and therefore require the same version of the glibc shared libraries at runtime. This breaks many uses in foreign architecture chroots; in my case an aarch64 being built with debootstrap:

On focal:

$ file /usr/bin/qemu-aarch64-static
/usr/bin/qemu-aarch64-static: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=abad039a2cfc5bc87215554230a572b085fbc37a, for GNU/Linux 3.2.0, with debug_info, not stripped

$ dpkg -S /usr/bin/qemu-aarch64-static
qemu-user-static: /usr/bin/qemu-aarch64-static

$ apt list qemu-user-static
qemu-user-static/focal-updates,focal-security,now 1:4.2-3ubuntu6.10 amd64 [installed]

But recent builds of 5.2 are not; for example from hirsute (qemu-user-static_5.2+dfsg-2ubuntu1_amd64.deb)

$ file /tmp/qemu-aarch64-static
/tmp/qemu-aarch64-static: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=91fda2fa08f46d1bb6d19b6f72a4819a7c20fd7e, for GNU/Linux 3.2.0, stripped

I hit this whilst trying to track down another bug and building upstream git HEAD on 20.04 with:

$ ../../qemu/configure --disable-system --enable-linux-user --static --target-list=aarch64-linux-user
$ make
$ file qemu-aarch64
qemu-aarch64: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=546ba11f0de940f7f3cbfaffae3c2bb54b683386, for GNU/Linux 3.2.0, with debug_info, not stripped

It looks like changes to the qemu build system are responsible. I asked on OFTC #qemu and at that time no-one was particularly aware of the significance/difference.

Looking at the configure output summary between the focal and hirsute/upstream builds I noticed that there is no longer a separate summary for LDFLAGS - it now only reports QEMU_LDFLAGS.

That seems significant since focal passed "-static" with LDFLAGS not QEMU_LDFLAGS:

LDFLAGS -Wl,--warn-common -m64 -static -g -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed
QEMU_LDFLAGS

whereas hirsute shows:

QEMU_LDFLAGS: -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -static-pie -m64 -g -O2 -fdebug-prefix-map=/<<BUILDDIR>>/qemu-5.2+dfsg=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed -fstack-protector-strong

(notice it is -static-pie not -static now)

I've seen mention in qemu commit messages of the meson build system changes causing issues around passing of linker flags but it needs someone familiar with the project and build system to figure this out.

Related branches

TJ (tj) on 2020-12-16
description: updated
TJ (tj) on 2020-12-16
description: updated
Revision history for this message
TJ (tj) wrote :

To be sure I built upstream v4.2.1 and it reports the expected and required linkage. Build needs some additional tweaking to avoid libssh deprecated functions causing -Werror to trigger:

$ sudo apt install liblzma-dev

$ ../../qemu/configure --target-list=aarch64-linux-user --static --disable-system --enable-linux-user --extra-cflags='-Wno-deprecated-declarations'

$ make -j8 V=1

$ file ./aarch64-linux-user/qemu-aarch64
./aarch64-linux-user/qemu-aarch64: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=70f5e10ac0eb9b63d7758850e2f18d0a047d4b79, for GNU/Linux 3.2.0, with debug_info, not stripped

Revision history for this message
TJ (tj) wrote :

I've posted a question to the qemu-devel mailing list with subject "Are user static builds really dynamically linked ?"

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thanks for the great report TJ.

Indeed configure --static converts to
4.2
LDFLAGS -Wl,--warn-common -m64 -static -g -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed
QEMU_LDFLAGS·······

5.0
QEMU_LDFLAGS -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -static-pie -m64 -g -O2 -fdebug-prefix-map=/<<PKGBUILDDIR>>=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--as-needed -fstack-protector-strong

So as reported no more LDFLAGS/QEMU_LDFLAGS split (there was some cleanup).
And also as reported I agree to the maybe even more important it seems it switched from "-static" to "-static-pie"

---

There is even what seems to be a related build warning.
I picked one sample from the static build and found these.

4.2:
c++ ... -m64 -static ... -o qemu-arm exec.o ... -lnettle -lrt

5.0:
c++ ... -static-pie -m64 ... -o qemu-arm exec.o ... -lnettle -lrt
(.text+0x277): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

IMHO - That sounds very related

@TJ - don't start a new build for it , but if you can easily check in the logs do you also see these warnings?

P.S. I've found the qemu 5.2 build in Hirsute to be affected as well

Changed in qemu (Ubuntu):
status: New → Triaged
tags: added: server-next
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

If -static-pie is indeed impoartant here - it is rather new and had problems before.
An excerpt from d/rules:

114 ifneq ($(filter $(DEB_HOST_ARCH),i386 x32),)
115 # XXX as of 2020-04-29, i386 buildd fails to link -static-pie executable:
116 # /usr/bin/ld: /usr/lib/i386-linux-gnu/libc.a(memset_chk-nonshared.o):
117 #»··unsupported non-PIC call to IFUNC `memset'
118 # so disable PIE for qemu-user-static build on i386 for now.
119 # On x32 -static-pie fails in different way.
120 # On amd64 -static-pie appears to work fine.
121 # On other architectures qemu correctly detect that -static-pie is not available
122 XXXstaticpie = --disable-pie
123 else
124 XXXstaticpie =
125 endif

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Confirmed to recreate from git as well:

v5.0.0
./aarch64-linux-user/qemu-aarch64: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=1521984faacf9472c1df8fe37c2a0fb5e2f55aba, for GNU/Linux 3.2.0, with debug_info, not stripped

v4.2.0
./aarch64-linux-user/qemu-aarch64: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=f646a883f4f587f17be145a4033d201c2579effa, for GNU/Linux 3.2.0, with debug_info, not stripped

FYI 4.2 also shows the warnings like
(.text+0x277): warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
So that might have been a red herring

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Building v5.0.0 with --disable-pie:

flags now are:
  QEMU_LDFLAGS -Wl,--warn-common -Wl,-z,relro -Wl,-z,now -static -m64 -fstack-protector-strong

And the file is as we'd like it to be:
./aarch64-linux-user/qemu-aarch64: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=af9a7155d48effae761623c97d59b2c0a0d77c2d, for GNU/Linux 3.2.0, with debug_info, not stripped

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Bisect agrees

$ git bisect log
git bisect start
# old: [b0ca999a43a22b38158a222233d3f5881648bb4f] Update version for v4.2.0 release
git bisect old b0ca999a43a22b38158a222233d3f5881648bb4f
# new: [fdd76fecdde1ad444ff4deb7f1c4f7e4a1ef97d6] Update version for v5.0.0 release
git bisect new fdd76fecdde1ad444ff4deb7f1c4f7e4a1ef97d6
# old: [381063d778a5aa9dcf84a2284a192d84746b2e0f] linux-user: microblaze: Update syscall numbers to kernel 5.5 level
git bisect old 381063d778a5aa9dcf84a2284a192d84746b2e0f
# old: [6d49d3a859b0f19226dbb0df5e7f50267b42f45c] luks: extract qcrypto_block_calculate_payload_offset()
git bisect old 6d49d3a859b0f19226dbb0df5e7f50267b42f45c
# old: [c8c35e5f51c4d54bced7aa05fbd8e2371e493182] Add rx-softmmu
git bisect old c8c35e5f51c4d54bced7aa05fbd8e2371e493182
# new: [8ffb7265af64ec81748335ec8f20e7ab542c3850] net: tulip: check frame size and r/w data length
git bisect new 8ffb7265af64ec81748335ec8f20e7ab542c3850
# old: [bb1ce44b15f159b67fafc5f4b285bbf20a1961e9] qga: Installer: Wait for installation to finish
git bisect old bb1ce44b15f159b67fafc5f4b285bbf20a1961e9
# old: [6d8e7738b0375d9bf247b73ed4739f741df03f21] tests/docker: Use Python3 PyYAML in the Fedora image
git bisect old 6d8e7738b0375d9bf247b73ed4739f741df03f21
# new: [84878f4c00a7beca1d1460e2f77a6c833b8d0393] target/mips: Fix loongson multimedia condition instructions
git bisect new 84878f4c00a7beca1d1460e2f77a6c833b8d0393
# old: [127fe8643340c330cbc0a4194ea56e3d522dbb71] Merge remote-tracking branch 'remotes/stsquad/tags/pull-testing-270320-2' into staging
git bisect old 127fe8643340c330cbc0a4194ea56e3d522dbb71
# old: [ee5195ee0fc87858088313f2c6f327ac41f5912f] configure: Drop adjustment of textseg
git bisect old ee5195ee0fc87858088313f2c6f327ac41f5912f
# old: [b26341241bbfe9cc126479a0dbed5d40d547f242] configure: Always detect -no-pie toolchain support
git bisect old b26341241bbfe9cc126479a0dbed5d40d547f242
# old: [2c674109c2da3f2e17b69f39e0b93b3f3d3dfa59] configure: Override the os default with --disable-pie
git bisect old 2c674109c2da3f2e17b69f39e0b93b3f3d3dfa59
# new: [127814629b32d5e0de2873d742e08cb9bd412af7] configure: Support -static-pie if requested
git bisect new 127814629b32d5e0de2873d742e08cb9bd412af7
# first new commit: [127814629b32d5e0de2873d742e08cb9bd412af7] configure: Support -static-pie if requested

127814629b32d5e0de2873d742e08cb9bd412af7 is the first new commit
commit 127814629b32d5e0de2873d742e08cb9bd412af7
Author: Richard Henderson <email address hidden>
Date: Tue Dec 17 15:30:14 2019 -1000

    configure: Support -static-pie if requested

    Recent toolchains support static and pie at the same time.

    As with normal dynamic builds, allow --static to default to PIE
    if supported by the toolchain. Allow --enable/--disable-pie to
    override the default.

    Reviewed-by: Alex Bennée <email address hidden>
    Signed-off-by: Richard Henderson <email address hidden>
    ---
    v2: Fix --disable-pie --static
    v3: Update for QEMU_LDFLAGS.

 configure | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)
bisect run success

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Now that root-cause and solution is rather clear I have submitted that to Debian (which is affected the same way).
But I will personally enter the EOY-downtime before I can fix it in an Ubuntu upload - OTOH this isn't super urgent I guess - and if my prio-assumption is wrong the fix is available to apply right here.

=> https://salsa.debian.org/qemu-team/qemu/-/merge_requests/16

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

My change was accepted upstream and in 5.2-3 - I'll merge that in the coming days.

Changed in qemu (Ubuntu):
importance: Undecided → High
Changed in qemu (Ubuntu Groovy):
importance: Undecided → Medium
description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

SRU prepared (template in bug description) and tested against the PPA (https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4389) and reviewed (https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/395786).

I will upload that to -unapproved soon

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Uploaded and ready for the review by the SRU Team

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Review and pre-tests ok, uploaded to 21.04

Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello TJ, or anyone else affected,

Accepted qemu into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:5.0-5ubuntu9.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in qemu (Ubuntu Groovy):
status: New → Fix Committed
tags: added: verification-needed verification-needed-groovy
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

1:5.0-5ubuntu9.2

$ file $(dpkg -L qemu-user-static | grep bin\/qemu-.*-static)
...
/usr/bin/qemu-xtensaeb-static: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=99dc333d7f8a78620600e5807de92b60dba54468, for GNU/Linux 3.2.0, stripped

1:5.0-5ubuntu9.3

$ file $(dpkg -L qemu-user-static | grep bin\/qemu-.*-static)
...
/usr/bin/qemu-xtensaeb-static: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=9af415b1e42c28a1a6fd39daa663b661a0c3e175, for GNU/Linux 3.2.0, stripped
$ file $(dpkg -L qemu-user-static | grep bin\/qemu-.*-static) | grep -v "statically linked" | wc -l
0

No dynamic links left - verified

tags: added: verification-done verification-done-groovy
removed: verification-needed verification-needed-groovy
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:5.2+dfsg-3ubuntu1

---------------
qemu (1:5.2+dfsg-3ubuntu1) hirsute; urgency=medium

  * Merge with Debian unstable, includes fixes for
    - qemu-user-static are partially dynamically linked (LP: #1908331)
    - qemu crashing when using spice without qemu-system-gui being
      installed (LP: #1908577)
    Remaining changes:
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-system-common.qemu-kvm.service: systemd unit to call
        qemu-kvm-init
      - d/qemu-system-common.install: install helper script
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: call dh_installinit and dh_installsystemd for qemu-kvm
    - Distribution specific machine type (LP: 1304107 1621042)
      - d/p/ubuntu/define-ubuntu-machine-types.patch: distro machine types
      - d/qemu-system-x86.NEWS Info on fixed machine type definitions
        for host-phys-bits=true (LP: 1776189)
      - add an info about -hpb machine type in debian/qemu-system-x86.NEWS
      - provide pseries-bionic-2.11-sxxm type as convenience with all
        meltdown/spectre workarounds enabled by default. (LP: 1761372).
      - ubuntu-q35 alias added to auto-select the most recent q35 ubuntu type
    - Enable nesting by default
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
        [ No more strictly needed, but required for backward compatibility ]
    - improved dependencies
      - Make qemu-system-common depend on qemu-block-extra
      - Make qemu-utils depend on qemu-block-extra
      - let qemu-utils recommend sharutils
    - tolerate ipxe size change on migrations to >=18.04 (LP: 1713490)
      - d/p/ubuntu/pre-bionic-256k-ipxe-efi-roms.patch: old machine types
        reference 256k path
      - d/control-in: depend on ipxe-qemu-256k-compat-efi-roms to be able to
        handle incoming migrations from former releases.
    - d/control-in: Disable capstone disassembler library support (universe)
    - d/qemu-system-x86.README.Debian: add info about updated nesting changes
    - d/control*, d/rules: disable xen by default, but provide universe
      package qemu-system-x86-xen as alternative
      [includes compat links changes of 5.0-5ubuntu4]
    - allow qemu to load old modules post upgrade (LP 1847361)
      - Drop d/qemu-block-extra.*.in, d/qemu-system-gui.*.in
      - d/rules: Drop generating package version into maintainer scripts
      - d/qemu-system-gui.prerm: add no-op prerm to overcome upgrade issues on
        the bad old prerm (LP 1906245 1905377)
    - d/p/ubuntu/lp-1907789-build-no-pie-is-no-functional-liker-flag.patch: fix
      ld usage of -no-pie (LP 1907789)

 -- Christian Ehrhardt <email address hidden> Tue, 05 Jan 2021 12:43:42 +0100

Changed in qemu (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:5.0-5ubuntu9.3

---------------
qemu (1:5.0-5ubuntu9.3) groovy; urgency=medium

  * d/p/ubuntu/lp-1907656-s390x-s390-virtio-ccw-Reset-PCI-devices-during-subsy:
    avoid PCI devices to become unavailable on reset (LP: #1907656)
  * d/rules: fix qemu-user-static to really be static (LP: #1908331)

 -- Christian Ehrhardt <email address hidden> Tue, 05 Jan 2021 15:46:16 +0100

Changed in qemu (Ubuntu Groovy):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers