qemu-arm-static swallows the program name, and shifts all arguments left

[Kai Horstmann]

[Impact]

 * The kernel changed handling of binfmt-P and that breaks qemu
   using binfmt registering TCG based emulators to run foreign
   architecture code.

 * This happened late 5.11.0-20 -> 5.13.0-20 and even as of today
   cloud-images start with the former :-/ That hides this unless you
   properly upgrade and reboot.

 * This was already reported and fixed in 1:6.1+dfsg-5 and this is
   backporting this fix

[Test Plan]

 * Based on the great report and evolving on that there is a small
   but very useful test.
   1. System (Kernel + userspace dependent, so it needs to be a VM)
      => Get a VM of the target release to test
   2. Install dependencies
    $ sudo apt update
    $ sudo apt upgrade -y
    $ sudo apt install -y qemu-user-static debootstrap
   3. Prep foreign arch chroots for armhf
      (other architectures would work as well)
      # I tested this with various userspace versions, but behavior
      # only depends on kernel + qmeu-user-static
    $ sudo debootstrap --foreign --verbose --arch=armhf jammy armTest-J
   4. Run something int he chroot, which will trigger binfmt -> qemu
    $ sudo chroot armTest-I echo 1 2 3

   Bad result: 2 3
   Good case with fix: 1 2 3

There also is an extended testcase in the report, but it tests and achieves the same, so I picked this simpler one. Everyone is welcome to test his own case once this is up for SRU verification.

[Where problems could occur]

 * This is changing binfmt integration for qemu-user-static, so of the
   many things qemu does we do not have to think about classic
   virtualization or even emulation. Only cross arch emulation through
   binfmt (like the test examples with qemu-user-static + foreign arch
   debootstrap) are what will be affected.

[Other Info]

 * Older qemu seems non-affected by using binfmt differently back then.
 * While this is actually a regression-by-kernel I want to fix this in
   qemu, the new qemu code works fine with the older kernel behavior as
   well.

---- ---- ----

Problem with qemu-arm-static version 1:6.0+dfsg-2expubuntu1, as of recent Impish
When using qemu-arm-static for cross-debootstrap for an ARM machine argument 0 of the invoked ARM program does not contain the name of the executable but argument 1, argument 2 is passed as argument 1 and so forth.
Thus all program calls are missing the 1st argument.
Eample: A shell script test.sh

-- Start --------------------
#!/bin/sh

echo "\$0=$0"
echo "\$1=$1"
echo "\$2=$2"
echo "\$*=$*"
-- End --------------------

Running the script as "./test.sh a bb ccc dddd" emits:
$0=./test.sh
$1=a
$2=bb
$*=a bb ccc dddd

Reproduce the erroneous behavior of qemu-arm-static
Create a directory armTest. Enter:
sudo debootstrap --foreign --verbose --arch=armhf impish armTest
sudo cp test.sh armTest
sudo cp -v /usr/bin/qemu-arm-static armTest/usr/bin

Now run
sudo chroot armTest ./test.sh a bb ccc dddd
The script is now run by the shell for the ARM architecture, and is invoked by qemu-arm-static.
The script now emits
$0=a
$1=bb
$2=ccc
$*=bb ccc dddd

argument 0 (the program name) disappeared, and all arguments are shifted up.

Evidence that actually qemu-arm-static is the culprit:
In /etc/apt/sources.list add the hirsute main, main-update, universe and universe-update repositories. Enter:
sudo apt-get update
sudo apt-get install qemu-user-static/hirsute

Confirm that you want to downgrade the program to version 1:5.2+dfsg-9ubuntu3.2 (of hirsute-update).

Copy the downgraded qemu-arm-static to armTest/usr/bin. Repeat the chroot command. Now the result is the same as invoking test.sh directly under your x86_64 architecture.

This bug does not only affect shell scripts. Apparently all programs are affected by the shifted program arguments.

This bug makes a complete cross-debootstrap impossible because the second debootstrap stage is run as chroot in the target architecture. Further steps to prepare a ready-to-boot image/SD card are equally impossible.

ProblemType: Bug
DistroRelease: Ubuntu 21.10
Package: qemu-user-static 1:6.0+dfsg-2expubuntu1
ProcVersionSignature: Ubuntu 5.13.0-20.20-generic 5.13.14
Uname: Linux 5.13.0-20-generic x86_64
ApportVersion: 2.20.11-0ubuntu70
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: LXQt
Date: Wed Oct 20 13:35:52 2021
KvmCmdLine: COMMAND STAT EUID RUID PID PPID %CPU COMMAND
MachineType: VMware, Inc. VMware Virtual Platform
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.13.0-20-generic root=UUID=f5ea75df-ad8e-4112-8537-372efa9f6a4b ro splash quiet
SourcePackage: qemu
UpgradeStatus: Upgraded to impish on 2021-10-16 (3 days ago)
dmi.bios.date: 11/12/2020
dmi.bios.release: 4.6
dmi.bios.vendor: Phoenix Technologies LTD
dmi.bios.version: 6.00
dmi.board.name: 440BX Desktop Reference Platform
dmi.board.vendor: Intel Corporation
dmi.board.version: None
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 1
dmi.chassis.vendor: No Enclosure
dmi.chassis.version: N/A
dmi.ec.firmware.release: 0.0
dmi.modalias: dmi:bvnPhoenixTechnologiesLTD:bvr6.00:bd11/12/2020:br4.6:efr0.0:svnVMware,Inc.:pnVMwareVirtualPlatform:pvrNone:sku:rvnIntelCorporation:rn440BXDesktopReferencePlatform:rvrNone:cvnNoEnclosure:ct1:cvrN/A:
dmi.product.name: VMware Virtual Platform
dmi.product.version: None
dmi.sys.vendor: VMware, Inc.

[Paride Legovini (paride)]

Hello and thanks for this bug report. I can confirm this issue, reproduced by:

# With qemu-user-static 1:5.2+dfsg-9ubuntu3.2 (Hirsute):

$ sudo chroot armTest /usr/bin/echo 1 2 3
1 2 3

# With qemu-user-static 1:6.0+dfsg-2expubuntu1 (Impish/Jammy):

$ sudo chroot armTest /usr/bin/echo 1 2 3
2 3

which is clearly wrong. (Note: I didn't copy qemu-arm-static in the chroot, I don't think that's needed.) @Christian: any idea of what could be going on here?

[Andrew L. Moore (slewsys)]

I'm seeing a slight variation of this with
Package: qemu-user-static/impish,now 1:6.0+dfsg-2expubuntu1 amd64
Uname: Linux 5.13.0-21-generic #21-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux
$ sudo chroot armTest ./test.sh a bb ccc dddd
/bin/sh: 0: Can't open a
$ sudo chroot armTest ./test.sh ./test.sh a bb ccc dddd
$0=./test.sh
$1=a
$2=bb
$*=a bb ccc dddd
$ sudo chroot armTest ./test.sh foo a bb ccc dddd
/bin/sh: 0: Can't open foo
$ sudo chroot armTest foo ./test.sh a bb ccc dddd
chroot: failed to run command ‘foo’: No such file or directory

[Andrew L. Moore (slewsys)]

Upgrading to Debian sid version 1:6.1 also appears to resolve the issue:
Package: qemu-user-static/now 1:6.1+dfsg-6 amd64
$ sudo chroot armTest ./test.sh a bb ccc dddd
$0=./test.sh
$1=a
$1=bb
$*=a bb ccc dddd
$

[Christian Ehrhardt  (paelzer)]

bug 1948684 reported almost the same (but for risc in qmeu user static).
Carrying the great Info generated by Björn there over here to handle it in one place.

Steps to reproduce (using golang for ease of cross-compiling):
--
$ cat rv.go
package main

import (
    "fmt"
    "os"
)

func main() {
    fmt.Println("Args are");
    for i, a := range os.Args {
        fmt.Println(i, a);
    }
}
$ go build ./rv.go
$ ./rv one two three
Args are
0 ./rv
1 one
2 two
3 three
$ export GOARCH=riscv64
$ go build ./rv.go
$ file ./rv
./rv: ELF 64-bit LSB executable, UCB RISC-V, version 1 (SYSV),
statically linked, Go
BuildID=9kA-_aeKgZkwtyYVUb7o/F43JNW5XxPX9ScBQrfs8/l6EzXXq8x8pfor-ByIh4/_zELq4T9-VKcpBXoempb,
not stripped
$ ./rv
Args are
0 ./rv
$ ./rv one
Args are
0 ./rv
$ ./rv one two three four
Args are
0 ./rv
1 two
2 three
3 four
$
--

As you can see, in the last run, the "one" argument is skipped.

There is a debian patch, linux-user-binfmt-P.diff (found in qemu_6.0+dfsg-2expubuntu1.debian.tar.xz), that I think is the issue.

When we execute the binfmt-interpreter directly:
$ /usr/libexec/qemu-binfmt/riscv64-binfmt-P /path/to/rv /path/to/rv one two
Args are
0 /path/to/rv
1 one
2 two

Things work, however via the kernels binfmt-misc:
$ /path/to/rv one two
Args are
0 /path/to/rv
1 two

It doesn't.

$ cat /proc/sys/fs/binfmt_misc/qemu-riscv64
enabled
interpreter /usr/libexec/qemu-binfmt/riscv64-binfmt-P
flags: POCF
offset 0
magic 7f454c460201010000000000000000000200f300
mask ffffffffffffff00fffffffffffffffffeffffff

Note that the 'P' flag is set.

From the qemu code (linux-user/main.c):
    /*
     * get binfmt_misc flags
     */
    preserve_argv0 = !!(qemu_getauxval(AT_FLAGS) & AT_FLAGS_PRESERVE_ARGV0);

    /*
     * Manage binfmt-misc preserve-arg[0] flag
     * argv[optind] full path to the binary
     * argv[optind + 1] original argv[0]
     */
    if (optind + 1 < argc && preserve_argv0) {
        optind++;
    }

Here, having P enabled will skip the argument.

I believe the debian patch needs to be reworked (https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg04639.html).

This breaks the ability to run debootstrap/mmdebstap for foreign archs.

[Christian Ehrhardt  (paelzer)]

And the assumption in https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1948684/comments/3 :

Björn Töpel (bjorn-topel) wrote 16 minutes ago: #3
This might have been fixed in Debian. From the changelog:

qemu (1:6.1+dfsg-5) unstable; urgency=medium

  * updated debian/patches/linux-user-binfmt-P.diff
    to work with in-kernel code
    Closes: #993658
  * d/rules: do not mark configure target as .PHONY
    since it is a real file

 -- Michael Tokarev <email address hidden> Mon, 06 Sep 2021 01:20:59 +0300

[Christian Ehrhardt  (paelzer)]

Here as well I want to be honest - I'm almost totally unavailable this week so it will be a few more days before I can take a deeper look.

Thanks Björn for this debugging already!
Thanks for Paride for already confirming the reproducibility of this!
And to Andrew for an equally great report!

[George Yohng (georgeis)]

Just a note: qemu-arm-static itself has nothing to do with it. I tried older versions from hirsute and focal, and they exhibit the same behaviour.

I can however confirm that the problem lies in the binfmt-P logic handling - if I re-register qemu-arm-static manually without the binfmt-P flag, it no longer swallows the first argument.

So my hunch is that whatever part of the kernel/anything passes the arguments to qemu clearly is not binfmt-P aware, while qemu itself is, thus the discrepancy.

[Christian Ehrhardt  (paelzer)]

I have site down and eventually together with paride was able to create a bit of a timeline and a fix.

I have - at first - wondered as I could reproduce it in Jammy, but not in Impish
I had qemu 1:6.0+dfsg-2expubuntu1 in both, why would they behave different.

And then I found that my Impish system was running an outdated kernel.
Rebooting into the newer one revealed that 5.11.0-20 -> 5.13.0-20 has brought the incompatibility.

Gladly the already identified fix (comment #6) applies fine and build in a PPA already.
https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4700

With that I was able to get things working again.

On the way we also found an even simpler testcase.

And finally I also hit the "/bin/sh: 0: cannot open a: No such file" issue, I can confirm that it seems to be the same and fixed by the same.

I'm now preparing an SRU template and merge requests for reviewing this fix.

[Christian Ehrhardt  (paelzer)]

Given this really breaks cross-arch toolchains of all sorts I'm even bumping this to critical.

[Christian Ehrhardt  (paelzer)]

This is already in jammy-proposed by now, but due to a long test backlog on ppc64 will be blocked there for a while.
Nevertheless that is enough to kick off the SRU of this, therefore I uploaded it to Impish-unapproved already.

[Steve Langasek (vorlon)]

Hello Kai, or anyone else affected,

Accepted qemu into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:6.0+dfsg-2expubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Seth Arnold (seth-arnold)]

These test cases seem complicated to me. A reporter on IRC reported that doing things within an rpi image chroot like 'mv a b' would report an error message "missing destination file operand":

Fri 05 23:26:13 < five623468263> root@ubuntu:/# mv etc/resolv.conf etc/resolv.conf.bak
Fri 05 23:26:13 < five623468263> mv: missing destination file operand after 'etc/resolv.conf.bak'
Fri 05 23:26:13 < five623468263> Try 'mv --help' for more information.
Fri 05 23:26:13 < five623468263> Trying to nano resolv.conf just opens a blank editor.

Fri 05 23:43:31 < five623468263> root@ubuntu:/# \mv /etc/resolv.conf /etc/resolv.conf.bak
Fri 05 23:43:32 < five623468263> mv: missing destination file operand after '/etc/resolv.conf.bak'
Fri 05 23:43:32 < five623468263> Try 'mv --help' for more information.
Fri 05 23:48:31 < five623468263> root@ubuntu:/# /bin/echo a b c
Fri 05 23:48:31 < five623468263> b c

Thanks for already doing the heavy lifting on this one. I was dreading trying to track this down and was pretty happy to see a bug report with the exact symptoms listed.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:6.0+dfsg-2expubuntu1.1) for impish have finished running.
The following regressions have been reported in tests triggered by the package:

casper/1.465 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/impish/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Daniel Salzman (salzmdan)]

I have built 1:6.0+dfsg-2expubuntu1.1 on my own for Impish and this patch solves my problems when building ARM packages on x86 using pbuilder and docker-buildx. Thank you!

[Christian Ehrhardt  (paelzer)]

Thanks for the check Daniel!

Hi Seth,
if you consider getting an RPI (or any other) cross-arch image chrooted or to run debootstrap is probably equally complex - and preference lies in what you do more often or already have around. But thanks for the hint on test alternatives.

BTW - I have resolved the reported autopkgtest issue on casper, this is now ready to go once matured enough. Given the natural complexity of this I'm happy for every other testing feedback.

[Christian Ehrhardt  (paelzer)]

Before upgrade

#1 as outlined in the test instructions
ubuntu@i-qemu-static:~$ sudo chroot armTest-I echo 1 2 3
2 3

#2 as suggested by Seth with a chroot into an RPI image
$ wget https://cdimage.ubuntu.com/releases/20.04.3/release/ubuntu-20.04.3-preinstalled-server-arm64+raspi.img.xz?_ga=2.28921023.260729611.1636356089-1983321591.1618404862
$ mv ubuntu-20.04.3-preinstalled-server-arm64+raspi.img.xz\?_ga\=2.28921023.260729611.1636356089-1983321591.1618404862 ubuntu-20.04.3-preinstalled-server-arm64+raspi.img.xz
$ unxz ubuntu-20.04.3-preinstalled-server-arm64+raspi.img.xz
$ sudo partx -a -v ubuntu-20.04.3-preinstalled-server-arm64+raspi.img
$ sudo mount /dev/loop7p2 /mnt/rpi
$ sudo chroot /mnt/rpi

Is it simpler or not, doesn't matter.
It is a valid alternative test :-)

ubuntu@i-qemu-static:~$ sudo chroot /mnt/rpi/ echo 1 2 3
2 3

To make this slightly more interesting I also used it to run RiscV code with the image from https://cdimage.ubuntu.com/releases/20.04.3/release/ubuntu-20.04.3-preinstalled-server-riscv64+unmatched.img.xz

ubuntu@i-qemu-static:~$ sudo chroot /mnt/risc/ echo 1 2 3
2 3

Oh in case anyone wonders, we can check the target arch, but due to the bug need it twice until the bug is fixed
$ sudo chroot /mnt/risc/ dpkg --print-architecture --print-architecture
riscv64
ubuntu@i-qemu-static:~$ sudo chroot /mnt/rpi/ dpkg --print-architecture --print-architecture
arm64
ubuntu@i-qemu-static:~$ sudo chroot armTest-I dpkg --print-architecture --print-architecture
armhf

----
Update

ubuntu@i-qemu-static:~$ sudo apt install qemu-user-static
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  accountsservice language-selector-common libaccountsservice0 net-tools
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  qemu-user-static
1 upgraded, 0 newly installed, 0 to remove and 12 not upgraded.
Need to get 12.2 MB of archives.
After this operation, 23.6 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu impish-proposed/universe amd64 qemu-user-static amd64 1:6.0+dfsg-2expubuntu1.1 [12.2 MB]
Fetched 12.2 MB in 2s (7712 kB/s)
(Reading database ... 95365 files and directories currently installed.)
Preparing to unpack .../qemu-user-static_1%3a6.0+dfsg-2expubuntu1.1_amd64.deb ...
Unpacking qemu-user-static (1:6.0+dfsg-2expubuntu1.1) over (1:6.0+dfsg-2expubuntu1) ...
Setting up qemu-user-static (1:6.0+dfsg-2expubuntu1.1) ...
Processing triggers for man-db (2.9.4-2) ...
Scanning processes...
Scanning linux images...

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

----

ubuntu@i-qemu-static:~$ sudo...

[Christian Ehrhardt  (paelzer)]

FYI - there were two test issues in jammy-proposed, those are also resolved by now and I hope it migrates there soon.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:6.0+dfsg-2expubuntu2

---------------
qemu (1:6.0+dfsg-2expubuntu2) jammy; urgency=medium

  * updated debian/patches/linux-user-binfmt-P.diff to work with in-kernel code
    (#993658) (LP: #1947860)

 -- Christian Ehrhardt <email address hidden> Wed, 03 Nov 2021 14:10:56 +0100

[Steve Langasek (vorlon)]

Verified on impish.

Before upgrade of qemu-user-static:

$ sudo chroot . echo 1 2 3
2 3
$

After upgrade:

$ sudo chroot . echo 1 2 3
1 2 3
$

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:6.0+dfsg-2expubuntu1.1

---------------
qemu (1:6.0+dfsg-2expubuntu1.1) impish; urgency=medium

  * updated debian/patches/linux-user-binfmt-P.diff to work with in-kernel code
    (#993658) (LP: #1947860)

 -- Christian Ehrhardt <email address hidden> Wed, 03 Nov 2021 14:27:05 +0100

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.