qemu-user-static needs to binfmt with --fix-binary

[Harald Sitter]

[Impact]

qemu static binaries disadvantageously get their binfmt rule created without the fix-binary flag. This breaks cross-architecture dockers, flatpak-builder, chroots and generally most use cases that entail kernel namespaces or chroots. By default the binfmt module will only open the interpreter file (e.g. qemu-arm-static for armhf binaries) when the kernel actually encounters a file for which the interpreter is needed. This is generally fine, except when the root at the time the file is encountered is no longer the root of the system (IOW: if / is in fact /foo on the system then the interpreter will effectively be looked for in /foo/usr/bin/qemu-arm-static, where it of course does not exist). This is generally fine for regular interpreters such as python, which you want to install inside the namespace/chroot if you want to interpret a python file. Since in qemu's case the interpreter is actually a static emulator it really must come from the host system and not the namespace/chroot. No root file system ever comes with an emulator on board, much less one suitable for the host system.

As this effectively breaks qemu-based container-use the missing flag really should get rectified. For example you can't easily cross compile for arm inside a chroot. Primary example of this is flatpak-builder which is used to build flatpak bundles. It has cross compilation capabilities expecting a suitable binfmt setup which will fail with obscure execvp errors (on account of not being able to open the interpreter/qemu binary inside the chroot). https://docs.plasma-mobile.org/AppDevelopment.html#creating-a-flatpak-for-the-phone

The fix-binary flag of binfmt is meant to specifically deal with this. The interpreter file (e.g. qemu-arm-static) is loaded when its binfmt rule is installed instead of when a file that requires it is encountered. When the kernel then encounters a file which requires that interpreter it simply execs the already open file descriptor instead of opening a new one (IOW: the kernel already has the correct file descriptor open, so possibly divergent roots no longer play into finding the interpreter thus allowing namespaces/containers/chroots of a foreign architecture to be run like native ones).

[Test Case]

Do note that once successfully loaded, the interpreter likely won't need opening again. So, testing generally should be done after a cold-boot to ensure binfmt hasn't already loaded the interpreter.

$ sudo apt install docker.io qemu-user-static
$ sudo docker run --rm -it armhf/ubuntu /bin/sh

Without fix-binary this gives an error "exec user process caused 'no such file or directory" (this is referring to the interpreter not being found).
With fix-binary this correctly opens a prompt.

[Regression Potential]

Since the file descriptor is always held open I guess memory consumption goes up by a couple bytes. There isn't any opportunity of regression with fix-binary as it simply changes the interpreter file opening from lazy to instant.
There is potential for the actual SRU patch going wrong, which would manifest in the binfmt rules not getting set up correctly (this would be shown by the aforementioned test case though; breaking the rule setup would result in no improvement to the docker run command).

[Other Info]
I also should point out that the exec() still happens inside the namespace/confinement, so while this essentially loads a file from the host it does not impair security as any confinement restrictions will apply to executable all the same, it is merely the binary data blob that comes from the host.

-----

qemu-user-static in 18.04 invokes update-binfmts without --fix-binary which can break the binfmt emulation when namespaces or chroots are involved. Specifically this defuncts cross compiling when relying on binfmt.

Newer versions already set --fix-binary for the static package. It'd be really grand if this was actually SRU'd though. Currently 18.04's binfmt for arm can inobviously fail with obscure errors. Notable example is bubblewrap and flatpak, which will squarely hit this problem 100% of the time.

[Christian Ehrhardt  (paelzer)]

Hi Harald,
this would be a backport of [1][2] which I'm fine with in general.
But there are few constraints we need to resolve first - and I get the feeling you are more acquainted binfmt than I am - so I'd ask you to help me a bit.

For the SRU policy [3] to apply this must be a bug fix and not a new feature and I'm not entirely sure on that. Furthermore we need a clear test case that can be verified when backported and in bionic-proposed. And finally I'm not deep enough into binfmt through qemu static, it would be great if you could take a step back and try to think "what could break if we do this and outline that".
To make those statements good and convincing is especially important since the affected shar of the Ubuntu community is rather low to start with.

TL;DR:
could you please add statements here that you'd think your be sufficient for the sections in [4]
- impact (included why it is a fix and not a feature)
- regression potential (not "it will be safe" but "what could go wrong")
- test case (a list of commands that would work in Bionic KVM guest for example)

[1]: https://salsa.debian.org/qemu-team/qemu/commit/b1fc77d6ebf848a81f5b874752e5af8e5b93bde8
[2]: https://salsa.debian.org/qemu-team/qemu/commit/6c5584b50a76c167186006c9b7de5ba53e56f5b9
[3]: https://wiki.ubuntu.com/StableReleaseUpdates
[4]: https://wiki.ubuntu.com/StableReleaseUpdates#SRU_Bug_Template

[Christian Ehrhardt  (paelzer)]

FYI - fix is in 2.12 in Cosmic and later, setting tasks accordingly

[Harald Sitter (apachelogger)]

Thanks, I've updated the description accordingly.