test -x fails inside shell scripts in containers

[Florian Lohoff]

(SRU template for systemd)

[impact]

bash (and some other shells) builtin test command -x operation fails

[test case]

on any affected host system, start nspawn container, e.g.:

$ sudo apt install systemd-container
$ wget https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-amd64-root.tar.xz
$ mkdir h
$ cd h
$ sudo tar xvf ../hirsute-server-cloudimg-amd64-root.tar.xz
$ sudo systemd-nspawn

Then from a bash shell, verify if test -x works:

root@h:~# ls -l /usr/bin/gpg
-rwxr-xr-x 1 1000 1000 1083472 Jan 16 09:53 /usr/bin/gpg
root@h:~# test -x /usr/bin/gpg || echo "fail"
fail

[regression potential]

any regression would likely occur during a syscall, most likely faccessat2(), or during other syscalls.

[scope]

this is needed for b/f

this is fixed upstream by commit bcf08acbffdee0d6360d3c31d268e73d0623e5dc which is in 247 and later, so this is fixed in h

this was pulled into Debian at version 246.2 in commit e80c5e5371ab77792bae94e0f8c5e85a4237e6eb, so this is fixed in g

in x, the entire systemd seccomp code is completely different and the patch doesn't apply, nor does it appear to be needed, as the problem doesn't reproduce in a h container under x.

[other info]

this needs fixing in libseccomp as well

[original description]

glibc regression causes test -x to fail inside scripts inside docker/podman, dash and bash are broken, mksh and zsh are fine:

root@0df2ce5d7a46:/# test -x /usr/bin/gpg || echo Fail
root@0df2ce5d7a46:/# dash -c "test -x /usr/bin/gpg || echo Fail"
Fail
root@0df2ce5d7a46:/# bash -c "test -x /usr/bin/gpg || echo Fail"
Fail
root@0df2ce5d7a46:/# mksh -c "test -x /usr/bin/gpg || echo Fail"
root@0df2ce5d7a46:/# zsh -c "test -x /usr/bin/gpg || echo Fail"
root@0df2ce5d7a46:/#

root@0df2ce5d7a46:/# zsh -c "[ -x /usr/bin/gpg ] || echo Fail"
root@0df2ce5d7a46:/# mksh -c "[ -x /usr/bin/gpg ] || echo Fail"
root@0df2ce5d7a46:/# dash -c "[ -x /usr/bin/gpg ] || echo Fail"
Fail
root@0df2ce5d7a46:/# bash -c "[ -x /usr/bin/gpg ] || echo Fail"
Fail

The -f flag works, as does /usr/bin/test:
# bash -c "test -f /usr/bin/gpg || echo Fail"
# bash -c "/usr/bin/test -x /usr/bin/gpg || echo Fail"
#

[Original bug report]
root@84b750e443f8:/# lsb_release -rd
Description: Ubuntu Hirsute Hippo (development branch)
Release: 21.04
root@84b750e443f8:/# dpkg -l gnupg apt
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-===============-============-==========================================
ii apt 2.1.20 amd64 commandline package manager
ii gnupg 2.2.20-1ubuntu2 all GNU privacy guard - a free PGP replacement

Hi,
for 3 days our CI pipelines to recreate Docker images fails for the Hirsute images. From comparison this seems to be caused by apt 2.1.20.

The build fails with:

0E: gnupg, gnupg2 and unupg1 do not seem to be installed, but one of them is required for this operation

The simple Dockerfile to reproduce the error - "docker build -t foo ."

FROM amd64/ubuntu:hirsute
MAINTAINER Florian Lohoff <email address hidden>

USER root

RUN apt-get update \
 && DEBIAN_FRONTEND=noninteractive apt-get -y install curl gnupg apt \
  && curl https://syncthing.net/release-key.txt | apt-key add -

Breaking it down it this seems to be an issue that there is new functionality in apt/apt-key e.g. security hardening that docker prohibits in its containers. Running this manually works only in an --privileged container.

So adding keys in unpriviledged container or possibly kubernetes will not work anymore.

Flo

[Florian Lohoff (fl0l0)]

Bug also applies to apt 2.2.0

root@72aa01291622:/# dpkg -l apt gnupg
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-===============-============-==========================================
ii apt 2.2.0 amd64 commandline package manager
ii gnupg 2.2.20-1ubuntu2 all GNU privacy guard - a free PGP replacement

root@72aa01291622:/# curl https://syncthing.net/release-key.txt | apt-key add -
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0E: gnupg, gnupg2 and gnupg1 do not seem to be installed, but one of them is required for this operation
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0

100 2462 100 2462 0 0 1969 0 0:00:01 0:00:01 --:--:-- 1969

[Julian Andres Klode (juliank)]

This is a feature, not a bug. We worked hard to not require gnupg anymore. apt-key does not work without gnupg (or gpg). It's also deprecated and has been obsoleted over 10 years ago with the introduction of trusted.gpg.d.

The correct thing to do here is to

wget -O /etc/apt/trusted.gpg.d/syncthing.asc https://syncthing.net/release-key.txt

or

wget -O <other path>/syncthing.asc https://syncthing.net/release-key.txt

and add signed-by=<other path>/syncthing.asc to your sources.list entry.

[Julian Andres Klode (juliank)]

And no, this is not new in 2.1.20. The gnupg dependency was demoted to Suggests in 1.5~alpha1 (Jun 2017), so it's been this way since 17.10.

[Julian Andres Klode (juliank)]

OK; I did not see that gpg was actually installed. Please report bugs properly with ubuntu-bug/apport next time instead of dumping data in the text report.

[Julian Andres Klode (juliank)]

This is a regression in glibc: [ -x /usr/bin/gpg] fails inside the script.

Downgrading libc6 (and rdeps) to 2.32-0ubuntu6 makes it work again.
Upgrading libc6 to 2.33-0ubuntu2 breaks it.

[Julian Andres Klode (juliank)]

Broken bash:

faccessat2(AT_FDCWD, "/usr/bin/gpg", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)

Good bash w/ old glibc:

faccessat(AT_FDCWD, "/usr/bin/gpg", X_OK) = 0

Good mksh:

newfstatat(AT_FDCWD, "/usr/bin/gpg", {st_mode=S_IFREG|0755, st_size=1083472, ...}, 0) = 0

[Julian Andres Klode (juliank)]

Potentially a seccomp confinement issue in podman and docker?

[Florian Weimer (fweimer)]

Yes, it's a seccomp issue that needs to be fixed on the container host.

There's a generic kludge here: https://github.com/opencontainers/runc/pull/2750

Recent docker/podman version be okay as well, but the fix (logically, not explicitly) depends on other package updates too (e.g., libseccomp).

Most container hosts on offer allow disabling seccomp filters in some way, but that's not entirely standardized.

[Julian Andres Klode (juliank)]

Fixed in libseccomp2 2.5.1

[Julian Andres Klode (juliank)]

The other question is whether the change in glibc should be rolled back such that it works when invoked in older container hosts.

[Florian Weimer (fweimer)]

Patches have been proposed for that, but were rejected:

[PATCH] syscalls: Document OCI seccomp filter interactions & workaround
https://<email address hidden>/

[RFC PATCH] Linux: Add seccomp probing to faccessat2
https://sourceware.org/pipermail/libc-alpha/2020-November/119955.html

We *really* need to clean this up properly, so that we are prepared if we need to add a new system call as part of a security fix.

[Christian Ehrhardt  (paelzer)]

Julian said in comment #9 that 2.5.1 would be good.
But then Florians comment #11 does not make me feel so sure.
In any case we now have:
 libseccomp | 2.5.1-1ubuntu1 | hirsute | source

Does that mean we are good now?

Subscribing Alex who did the 2.5.1 upload ...

[Martin Pitt (pitti)]

I've been scratching my head over this regression [1] for a while now, in the context of running a hirsute container on a 20.04 host (in particular, a GitHub workflow machine) In my case, the symptom is that after upgrading glibc, `which` is broken; that of course also uses faccessat(), similar to test -x.

I tried all sorts of the "usual" workarounds, as seccomp has been giving trouble for a while now [2]. But this failure is robust against fuse-overlayfs vs. vfs (inefficient full copies of the file system), root vs. user podman, podman vs. docker, and, relevant for this bug, it *also happens* with --security-opt=seccomp=unconfined and/org --privileged, both of which should disable seccomp.

Hence I believe this bug can't at least only be in libseccomp.

[1] https://github.com/martinpitt/umockdev/runs/1984769591?check_suite_focus=true#step:3:1019
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1900021

[Florian Weimer (fweimer)]

To paper over the faccessat2 issue, a libseccomp update is enough *if* the container runtime already knows about the faccessat2 system call and mentions it in its profiles. But with the current design, every new system call will need similar updates to several components (not just libseccomp) just to produce the ENOSYS error, which is not a sustainable development model. Hence the runc pull request mentioned in comment 8.

[Alex Murray (alexmurray)]

As I understand it I don't see there is any issue here with libseccomp in Ubuntu as it currently stands - whilst the aforementioned runc workaround commit description specifies a number of shortcomings with libseccomp and the inability to easily handle and distinguish newly added syscalls between it and glibc etc, until there is some more generic mechanism for either libseccomp policy authors, or libseccomp itself, to easily identify what syscalls are supported by a given system and therefore whether the generated policy is sufficient to enumerate these, there is no obvious "fix" for libseccomp itself.

[Balint Reczey (rbalint)]

Following all the discussions fixing the container runtimes seems to be the way out of this.
For runc https://github.com/opencontainers/runc/pull/2750 should be SRUd to all releases.

[Balint Reczey (rbalint)]

Running "test -x ..." also fails in systemd-nspawn for systemd < 247, I think only the following patch needs to be SRU-d to earlier systemd versions:
https://github.com/systemd/systemd/commit/bcf08acbffdee0d6360d3c31d268e73d0623e5dc

[Oded Arbel (oded-geek)]

Seems to still be an issue with docker on groovy/focal/[...]. Any plans to backport the new seccomp to those?

[Alex Murray (alexmurray)]

@oded-geek - yes, the libseccomp SRU to backport 2.5.1 to these releases is being handled in https://bugs.launchpad.net/ubuntu/+source/libseccomp/+bug/1891810

[Launchpad Janitor (janitor)]

This bug was fixed in the package runc - 1.0.0~rc93-0ubuntu1

---------------
runc (1.0.0~rc93-0ubuntu1) hirsute; urgency=medium

  * New upstream release (LP: #1919182).
    - runc now has special handling for seccomp profiles to avoid making new
      syscalls unusable for glibc (LP: #1916485).
  * Remove patch addressing a bug fixed by upstream:
    - debian/patches/test--fix_TestGetAdditionalGroups.patch
  * Refresh patch:
    - debian/patches/test--skip-fs-related-cgroups-test.patch
  * Backport upstream patch to fix patchpbf test on armhf:
    - debian/patches/fix-patchpbf-test-on-32-bit.patch

 -- Lucas Kanashiro <email address hidden> Tue, 10 Mar 2021 09:30:36 -0300

[Łukasz Zemczak (sil2100)]

Hello Florian, or anyone else affected,

Accepted runc into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc93-0ubuntu1~20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Florian, or anyone else affected,

Accepted runc into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc93-0ubuntu1~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Florian, or anyone else affected,

Accepted runc into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/runc/1.0.0~rc93-0ubuntu1~18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted runc (1.0.0~rc93-0ubuntu1~18.04.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

opengcs/0.3.4+dfsg2-0ubuntu3 (s390x, armhf, amd64, i386, ppc64el, arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#runc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted runc (1.0.0~rc93-0ubuntu1~20.04.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

golang-github-docker-go-connections/0.4.0-1 (arm64, armhf, ppc64el, amd64, s390x)
opengcs/0.3.4+dfsg2-0ubuntu3 (amd64, s390x, arm64, armhf, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#runc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted runc (1.0.0~rc93-0ubuntu1~20.10.1) for groovy have finished running.
The following regressions have been reported in tests triggered by the package:

opengcs/0.3.4+dfsg2-0ubuntu4 (amd64, ppc64el, s390x, arm64, armhf)
golang-github-docker-go-connections/0.4.0-1 (s390x, arm64, ppc64el, amd64, armhf)
libpod/2.0.6+dfsg1-1ubuntu1 (amd64, ppc64el, s390x, arm64, armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/groovy/update_excuses.html#runc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Stiverton (andrewstiver)]

Will this be fixed in the 21.04 tag docker image as well? I am having the problem there also.

[Łukasz Zemczak (sil2100)]

Hello Florian, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Florian, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.46 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (237-3ubuntu10.46) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.4/5.4.0-71.79~18.04.1 (i386)
polkit-qt-1/unknown (i386)
openssh/1:7.6p1-4ubuntu0.3 (ppc64el, arm64, s390x, i386, amd64, armhf)
systemd/237-3ubuntu10.46 (amd64)
linux-hwe-5.0/5.0.0-65.71 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.6) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

multipath-tools/0.8.3-1ubuntu2 (s390x)
munin/2.0.56-1ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

root@lp1916485-f:~# dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.5 amd64 system and service manager

root@lp1916485-f:~# wget https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-amd64-root.tar.xz
...
root@lp1916485-f:~# mkdir h
root@lp1916485-f:~# cd h
root@lp1916485-f:~/h# tar xvf ../hirsute-server-cloudimg-amd64-root.tar.xz
...
root@lp1916485-f:~/h# systemd-nspawn
Spawning container h on /root/h.
Press ^] three times within 1s to kill container.
root@h:~# ls -l /usr/bin/gpg
-rwxr-xr-x 1 root root 1083472 Feb 22 09:33 /usr/bin/gpg
root@h:~# test -x /usr/bin/gpg || echo "fail"
fail

root@lp1916485-f:~/h# dpkg -l libseccomp2|grep libseccomp
ii libseccomp2:amd64 2.5.1-1ubuntu1~20.04.1 amd64 high level interface to Linux seccomp filter
root@lp1916485-f:~/h# dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.6 amd64 system and service manager
root@lp1916485-f:~/h# systemd-nspawn
Spawning container h on /root/h.
Press ^] three times within 1s to kill container.
root@h:~# ls -l /usr/bin/gpg
-rwxr-xr-x 1 root root 1083472 Feb 22 09:33 /usr/bin/gpg
root@h:~# test -x /usr/bin/gpg || echo "fail"
root@h:~#

[Dan Streetman (ddstreet)]

marking verification-done-focal for systemd (I don't see any verification steps listed for runc)

[Dan Streetman (ddstreet)]

root@lp1916485-b:~# wget https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-amd64-root.tar.xz
...
root@lp1916485-b:~# mkdir h
root@lp1916485-b:~# cd h
root@lp1916485-b:~/h# tar xvf ../hirsute-server-cloudimg-amd64-root.tar.xz
...

root@lp1916485-b:~/h# dpkg -l systemd|grep systemd
ii systemd 237-3ubuntu10.45 amd64 system and service manager
root@lp1916485-b:~/h# dpkg -l libseccomp2|grep seccomp
ii libseccomp2:amd64 2.4.3-1ubuntu3.18.04.3 amd64 high level interface to Linux seccomp filter
root@lp1916485-b:~/h# systemd-nspawn
Spawning container h on /root/h.
Press ^] three times within 1s to kill container.
root@h:~# ls -l /usr/bin/gpg
-rwxr-xr-x 1 root root 1083472 Feb 22 09:33 /usr/bin/gpg
root@h:~# test -x /usr/bin/gpg || echo "fail"
fail

root@lp1916485-b:~/h# dpkg -l systemd|grep systemd
ii systemd 237-3ubuntu10.46 amd64 system and service manager
root@lp1916485-b:~/h# dpkg -l libseccomp2|grep seccomp
ii libseccomp2:amd64 2.5.1-1ubuntu1~18.04.1 amd64 high level interface to Linux seccomp filter
root@lp1916485-b:~/h# systemd-nspawn
Spawning container h on /root/h.
Press ^] three times within 1s to kill container.
root@h:~# ls -l /usr/bin/gpg
-rwxr-xr-x 1 root root 1083472 Feb 22 09:33 /usr/bin/gpg
root@h:~# test -x /usr/bin/gpg || echo "fail"
root@h:~#

[Dan Streetman (ddstreet)]

marking verification-done-bionic for systemd (I don't see any verification steps listed for runc)

[Dan Streetman (ddstreet)]

@kanashiro I believe you uploaded runc, can you perform verification for it

[Lucas Kanashiro (lucaskanashiro)]

To verify runc I am launching a docker container and calling "test -x" like was done for systemd.

Groovy
======

ubuntu@docker-groovy:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.10 (Groovy Gorilla)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.10"
VERSION_ID="20.10"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=groovy
UBUNTU_CODENAME=groovy
ubuntu@docker-groovy:~$ dpkg -l | grep runc
ii runc 1.0.0~rc93-0ubuntu1~20.10.1 amd64 Open Container Project - runtime
ubuntu@docker-groovy:~$ docker run -it ubuntu:focal /bin/bash
Unable to find image 'ubuntu:focal' locally
focal: Pulling from library/ubuntu
a70d879fa598: Pull complete
c4394a92d1f8: Pull complete
10e6159c56c0: Pull complete
Digest: sha256:3c9c713e0979e9bd6061ed52ac1e9e1f246c9495aa063619d9d695fb8039aa1f
Status: Downloaded newer image for ubuntu:focal
root@7fa381c4877b:/# which ls
/usr/bin/ls
root@7fa381c4877b:/# ls -l /usr/bin/ls
-rwxr-xr-x 1 root root 142144 Sep 5 2019 /usr/bin/ls
root@7fa381c4877b:/# test -x /usr/bin/ls || echo "fail"
root@7fa381c4877b:/#

Focal
=====

ubuntu@docker-focal:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="20.04.2 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.2 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
ubuntu@docker-focal:~$ dpkg -l | grep runc
ii runc 1.0.0~rc93-0ubuntu1~20.04.1 amd64 Open Container Project - runtime
ubuntu@docker-focal:~$ sudo docker run -it ubuntu:focal /bin/bash
Unable to find image 'ubuntu:focal' locally
focal: Pulling from library/ubuntu
a70d879fa598: Pull complete
c4394a92d1f8: Pull complete
10e6159c56c0: Pull complete
Digest: sha256:3c9c713e0979e9bd6061ed52ac1e9e1f246c9495aa063619d9d695fb8039aa1f
Status: Downloaded newer image for ubuntu:focal
root@bf6b6e1534e5:/# which ls
/usr/bin/ls
root@bf6b6e1534e5:/# test -x /usr/bin/ls || echo "fail"
root@bf6b6e1534e5:/#

Bionic
======

ubuntu@docker-bionic:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
ubuntu@docker-bionic:~$ dpkg -l | grep runc
ii runc 1.0.0~rc93-0ubuntu1~18.04.1 amd64 Open Container Project - runtime
ubuntu@docker-bionic:~$ sudo docker run -it ubuntu:focal /bin/bash
Unable to find image 'ubuntu:focal' locally
focal: Pulling from library/ubuntu
a70d879fa598: Pull complete
c4394a92d1f8: Pull complete
10e6159c56c0: ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 245.4-4ubuntu3.6

---------------
systemd (245.4-4ubuntu3.6) focal; urgency=medium

  * debian/patches/lp1916485-Newer-Glibc-use-faccessat2-to-implement-faccessat.patch:
    Add support for faccessat2 (LP: #1916485)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=affb2c6507dccfeed02820a2267639648e2a2260
  * d/p/lp1918696-shared-seccomp-util-address-family-filtering-is-brok.patch:
    Stop attempting to restrict address families on ppc archs
    (LP: #1918696)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=daff4b6604362fcb5d305682216d5ca15a4c5738
  * d/p/lp1891810-seccomp-util-add-new-syscalls-from-kernel-5.6-to-sys.patch:
    Add openat2() syscall to seccomp filter list
    (LP: #1891810)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=69c8a684e2513b2f6530e5a5cf15c83abfb7bc74
  * d/p/lp1915887-Downgrade-a-couple-of-warnings-to-debug.patch:
    Downgrade some log messages so they stop spamming logs
    (LP: #1915887)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3c2c4731b90ed430ca1790270e69cd125643b94b
  * d/p/lp1887744-basic-unit-file-when-loading-linked-unit-files-use-l.patch:
    Use src name, not dst name, of symlinked unit files (LP: #1887744)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=03770601097cfdc09adeadf5593083da69345409

 -- Dan Streetman <email address hidden> Wed, 17 Mar 2021 17:36:08 -0400

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Łukasz Zemczak (sil2100)]

I see verification of runc in comment #37 for all series - but groovy isn't marked as verified. Is there anything missing in the verification? Or can we release runc for groovy safely?

[Lucas Kanashiro (lucaskanashiro)]

Sorry, I forgot to update the tags. Nothing is missing in the runc verification, we can release it.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 237-3ubuntu10.46

---------------
systemd (237-3ubuntu10.46) bionic; urgency=medium

  * d/p/lp1916485-Newer-Glibc-use-faccessat2-to-implement-faccessat.patch:
    Add support for faccessat2 (LP: #1916485)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b5f11a9baecf0cefb503632e938d473234172128
  * d/p/lp1918696-shared-seccomp-util-address-family-filtering-is-brok.patch:
    Stop attempting to restrict address families on ppc archs
    (LP: #1918696)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4569a047ece8b1b300ef63e49b5aea8aba35c500
  * d/p/lp1891810-seccomp-util-add-new-syscalls-from-kernel-5.6-to-sys.patch:
    Add openat2() syscall to seccomp filter list
    (LP: #1891810)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=2ddfbfa79af4f22b7adf946c4299433fd74a4f17

 -- Dan Streetman <email address hidden> Wed, 17 Mar 2021 17:38:05 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package runc - 1.0.0~rc93-0ubuntu1~20.10.1

---------------
runc (1.0.0~rc93-0ubuntu1~20.10.1) groovy; urgency=medium

  * Backport version 1.0.0~rc93-0ubuntu1 from Hirsute (LP: #1919322,
    LP: #1916485).

 -- Lucas Kanashiro <email address hidden> Tue, 16 Mar 2021 15:23:05 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package runc - 1.0.0~rc93-0ubuntu1~20.04.1

---------------
runc (1.0.0~rc93-0ubuntu1~20.04.1) focal; urgency=medium

  * Backport version 1.0.0~rc93-0ubuntu1 from Hirsute (LP: #1919322,
    LP: #1916485).

 -- Lucas Kanashiro <email address hidden> Tue, 16 Mar 2021 15:34:35 -0300

[Launchpad Janitor (janitor)]

This bug was fixed in the package runc - 1.0.0~rc93-0ubuntu1~18.04.1

---------------
runc (1.0.0~rc93-0ubuntu1~18.04.1) bionic; urgency=medium

  * Backport version 1.0.0~rc93-0ubuntu1 from Hirsute (LP: #1919322,
    LP: #1916485).
    - Use Go 1.13 to build it, with the default Go 1.10 it FTBFS.
      + d/control: b-d on golang-1.13-go instead of golang-any,
      + d/rules: add Go 1.13 to the $PATH.
    - d/rules: set GOCACHE to a temporary directory. dh-golang sets it to
      "off" which is not accepted by Go >= 1.12.
    - d/rules: set GO111MODULE to "off" to avoid getting modules info online.

 -- Lucas Kanashiro <email address hidden> Tue, 16 Mar 2021 15:47:19 -0300

[Sergio Durigan Junior (sergiodj)]

Before I change the status of this bug, I would like to report my findings here.

I am testing things on a Bionic s390x machine with everything up-to-date:

# apt policy systemd
systemd:
  Installed: 237-3ubuntu10.46
...
# apt policy containerd
containerd:
  Installed: 1.4.4-0ubuntu1~18.04.2
...
# apt policy docker.io
docker.io:
  Installed: 20.10.2-0ubuntu1~18.04.2
...
# apt policy runc
runc:
  Installed: 1.0.0~rc93-0ubuntu1~18.04.1
...

Following the reproduction steps listed in the Description section still fail for me:

# systemd-nspawn
Spawning container h on /root/h.
Press ^] three times within 1s to kill container.
# bash -c 'test -x /usr/bin/gpg || echo Fail'
Fail

When I'm in a hirsute Docker container, it also fails:

$ docker run -it --rm ubuntu:hirsute
root@78506947b11f:/# bash -c 'test -x /usr/bin/gpg || echo Fail'
Fail

This is impacting the build of the 21.04 OCI images on s390x (amd64, arm64 and ppc64el succeed).

I'm still not sure what's causing this, nor why this is happening only on s390x. I will post more details when I have them.

[Christian Ehrhardt  (paelzer)]

Guest is Hirsute to have
  libc6:s390x 2.33-0ubuntu5 s390x

The following (not optimized for speed but readability) gives us a simple environment-matrix for comparisons:

for r in xenial bionic focal groovy hirsute; do
    uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=s390x label=daily release=${r};
    uvt-kvm create --host-passthrough --password=ubuntu ${r}-test-nspawn release=${r} arch=s390x label=daily;
    uvt-kvm wait ${r}-test-nspawn
    uvt-kvm ssh ${r}-test-nspawn "sudo apt update"
    uvt-kvm ssh ${r}-test-nspawn "sudo apt upgrade -y"
    uvt-kvm ssh ${r}-test-nspawn "wget https://cloud-images.ubuntu.com/hirsute/current/hirsute-server-cloudimg-s390x-root.tar.xz"
    uvt-kvm ssh ${r}-test-nspawn "mkdir h"
    uvt-kvm ssh ${r}-test-nspawn "sudo tar -xzf hirsute-server-cloudimg-s390x-root.tar.xz -C h";
    uvt-kvm ssh ${r}-test-nspawn "sudo apt install -y systemd-container"
    uvt-kvm ssh ${r}-test-nspawn "sudo reboot";
    sleep 5s
    uvt-kvm wait ${r}-test-nspawn
done

Test (as before) is:
$ cd h
$ sudo systemd-nspawn
$ bash -c 'test -x /usr/bin/gpg || echo Fail'

Out of that I can confirm (and further limit releases to just bionic) that on s390x we have:
Xenial - 4.4.0-210/229-4ubuntu21.31 - works
Bionic - 4.15.0-142/237-3ubuntu10.46 - fails
Focal - 5.4.0-72/245.4-4ubuntu3.6 - works
Groovy - 5.8.0-50/246.6-1ubuntu1.3 - works
Hirsute - 5.11.0-16/247.3-3ubuntu3 - works

Next I split the Bionic case to for a few usual suspects (kernel/systemd/glibc):
Already on step #1 Kernel I found something:
Bionic - 4.15.0-142/237-3ubuntu10.46 - fails
Bionic - 5.4.0-72/237-3ubuntu10.46 - works

Ok so the new kernel fixes it (whatever it is) so what about things in proposed already:
There is a new 4.15 kernel and a new systemd
Bionic - 4.15.0-143/237-3ubuntu10.47 - fails

So none of the builds in proposed has the fix, but something between 4.15 and 5.4 kernels fixes it. That might also be the reason why the other releases are fine - the kernel levels are either >=5.4 (fixed) or <4.15 (not having the issue).

[Dan Streetman (ddstreet)]

Problem seems to be on s390x with the 4.15 kernel, faccessat2() still returns EPERM:

faccessat2(AT_FDCWD, "/bin/bash", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)

while on amd64 it now returns ENOSYS which is correctly handled with fallback to faccessat()

faccessat2(AT_FDCWD, "/bin/bash", X_OK, AT_EACCESS) = -1 ENOSYS (Function not implemented)
faccessat(AT_FDCWD, "/bin/bash", X_OK) = 0

(note both traces were of bash running 'test -x /bin/bash')

[Dan Streetman (ddstreet)]

Interestingly, faccessat2() does correctly return ENOSYS when using a simple chroot instead of systemd-nspawn:

ubuntu@test-s390x:~/h$ sudo systemd-nspawn
Spawning container h on /home/ubuntu/h.
Press ^] three times within 1s to kill container.
root@h:~# test -x /bin/bash || echo "fail"
fail
root@h:~# exit
logout
Container h exited successfully.
ubuntu@test-s390x:~/h$ sudo chroot .
root@test-s390x:/# test -x /bin/bash || echo "fail"
root@test-s390x:/#

and to verify, stracing bash:

faccessat2(AT_FDCWD, "/bin/bash", X_OK, AT_EACCESS) = -1 ENOSYS (Function not implemented)
faccessat(AT_FDCWD, "/bin/bash", X_OK) = 0

[Dan Streetman (ddstreet)]

It seems the s390x failure on bionic is fixed by the patch(es) for bug 1895132. I'm not 100% sure why it is returning the normal ENOSYS for invalid syscalls except while running under systemd-nspawn, but it might be due to the different syscall entry path taken when _TIF_TRACE is set, and that does get set if _TIF_SECCOMP is set.

In any case, I'll send the kernel patches to the ML.

[Dan Streetman (ddstreet)]

https://launchpad.net/~ddstreet/+archive/ubuntu/lp1916485

in case anyone wants to test with the patched kernel

[Sergio Durigan Junior (sergiodj)]

Thanks for the investigation, Dan. I tested the Linux package from your PPA on a s390x machine and can confirm that it does solve the issue.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-04-22 07:23 EDT-------
Hi,
I've also run some tests on s390x:

On Ubuntu 18.04 (bionic) with libseccomp2 2.5.1-1ubuntu1~18.04.1 and linux 4.15.0.142.129:
In a systemd-nspawn hirsute container:
bash -c "test -x /bin/bash"
returns 1 and strace shows:
faccessat2(AT_FDCWD, "/bin/bash", X_OK, AT_EACCESS) = -1 EPERM (Operation not permitted)
and seccomp-tools dump showed that the seccomp-filter allows faccessat2(=0x1b7) syscall!

After upgrading to 4.15.0-143-generic #147+hf1916485v20210421b1 from your PPA, the command works fine:
faccessat2(AT_FDCWD, "/bin/bash", X_OK, AT_EACCESS) = -1 ENOSYS (Function not implemented)
faccessat(AT_FDCWD, "/bin/bash", X_OK) = 0

In a "docker run -it ubuntu:hirsute /bin/bash" container (runc-1.0.0~rc93-0ubuntu1~18.04.1), the command also works fine and the seccomp-filter applied by docker also allows faccessat2.

On Ubuntu 20.10 (groovy) with libseccomp2 2.5.1-1ubuntu1~20.10.1 and linux 5.8.0.51.56, both the tests in systemd-nspawn and in docker container are working fine and the dump of the seccomp-filter shows that faccessat2 is allowed.
(On the same system before updating libseccomp2 to the mentioned version, libseccomp2 2.4.3-1ubuntu4 was used. There the dump of the seccomp-filter showed that the faccessat2 syscall was not allowed and thus the test command failed.)

Thanks.

[Sergio Durigan Junior (sergiodj)]

Dan, let me know if you need help driving the Linux kernel SRU forward. Thanks!

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2021-05-03 10:12 EDT-------
@Canonical, any updates available here? Many thx in adbvance

[Sergio Durigan Junior (sergiodj)]

Hello! The kernel team has applied the fix to their pre-release branch. They have a 5-week release cycle, so we should be seeing a new Bionic Linux kernel with the fix in the following 3-4 weeks. Thanks.

[Brian Murray (brian-murray)]

The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release

[Matt Thalman (mthalman)]

According to https://stackoverflow.com/questions/66319610/gpg-error-in-ubuntu-21-04-after-second-apt-get-update-during-docker-build, this bug fix is supposed to fix the issue of getting the following error when running "apt-get update" in an Ubuntu 21.04 container: "W: GPG error: http://ports.ubuntu.com/ubuntu-ports hirsute InRelease: gpgv, gpgv2 or gpgv1 required for verification, but neither seems installed".

I was running into this error when attempting to build my Dockerfiles based on arm64v8/ubuntu:21.04 and arm32v7/ubuntu:21.04. After upgrading my runc version to 1.0.1, the error went away but only for arm64v8/ubuntu:21.04. The Dockerfile based on arm32v7/ubuntu:21.04 still encountered the error. In both cases, I am running the build on an AArch64 device, so it's using emulation for the arm32v7/ubuntu:21.04 scenario. It would appear that it's still broken for that scenario?

The repro is very simple, just run the following command on an AArch64 device: "docker run --rm arm32v7/ubuntu:21.04 apt-get update". It will output the following:

Unable to find image 'arm32v7/ubuntu:21.04' locally
21.04: Pulling from arm32v7/ubuntu
48989deb32eb: Pulling fs layer
48989deb32eb: Verifying Checksum
48989deb32eb: Download complete
48989deb32eb: Pull complete
Digest: sha256:b61c1421a092dd4ffc0b14a6b683513d775d5daa275598c74cd34090a0424a19
Status: Downloaded newer image for arm32v7/ubuntu:21.04
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Get:1 http://ports.ubuntu.com/ubuntu-ports hirsute InRelease [269 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports hirsute-updates InRelease [115 kB]
Err:1 http://ports.ubuntu.com/ubuntu-ports hirsute InRelease
  gpgv, gpgv2 or gpgv1 required for verification, but neither seems installed
Get:3 http://ports.ubuntu.com/ubuntu-ports hirsute-backports InRelease [101 kB]
Err:2 http://ports.ubuntu.com/ubuntu-ports hirsute-updates InRelease
  gpgv, gpgv2 or gpgv1 required for verification, but neither seems installed
Get:4 http://ports.ubuntu.com/ubuntu-ports hirsute-security InRelease [110 kB]
Err:3 http://ports.ubuntu.com/ubuntu-ports hirsute-backports InRelease
  gpgv, gpgv2 or gpgv1 required for verification, but neither seems installed
Err:4 http://ports.ubuntu.com/ubuntu-ports hirsute-security InRelease
  gpgv, gpgv2 or gpgv1 required for verification, but neither seems installed
Reading package lists...
W: GPG error: http://ports.ubuntu.com/ubuntu-ports hirsute InRelease: gpgv, gpgv2 or gpgv1 required for verification, but neither seems installed
E: The repository 'http://ports.ubuntu.com/ubuntu-ports hirsute InRelease' is not signed.
W: GPG error: http://ports.ubuntu.com/ubuntu-ports hirsute-updates InRelease: gpgv, gpgv2 or gpgv1 required for verification, but neither seems installed
E: The repository 'http://ports.ubuntu.com/ubuntu-ports hirsute-updates InRelease' is not signed.
W: GPG error: http://ports.ubuntu.com/ubuntu-ports hirsute-backports InRelease: gpgv, gpgv2 or gpgv1 required for verification, but neither seems installe...

[Michael Hudson-Doyle (mwhudson)]

Are using the docker packages from the Ubuntu archive? It doesn't quite look like it but I'm not completely sure how to tell.

[Sergio Durigan Junior (sergiodj)]

On Tuesday, September 21 2021, Matt Thalman wrote:

> Client:
> Version: 20.10.7
> API version: 1.41
> Go version: go1.16.4
> Git commit: f0df35096d5f5e6b559b42c7fde6c65a2909f7c5
> Built: Sat Sep 11 15:09:09 2021
> OS/Arch: linux/arm64
> Context: default
> Experimental: true
>
> Server: Docker Engine - Community
> Engine:
> Version: 20.10.8
> API version: 1.41 (minimum version 1.12)
> Go version: go1.16.6
> Git commit: 75249d8
> Built: Fri Jul 30 19:53:13 2021
> OS/Arch: linux/arm64
> Experimental: false
> containerd:
> Version: 1.4.9
> GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
> runc:
> Version: 1.0.1
> GitCommit: v1.0.1-0-g4144b63
> docker-init:
> Version: 0.19.0
> GitCommit: de40ad0

I don't have time to try to reproduce right now, but as mwhudson said it
doesn't look like you're using the Ubuntu docker.io package. The first
thing that caught my attention is the Go version used to build the
package: we use Go 1.13, whereas you used Go 1.16. The other suspicious
thing is the GitCommit field, which should contain our tags (for example
"20.10.7-0ubuntu1~20.04.1" on Focal).

Are you using Ubuntu on the host? Perhaps your comment was made because
you're experiencing this error with the Ubuntu docker image, but bear in
mind that this bug is about the docker.io/runc/containerd packages that
run on the Ubuntu host.

Thanks,

--
Sergio
GPG key ID: E92F D0B3 6B14 F1F4 D8E0 EB2F 106D A1C8 C3CB BF14

[Sergio Durigan Junior (sergiodj)]

I took the liberty to clean up this bug and mark things as Invalid/Fix Released as needed. Hopefully I got everything right, but feel free to reopen/re-classify a task if there's something wrong.

Thanks.

[Mathew Hodson (mhodson)]

Fixed in Hirsute, Focal, Groovy, Bionic, and Xenial with libseccomp 2.5.1-1ubuntu1