Tar fails to extract archives that include folders with certain permissions on armhf

[Logan Bussell]

Thank you @loganbussell-msft for the bug report!

[Impact]

Currently running containers using modern versions of glibc such as the one available in noble on older hosts causes permissions issues inside the container. This is due to newer versions of glibc expecting the fchmodat2 syscall to be available and to return ENOSYS in case it is not. However docker seccomp profile defaults to returning EPERM for all non defined syscalls and writing an entry for fchmodat2 in the docker seccomp profile to return ENOSYS does not work on systems where libseccomp does not have support for fchmodat2.

Running armhf noble docker containers on arm64 jammy hosts has been seen to exhibit this behavior and a patch to libseccomp for jammy is required to fix the issue.

Other architectures may also be affected by this issue that such as ppc64le as reported by @mark-elvers.

I have backported a fix from upstream that adds the missing syscalls to libseccomp and verified it on an ampere arm machine as well as on a raspberry pi 4

[Test Plan]

1- On an ARM 64 machine install the latest version of docker on a jammy host by following the official docker documentation. [https://docs.docker.com/engine/install/ubuntu/]

2- Create an armhf noble docker container:
$ docker run --rm -it --platform linux/arm/v7 --entrypoint bash ubuntu.azurecr.io/ubuntu:noble

3- inside the docker container execute the following commands to create a new tar file and then extract it:

mkdir /test \
    && chmod 775 /test \
    && cd /test \
    && mkdir 775 \
    && chmod 775 775 \
    && touch 775/test.txt \
    && chmod 644 775/test.txt \
    && tar -czvf /test.tar.gz .

mkdir -p /test2 \
    && tar -tzvf /test.tar.gz \
    && tar -oxzf /test.tar.gz -C /test2

4- you will see the following errors:

tar: ./775: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: Exiting with failure status due to previous errors

5- When libseccomp is patched the command will run with no permission issues

[Where problems could occur]

* the issue might still occur on other platforms
* if using an older version of docker the issue will still occur

[Original Description]
When running Ubuntu Noble in an arm32 Docker container, on certain hosts (Azure VM CI agents), tar fails to extract certain archives that include folders with specific permissions set.

Here's a concise repro. The error occurs in when building the Dockerfile. I can only get this to work on Azure VMs, but can't find out why.

```Dockerfile
FROM ubuntu.azurecr.io/ubuntu:noble

# Create the problematic archive
RUN mkdir /test \
    && chmod 775 /test \
    && cd /test \
    && mkdir 775 \
    && chmod 775 775 \
    && touch 775/test.txt \
    && chmod 644 775/test.txt \
    && tar -czvf /test.tar.gz .

# Extracting it gives an error
RUN mkdir -p /test2 \
    && tar -tzvf /test.tar.gz \
    && tar -oxzf /test.tar.gz -C /test2
```

What I expected to happen: The test.tar.gz archive should be successfully extracted to the /test2 directory.

What happened instead: Tar throws the following error:
```
tar: ./775: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: Exiting with failure status due to previous errors
```

The Ubuntu container is running as root so there shouldn't be any permission errors.

Since this is running in a container, I observed this happening on the following kernel:
`Linux version 5.15.148.2-2.cm2 (root@CBL-Mariner) (gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37) #1 SMP Fri Feb 23 23:38:33 UTC 2024`
As well as
`Linux <hostname> 6.5.0-1017-azure #17~22.04.1-Ubuntu SMP Sat Mar 9 10:04:07 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux`

I was not able to reproduce it using Ubuntu 22.04 Jammy (ubuntu.azurecr.io/ubuntu:jammy), using the same kernel as above.

Additionally I was not able to reproduce this on the kernel `Linux cb0507859b24 5.15.146.1-microsoft-standard-WSL2 #1 SMP Thu Jan 11 04:09:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux`, which is running on my work machine, using Docker qemu emulation for the arm32 image.

Ubuntu version: Ubuntu Noble Numbat (development branch) 24.04 (from ubuntu.azurecr.io/ubuntu:noble)
tar version: `1.35+dfsg-3`

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mark Elvers (mark-elvers)]

This also affects ppc64le Docker images. These commands work fine on x86_64, arm64 and s390 but fail on POWER9.

```
docker run -it --rm ubuntu:noble
apt-get -y update
apt install -y wget
cd /tmp
wget a-tar-file-of-your-choice.tar.gz
tar -xzf a-tar-file-of-your-choice.tar.gz
```

Error message:

...
tar: your/file.1: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: your/file.2: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: your/file.3: Cannot change mode to rwxrwxr-x: Operation not permitted
tar: Exiting with failure status due to previous errors
```

[Mark Elvers (mark-elvers)]

If you compile tar from scratch within the Docker container, then you do not see the error.

```
wget https://ftp.gnu.org/gnu/tar/tar-1.35.tar.gz
tar -xzf tar-1.35.tar.gz
```

Ignore the errors from the tar process :-)

```
apt install build-essential libacl1-dev -y
cd tar-1.35
FORCE_UNSAFE_CONFIGURE=1 ./configure --prefix=/usr
make install
```

Now `tar -xf` works as expected.

[Mark Elvers (mark-elvers)]

I did some analysis [here](https://github.com/ocaml/infrastructure/issues/121).

libseccomp needs to be >= 2.55 and Docker >= 25.0.3 and then this issue goes away. Without these the system call `fchmodat2` return EPERM rather than `ENOSYS`.

[Ghadi Rahme (ghadi-rahme)]

Hello anyone affected,

I have written a patch for jammy for libseccomp to fix the bug.
Thank you @mark-elvers for confirming that ppc64le is also affected

[Paride Legovini (paride)]

Uploaded:

Uploading libseccomp_2.5.3-2ubuntu2.1.dsc
Uploading libseccomp_2.5.3.orig.tar.gz
Uploading libseccomp_2.5.3-2ubuntu2.1.debian.tar.xz
Uploading libseccomp_2.5.3-2ubuntu2.1_source.buildinfo
Uploading libseccomp_2.5.3-2ubuntu2.1_source.changes

[Robie Basak (racb)]

Are we sure this issue doesn't affect 24.04? This bug was filed on 27 March, a day after libseccomp 2.5.5-1ubuntu1 was published into the Noble release pocket. It looks like this was first fixed upstream in 2.5.5. The original reporter (understandably) did not report the version of the libseccomp package used. After fixing the Test Plan as I've requested below, please run that against 24.04 to make sure, and report the results.

Presumably this bug isn't currently fixed in 23.10 since 2.5.4-1ubuntu3 is current there. This is a prerequisite to fixing 22.04. Please see https://wiki.ubuntu.com/StableReleaseUpdates#Newer_Releases.

> On an ARM 64 machine install the latest version of docker on a jammy host by following the official docker documentation. [https://docs.docker.com/engine/install/ubuntu/]

Please update the Test Plan to use Docker as shipped by Ubuntu. Or, if this issue only affects a version of Docker external to the Ubuntu archive, then we might still be able to fix libseccomp for you, but it changes the calculus considerably. Please document the specifics, and tie it to a specific version that has specific behaviour (what, exactly?) rather than the moving target of "latest".

> However docker seccomp profile defaults to returning EPERM for all non defined syscalls and writing an entry for fchmodat2 in the docker seccomp profile to return ENOSYS does not work on systems where libseccomp does not have support for fchmodat2.

Will libseccomp allow Docker to return ENOSYS for all non defined syscalls instead? This seems like it would be the correct general fix to me, to avoid future breakages of the same class. Is this possible, and how would doing this instead change the risk to 22.04 in this fix? If this issue only affects an upstream version of Docker, can that be fixed there please, rather than risking regression to all other libseccomp consumers by working around this in libseccomp in 22.04? Then I think your "use the latest version of upstream Docker" would just start working?

The proposed patch looks like it adds system call numbers for fchmodat2 for all architectures, as well as for a bunch of other system calls. The architecture-specific changes seem to apply to cacheflush and memfd_secret system calls only. So is this an armhf-specific problem, or is it a general one that affects all architectures?

Following "the requirements for stable updates are not necessarily the same as those in the development release" from https://wiki.ubuntu.com/StableReleaseUpdates#Why, why is this SRU patch not minimal, adding the definition for the required system call for the affected architectures only?

Or, if the problem we're solving is wider than this and requires the full patch, then I think we need to include that in our regression analysis please, together with a Test Plan that exercises relevant code paths.

In particular, I'm concerned that the proposed change will affect much more than the bug being fixed, increasing regression risk including to other consumers of libseccomp, and that these additional cases aren't covered by the Test Plan. Many more system calls could suddenly "start working" from the perspective of glibc, rather th...

[Mark Elvers (mark-elvers)]

I confirm that this also affects Noble.

If libseccomp2 is >= 2.55, then Docker must be >= 25.0.3.

I looked at fixing the Docker profile, and this works for `docker run`, but `docker build` always uses the build-in/default profile, so it's a limited workaround.

[Robie Basak (racb)]

> I confirm that this also affects Noble.

Thanks! Oracular and Noble are at the same version of libseccomp then, so I'll mark those tasks as still open.

> ...but `docker build` always uses the build-in/default profile, so it's a limited workaround.

Can Docker be fixed upstream instead, such that it works correctly with older libseccomp versions?

The instructions from the Test Plan led me to https://download.docker.com/linux/ubuntu/dists/noble/pool/stable/amd64/docker-ce-cli_26.1.4-1~ubuntu.24.04~noble_amd64.deb, which ships /usr/bin/docker, but that dynamic executable only uses libc. How is it accessing libseccomp dynamically, if at all? Is is using dlopen?

[Mark Elvers (mark-elvers)]

Presumably via /usr/bin/runc.

```
# ldd /usr/bin/runc
 linux-vdso.so.1 (0x0000003940e63000)
 libseccomp.so.2 => /lib/riscv64-linux-gnu/libseccomp.so.2 (0x0000003940e3a000)
 libc.so.6 => /lib/riscv64-linux-gnu/libc.so.6 (0x0000003940cba000)
 /lib/ld-linux-riscv64-lp64d.so.1 (0x0000003940e65000)
```

[Ghadi Rahme (ghadi-rahme)]

Hello Robie and anyone else affected,

> Are we sure this issue doesn't affect 24.04?

24.04 is does have the patched version of libseccomp and is not affected by the issue when using the upstream version of docker. I have tested it on an ARM machine based on the test plan I wrote and it's not affected.

> Presumably this bug isn't currently fixed in 23.10 since 2.5.4-1ubuntu3 is current there.

Correct! I skipped Mantic thinking that since it is close to EOL it wouldn't be worth it but since it is a hard requirement I will back-port the patch to it as well.

> Please update the Test Plan to use Docker as shipped by Ubuntu. Or, if this issue only affects a version of Docker external to the Ubuntu archive, then we might still be able to fix libseccomp for you...

The docker version shipped by ubuntu (docker.io) is affected by the issue on all ubuntu releases (including noble). The fix needed is the following: https://github.com/moby/moby/pull/47341/files and https://github.com/containerd/containerd/commit/a6e52c74fa043a63d7dae4ac6998215f6c1bb6ac
Which as @mark-elvers pointed out is available since docker 25.0.3
The thing is that all this patch does is change the default seccomp profile used by docker (which can be done by the user through the argument: --security-opt seccomp=/path/to/seccomp/profile.json in docker).
Of course I understand that this does mean that docker.io will still need to be updated even if a workaround is possible. But the reason I am stating this is because before back-porting this patch to docker.io, the libseccomp patches will need to be back-ported and made available first. Noble and oracular being the only exceptions to this rule since they already have a libseccomp version that is aware of fchmodat2.
I assume the method to follow would be to get the libseccomp patch in all the ubuntu releases, verify that the patch works with the upstream version of docker, then patch docker.io on these versions and verify that docker.io works?

> Will libseccomp allow Docker to return ENOSYS for all non defined syscalls instead? This seems like it would be the correct general fix to me, to avoid future breakages of the same class.

Sadly no. The default behavior in the seccomp profile is that all syscalls that have no rules defined to them in the docker seccomp profile will return EPERM. This applies to syscalls that libseccomp is aware of as well as those of which it is not aware of. The return value can be modified but it cannot be limited to syscalls not defined in libseccomp.

> Is this possible, and how would doing this instead change the risk to 22.04 in this fix?

Current applications have been running fine in docker with EPERM as the return value for denied syscalls, changing this default return to ENOSYS will affect any application that makes use of these syscalls that get denied. The docker seccomp profile denies syscalls based on this default return value and for the few other syscalls that need to get denied with a different value it has exceptions for them written in the default seccomp profile. Changing this default will require a rewrite to this profile and will make the majority of syscalls that get denied with E...