Install failure for libc6 2.31-0ubuntu6 on armhf

[Matt Thalman]

Attempting to install the latest libc6 package (currently is version 2.31-0ubuntu6) with armhf architecture, results in errors saying "Cannot utime: Operation not permitted".

The following Dockerfile can reproduce this issue:

FROM arm32v7/ubuntu:focal
RUN apt-get update \
    && apt-get install -y libc6

Here's the output of trying to build such a Dockerfile:

Step 1/2 : FROM arm32v7/ubuntu:focal
 ---> 6279a48140d8
Step 2/2 : RUN apt-get update && apt-get install -y libc6
 ---> Running in a647fd988a29
Get:1 http://ports.ubuntu.com/ubuntu-ports focal InRelease [255 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal-updates InRelease [79.7 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports focal-backports InRelease [79.7 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports focal-security InRelease [79.7 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports focal/multiverse armhf Packages [142 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports focal/main armhf Packages [1239 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports focal/restricted armhf Packages [10.8 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports focal/universe armhf Packages [11.0 MB]
Fetched 12.9 MB in 4s (2954 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  gcc-10-base libc-bin libcrypt1 libgcc-s1
Suggested packages:
  manpages glibc-doc locales
The following NEW packages will be installed:
  gcc-10-base libcrypt1 libgcc-s1
The following packages will be upgraded:
  libc-bin libc6
2 upgraded, 3 newly installed, 0 to remove and 55 not upgraded.
Need to get 2770 kB of archives.
After this operation, 618 kB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports focal/main armhf gcc-10-base armhf 10-20200307-0ubuntu1 [18.7 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports focal/main armhf libgcc-s1 armhf 10-20200307-0ubuntu1 [36.2 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports focal/main armhf libcrypt1 armhf 1:4.4.10-10ubuntu4 [93.5 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports focal/main armhf libc6 armhf 2.31-0ubuntu6 [2133 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports focal/main armhf libc-bin armhf 2.31-0ubuntu6 [489 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 2770 kB in 2s (1695 kB/s)
Selecting previously unselected package gcc-10-base:armhf.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 4126 files and directories currently installed.)
Preparing to unpack .../gcc-10-base_10-20200307-0ubuntu1_armhf.deb ...
Unpacking gcc-10-base:armhf (10-20200307-0ubuntu1) ...
Setting up gcc-10-base:armhf (10-20200307-0ubuntu1) ...
Selecting previously unselected package libgcc-s1:armhf.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 4132 files and directories currently installed.)
Preparing to unpack .../libgcc-s1_10-20200307-0ubuntu1_armhf.deb ...
Unpacking libgcc-s1:armhf (10-20200307-0ubuntu1) ...
Replacing files in old package libgcc1:armhf (1:9.2.1-21ubuntu1) ...
Setting up libgcc-s1:armhf (10-20200307-0ubuntu1) ...
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 4134 files and directories currently installed.)
Preparing to unpack .../libc6_2.31-0ubuntu6_armhf.deb ...
debconf: unable to initialize frontend: Dialog
debconf: (TERM is not set, so the dialog frontend is not usable.)
debconf: falling back to frontend: Readline
debconf: unable to initialize frontend: Readline
debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/arm-linux-gnueabihf/perl/5.30.0 /usr/local/share/perl/5.30.0 /usr/lib/arm-linux-gnueabihf/perl5/5.30 /usr/share/perl5 /usr/lib/arm-linux-gnueabihf/perl/5.30 /usr/share/perl/5.30 /usr/local/lib/site_perl /usr/lib/arm-linux-gnueabihf/perl-base) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7.)
debconf: falling back to frontend: Teletype
Checking for services that may need to be restarted...
Checking init scripts...
Checking for services that may need to be restarted...
Checking init scripts...
Nothing to restart.
Unpacking libc6:armhf (2.31-0ubuntu6) over (2.30-0ubuntu3) ...
tar: ./control: Cannot utime: Operation not permitted
tar: ./md5sums: Cannot utime: Operation not permitted
tar: ./shlibs: Cannot utime: Operation not permitted
tar: ./symbols: Cannot utime: Operation not permitted
tar: ./triggers: Cannot utime: Operation not permitted
tar: .: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors
dpkg-deb: error: tar subprocess returned error exit status 2
dpkg: error processing archive /var/cache/apt/archives/libcrypt1_1%3a4.4.10-10ubuntu4_armhf.deb (--unpack):
 dpkg-deb --control subprocess returned error exit status 2
Errors were encountered while processing:
 /var/cache/apt/archives/libcrypt1_1%3a4.4.10-10ubuntu4_armhf.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
The command '/bin/sh -c apt-get update && apt-get install -y libc6' returned a non-zero code: 100

[Thelamer (ryankuba)]

Just to add this has to do with the hosts underlying file system.
On my Odroid with FAT get this, but Qemu on an x86 host and ext4 no problem.

[Sebastiaan van Stijn (thajeztah)]

We ran into the same issue in the containerd packaging scripts (https://github.com/docker/containerd-packaging/pull/151), and it looks like the `Cannot utime: Operation not permitted` is caused by seccomp blocking something.

Now the question is if the default seccomp profile is blocking something that does not have to be blocked, if a different syscall is made when installing this package or if a syscall is made that is not understood by libseccomp (perhaps different syscalls were added for 32-bit platforms).

Not an expert in this area, but thought I'd at least add the info I have at this moment.

[Sebastiaan van Stijn (thajeztah)]

Some further discussion in https://github.com/moby/moby/issues/40734. Possibly one of the new syscalls (added in Linux 5.4) are now being used, but not whitelisted in docker's seccomp profile; https://github.com/moby/moby/issues/40734

[Sebastiaan van Stijn (thajeztah)]

ok; "success" - the problem is solved when installing libseccomp 2.4.3. Unfortunately, that version is not available on Ubuntu versions < 20.03 (https://packages.ubuntu.com/search?keywords=libseccomp2).

So for debugging, I installed the package from the ubuntu 20.03 repository.

What it comes down to (IIUC);

The container we're running (ubuntu:20.03) makes a syscall that's introduced in Linux 5.x, but docker in this case is running on a 4.x kernel (the host is Ubuntu 16.04). The version of libseccomp installed on the host is not taking kernel 5.x syscalls into account, receives an error, and (likely) in that case blocks the syscall, because a whitelist is used.

Solutions for this would be to;

- ask Ubuntu and Debian package maintainers to provide libseccomp 2.4.3 packages for older (LTS) releases. It's a patch release, so possibly acceptable for them. On the other hand; it's adding "features" for a kernel version that's not used by those versions of Ubuntu / Debian.
- somehow make libseccomp handle "unknown" syscalls, and perhaps allow them (instead of blocking)? (not exactly sure how it's handling these, so I'd have to read up on that); probably that's the same (similar) as changing our "whitelist" to a "blacklist" (which could weaken security)

[Florian Weimer (fw)]

> - somehow make libseccomp handle "unknown" syscalls, and perhaps
> allow them (instead of blocking)? (not exactly sure how it's
> handling these, so I'd have to read up on that); probably that's the
> same (similar) as changing our "whitelist" to a "blacklist" (which
> could weaken security)

Blocking not otherwise specified system calls with ENOSYS instead of
EPERM generally has this effect. Some container runtimes incorrectly
use EPERM, though. I don't know if this is the issue with Docker here.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[RHastie (richard-hastie)]

This bug is fully reproducible on a host running a Focal (20.04) armhf kernel 5.4 with Docker also building a container based on Focal.

Specifically, I was able to reproduce this on a RasPi 4B+ using the current Focal image

[Martin (martin-herren)]

Is there any progress on this issue or a workaround available ?

It is really annoying as focal is now out and i'm in the process of releasing new versions of 2 softwares and like to add packages for focal to my repository but this issue prevents my ci/cd pipeline to generates .deb package for focal. Guess i'll let support for focal out for the next releases.

[simone-c (simone-c)]

I can also reproduce it by running i386/ubuntu:focal on a Bionic x64 host:

Unpacking libc6:i386 (2.31-0ubuntu9) over (2.30-0ubuntu2) ...
tar: ./control: Cannot utime: Operation not permitted
tar: ./md5sums: Cannot utime: Operation not permitted
tar: ./shlibs: Cannot utime: Operation not permitted
tar: ./symbols: Cannot utime: Operation not permitted
tar: ./triggers: Cannot utime: Operation not permitted
tar: .: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors
dpkg-deb: error: tar subprocess returned error exit status 2

[Aaron Wright (anwright)]

I can also reproduce this running on armhf (Odroid XU4), Arabian buster host, building a container from Ubuntu 20.04:
Unpacking libc6:armhf (2.31-0ubuntu9) over (2.30-0ubuntu3) ...
tar: ./control: Cannot utime: Operation not permitted
tar: ./md5sums: Cannot utime: Operation not permitted
tar: ./shlibs: Cannot utime: Operation not permitted
tar: ./symbols: Cannot utime: Operation not permitted
tar: ./triggers: Cannot utime: Operation not permitted
tar: .: Cannot utime: Operation not permitted
tar: Exiting with failure status due to previous errors
dpkg-deb: error: tar subprocess returned error exit status 2

[Max (maxried)]

Same here, when trying to build a docker image on Raspberry Pi 2 (armhf) with docker from the ubuntu repo. No special modifications, just the plain OS. Fails on installing libc6.

[Sandro (supersandro2000)]

I have tested this on an Raspberry Pi 3b with libseccomp2 2.4.3-1ubuntu1, kernel 5.4.0-1011-raspi and it still happens.

[Alex DeLorenzo (alex-delorenzo)]

This is a problem on Docker with armhf hosts when building new images using the `ubuntu:focal` image from Docker Hub[1]. Images will fail to build because of the reported error

I've included the error that results from running `docker build` on a Dockerfile referencing `ubuntu:focal` and installing libcrypt1

Images will build using the `ubuntu:rolling` image.

Host details:
Ubuntu 18.04.4 LTS
Docker version 19.03.11, build 42e35e6

[1] Exact image: https://hub.docker.com/layers/ubuntu/library/ubuntu/focal/images/sha256-5747316366b8cc9e3021cd7286f42b2d6d81e3d743e2ab571f55bcd5df788cc8?context=explore

[Dimitri John Ledkov (xnox)]

new libseccomp backport is in progress to be backported.

[Matt Thalman (mthalman)]

Can someone explain the nature of the fix to libseccomp and how that is supposed to resolve this particular issue? Now that the fix in libseccomp - 2.4.3-1ubuntu3.20.04.2 is released, I've tried the same exact repro steps that I described in this issue and the error still exists. I don't believe this is a duplicate of https://bugs.launchpad.net/bugs/1877633.

[Matt Thalman (mthalman)]

My theory is that this issue is fixed only for armhf Docker host, not arm64. I'm running these repro steps on a Jetson TX2 device (arm64) where it is not working. But I've heard from others that are running arm32 hardware where it works for them.

[Matt Thalman (mthalman)]

Just want to give an update on this. I've determined that it isn't solely related to running on an arm64 machine. I've upgraded my Jetson TX2 device from Xenial to Focal and it can now install libc6 fine. I've heard from others that things work on Buster as well. So the issue seems to be either with Xenial in general or Xenial specifically on arm64.