Podman (crun) regression in Ubuntu 22.04: OCI runtime error: chmod `run/shm`: Operation not supported

[Faustin]

[Impact]
podman containers using the default crun backend do not work anymore with the 6.5 HWE kernel in 22.04

[Test plan]
1. Boot jammy with 6.5 HWE kernel (e.g. install linux-virtual-hwe-22.04 in a VM)
2. Make sure you have crun installed, and not runc, because podman can also use runc and this bug is about crun.
3. `podman build -t systemd .` with the Dockerfile:
FROM ubuntu:noble
RUN apt install -U systemd -y
CMD ["/lib/systemd/systemd"]
4. Run it `podman run --systemd always systemd` you should not get
Error: OCI runtime error: chmod `run/shm`: Operation not supported
5. Repeat the above steps on the jammy GA kernel, to make sure we did not regress that use case which is unaffected by this bug.

Optimally submitter can do end-to-end-verification on their side.

[Where problems could occur]
The patch ignores ENOTSUP for fchmodat() in one function, so at most we could silently hide some other issues in fchmodat() inside that function, e.g. AppArmor denials. But generally that is what you would want as a behavior for ENOTSUP...

[Original bug report]

The problem is very well described in https://github.com/actions/runner-images/issues/9425.

## COPY FROM LINK

I think there might be a regression in this release of the ubuntu-22.04 image which breaks podman.[1]

The image updated the kernel from 6.2.y to 6.5.y, but podman/crun don't seem to be updated.

Our build fails with this error link to run:[2]

STEP 1/1: FROM ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac
Trying to pull ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac...
Getting image source signatures
Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1
Copying blob sha256:041b542221cfde2f9fa4ac13f8b5804e25b23ab48ba47db2822c382a134256e1
Copying config sha256:1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5
Writing manifest to image destination
Storing signatures
COMMIT localhost/builder
--> 1eba10d0345
Successfully tagged localhost/builder:latest
Successfully tagged ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac
1eba10d0345cc6df78b7c3a6ced45da9db675d05eb20d5d286996e4f7ffb24d5
Error: OCI runtime error: chmod `run/shm`: Operation not supported
Error: Process completed with exit code 126.

This is with this image version:

Current runner version: '2.313.0'
Operating System
  Ubuntu
  22.04.4
  LTS
Runner Image
  Image: ubuntu-22.04
  Version: 20240225.1.0
  Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240225.1/images/ubuntu/Ubuntu2204-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1

Trying to reproduce, it seems like I am only able to get this image version 20240218.1.0 where the issue does not appear.

Tried to reproduce in this repo[3], but I'm not able to get this with image version 20240225.1.0.

Is this a known issue and version 20240225.1.0 is not in use anymore?

This blog post seems to suggest that the crun version is too old.[4]

[1]https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240225.1
[2]https://github.com/gardenlinux/gardenlinux/actions/runs/8061893233/job/22020522535
[3]https://github.com/fwilhe/turbo-lamp/blob/main/.github/workflows/podman.yml
[4]https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/

[Julian Andres Klode (juliank)]

If you find out which commit(s) in crun need to be cherry-picked and how to reproducibly verify that this fixes the issue, I'm happy to upload them for you.

[John Chittum (jchittum)]

upstream commit in `crun` : https://github.com/containers/crun/pull/1309/commits/57262a2710c83fa08767f0ce3ba7a80993515bb2

I think a trivial reproducer, outside GH runners, is

* start a system with a 6.5 kernel (for a VM downloaded from cloud-images, need to install the HWE kernel. or start on your favourite $BIG_CLOUD as they should have all rolled. Azure is an environment)
* install podman from Ubuntu 22.04 archive
* try to run a container

That _should_ be it, if I'm reading everything correctly.

[Julian Andres Klode (juliank)]

Fix and follow up fix cherry-picked in https://launchpad.net/~juliank/+archive/ubuntu/podman/+packages

[Julian Andres Klode (juliank)]

If you have some time please let me know if that works for you. I tried spinning up a jammy lxd VM but sadly my VM agent crashed or something and I can't access the console due to other bugs so it takes a while to test this on my end - but this is basically the upstream patch applied.

[Julian Andres Klode (juliank)]

I installed podman on a jammy host in Azure and I tried running a default noble image and building an image from whater gardenlinux is (same as in the bug report), but both work fine.

ubuntu@test-crun-20240307:~$ uname -r
6.5.0-1015-azure
ubuntu@test-crun-20240307:~$ podman run --rm -it ubuntu:jammy bash
#Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/shortnames.conf)
Trying to pull docker.io/library/ubuntu:jammy...
Getting image source signatures
Copying blob bccd10f490ab done
Copying config ca2b0f2696 done
Writing manifest to image destination
Storing signatures
root@e7e468635c7e:/#
exit

ubuntu@test-crun-20240307:~$ podman build --squash .
STEP 1/2: FROM ghcr.io/gardenlinux/builder:3ab2bb52bc46bb200c761369c087e9413d1ce0ac
STEP 2/2: RUN echo hello
hello
COMMIT
Getting image source signatures
Copying blob 4a906f38f63e skipped: already exists
Copying blob 60432b4390ff done
Copying config 3f586b4fd7 done
Writing manifest to image destination
Storing signatures
--> 3f586b4fd7e
3f586b4fd7e43d2f13e42a1de376489907c53151f679af1bebf9e2795f013935

[Faustin (fauust)]

EDIT: that was a response to #1, I did not refresh the launchpad page before posting.

I am not sure that this is something easy to find (at least I don't know how to proceed). And above all, my understanding is that the problem seems to be related to the "old" crun version and quite new kernels (6.5).

Also, 22.04 (and Bullseye) comes with crun 0.17 and there is a huge gab with the next release 23.04 that seems to come with crun 1.8-1 (same version in Bookworm).

Anyway, after testing, I can't reproduce the problem on a generic cloud-init image, so that can maybe help?

Here is my test setup:
- Ubuntu 22.04 KVM/Qemu started from generic cloud-init images (amd64);
- kernel upgraded to 6.5.0-1015-azure;
- podman/crun installed from ubuntu repos.

Following command works:
$ podman run --rm -d ghcr.io/fauust/docker-systemd:debian-11

[Julian Andres Klode (juliank)]

The problem we have is that we need to be able to reproduce it so we can verify the change actually fixes the bug, otherwise we can't even upload it.

[Markus Falb (littlecatordog)]

to reproduce create a systemd image

...snip
$ cat - >Containerfile <<EOF
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y systemd
CMD ["/lib/systemd/systemd"]
EOF
podman build --tag cruntest .
snap...

then run it

...snip
$ podman run --systemd always -d cruntest
Error: OCI runtime error: chmod `run/shm`: Operation not supported
snap...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Markus Falb (littlecatordog)]

for the record, the crun from https://launchpad.net/~juliank/+archive/ubuntu/podman/+packages passes my reproducer

Setting up crun (0.17+dfsg-1.1ubuntu0.1)
...

...snip
$ podman run --systemd always -d cruntest
792cc9c24c9c1e936686e63c26befdb67b7e45dd8f9e70c608d164fb49fff7a4
snap...

no error message, compare with my reproducer in comment #8

[Faustin (fauust)]

Hi Markus!

The problem is not which image to use (you can use one of mine, see https://github.com/fauust/docker-systemd/ and #6). Or as explained in #5 by Julian, you can also try to build gardenlinux.

But if your are able to reproduce the bug on your setup that's good, can you please describe your setup then?

Thanks!

[Julian Andres Klode (juliank)]

I can reproduce the issue now, I missed a --systemd always argument. I used

ubuntu@test-crun-20240307:~$ cat Dockerfile
FROM ubuntu:noble
RUN apt install -U systemd -y
ENTRYPOINT /usr/bin/systemd
ubuntu@test-crun-20240307:~$ podman run --systemd always systemd
Error: OCI runtime error: chmod `run/shm`: Operation not supported

[Julian Andres Klode (juliank)]

The success case surprisingly ends up being:

ubuntu@test-crun-20240307:~$ podman run --systemd always systemd
Explicit --user argument required to run as user manager.

But using CMD makes it work. weird stuff

[Julian Andres Klode (juliank)]

Test plan adjusted; fix uploaded.

[Faustin (fauust)]

Hi Julian!

Strange, the GH CI was failing for me without this argument, see https://github.com/fauust/docker-systemd/blob/main/.github/workflows/build.yml#L129-L140 and https://github.com/fauust/docker-systemd/actions/runs/8139697623/job/22386833452.

[Julian Andres Klode (juliank)]

Indeed, the issue was that I tried to use ENTRYPOINT instead of CMD to start systemd.

[Markus Falb (littlecatordog)]

Hi Faustin,

#12 This bug is a little bit about which image to use, because not every image is triggering this bug. First, systemd must be present in the image. Second, some images are known to not trigger it, e.g. Debian older than 12, Fedora, Rocky, the whole RedHat world does not show the behaviour.

#16 I used the --systemd always switch in #8 because it makes my life easier not because it is strictly necessary. If I did not use it I had to specify --tmpfs and volume for /sys/fs/cgroup manually (it does a bunch foo magic in the background), see podman-run(1). But, of course, the complexity grows with rootful vs. rootless or cgroups v1 vs. cgroups v2 and stuff. It's easy to mix issues.

FWIW, here are my references for this bug.

https://noobient.com/2023/11/15/fixing-ubuntu-containers-failing-to-start-with-systemd/
https://github.com/containers/crun/issues/1308
https://github.com/containers/crun/pull/1309

[Faustin (fauust)]

Hi Markus!
Indeed not all images are impacted, here is a non exhaustive list (alpine is particular since it use OpenRC):
https://github.com/fauust/docker-systemd/actions/runs/8139697623

My problem was that I was not able to reproduce it, see #6.

Anyway, I am a bit lost here but apparently some of you could reproduce the bug and more importantly a fix is in the pipe. Hopefully it will hit next GH runner images soon so that I will not have to modify all my CI depending on it.

The current workaround in my case (forcing arguments) is a bit more complex since there is another layer above podman (molecule) and I don't want to spend time fixing it if the fix works, see: https://github.com/fauust/ansible-role-mariadb/actions.

In any case, thank you all for your reactivity on this!

[Faustin (fauust)]

Ok, in #6 I tested with debian 11 which is not impacted, re-testing...

Ok all good, I can reproduce it too:

$ uname -r
6.5.0-1015-azure
$ podman run --rm -d ghcr.io/fauust/docker-systemd:debian-12
Error: OCI runtime error: chmod `run/shm`: Operation not supported

[Robie Basak (racb)]

> [Test plan]
> Needs to be finalized...

Is this finalized yet please?

[Andreas Hasenack (ahasenack)]

Echoing Robie's question, and marking task as incomplete pending a test plan update.

[Julian Andres Klode (juliank)]

Sorry, I just did not fix the first line when editing the test plan

[Andreas Hasenack (ahasenack)]

I see the same code in mantic's crun package, and the problem can also be reproduced there:

$ uname -a
Linux nsnx2 6.5.0-25-generic #25-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 7 14:58:39 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

$ grep VERSION_CODENAME /etc/os-release
VERSION_CODENAME=mantic

$ podman run --systemd always systemd
Error: OCI runtime error: crun: chmod `run/shm`: Operation not supported

I'm going to add a mantic task to this bug, and either expect a mantic upload to fix it, or clarification as to why mantic will not be fixed.

[Andreas Hasenack (ahasenack)]

I'm also updating the test case to make sure crun is installed, because runc can also satisfy podman's dependency on an OCI runtime.

[Andreas Hasenack (ahasenack)]

I updated the test case one more time to request a run with the jammy GA kernel, to make sure we haven't regressed that (unaffected) use case.

I'm accepting the jammy upload to get things going, but the mantic question from comment #24 will need to be addressed before the jammy fix can hit updates. If mantic will get an update as well (preferable), then please amend the test case to also include mantic.

Note that the jammy upload is deleting debian/.gitlab-ci.yml. This change is not needed, nor mentioned in d/changelog. Since it doesn't affect the build, I'm not blocking on it, but it would be cool to watch out for such changes in the future :)

[Andreas Hasenack (ahasenack)]

Hello Faustin, or anyone else affected,

Accepted crun into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/crun/0.17+dfsg-1.1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Markus Falb (littlecatordog)]

I installed the package

...snip
# dpkg -l crun
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-======================-============-==============================================
ii crun 0.17+dfsg-1.1ubuntu0.1 amd64 lightweight OCI runtime for running containers
snap...

and run the the test as in #8:

...snip
# cat - >Containerfile <<EOF
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y systemd
CMD ["/lib/systemd/systemd"]
EOF
podman build --tag cruntest .
...
snap...

...snip
# podman run --systemd always -d cruntest
92a5f9832d1c8e6a1ceeb18f78e9252fb9a8fcda6dd875581498d4391b99211e
snap...

No error. So I would say it looks good.

[Markus Falb (littlecatordog)]

I also tried the package in some CI runs on github. Looks Good.
I changed verification-needed-jammy to verification-done-jammy, however, I would appreciate if another one affected can confirm that it works for him too. Don't know about tag verification-needed, so I did not touch it.

[Andreas Hasenack (ahasenack)]

This is blocked on a mantic upload fixing the same issue.

[Andreas Hasenack (ahasenack)]

> # dpkg -l crun

Markus, thanks for the verification, but crun can be installed together with runc, and it's not clear which one will end up being used. That's why it's best to also make sure runc is *not* installed.

[Markus Falb (littlecatordog)]

Yeah, but the default is crun. I could have modified the config file of course.
But I have not:

...snip
# podman run --systemd always -d cruntest
231947c35e1d58cfb3d1ca37e52b6324288af515e3ce6c25a0873a721a213f1b
# podman run --systemd always --runtime crun -d cruntest
e655a24b5c11d7bd572942533db4b95fe53cbd0338c35b76c5f0129e78f806e9
# podman inspect 231947c35e1d | jq .[0].OCIRuntime
"crun"
# podman inspect e655a24b5c11 | jq .[0].OCIRuntime
"crun"
snap...

[Julian Andres Klode (juliank)]

jammy to mantic is not a particular interesting upgrade path, it's opt-in only, so if nobody is interested in a mantic fix we shouldn't block the jammy LTS fix for it. jammy users will be upgrading to noble and not be affected by a mantic regression.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andreas Hasenack (ahasenack)]

Marking mantic as "won't fix" according to comment #33 above.

[Andreas Hasenack (ahasenack)]

There was no clear indication which running kernel was used during the verifications, so I did it:

Using the Dockerfile from the test plan:
FROM ubuntu:noble
RUN apt install -U systemd -y
CMD ["/lib/systemd/systemd"]

And building with "podman build -t systemd ."

a) 5.15.0-105 kernel (jammy GA kernel)
crun 0.17+dfsg-1.1 from jammy fails (odd, I thought it was unaffected):

$ podman run --systemd always systemd
Error: OCI runtime error: chmod `run/shm`: Operation not supported

crun 0.17+dfsg-1.1ubuntu0.1 from jammy-proposed works:

$ podman run --systemd always systemd
(nothing here)

In a separate terminal we can see it's running:
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b2decb3d2820 localhost/systemd:latest /lib/systemd/syst... 34 seconds ago Up 34 seconds ago flamboyant_hertz

b) Jammy HWE kernel
$ uname -r
6.5.0-34-generic

crun 0.17+dfsg-1.1 from jammy fails:
$ podman run --systemd always systemd
Error: OCI runtime error: chmod `run/shm`: Operation not supported

crun 0.17+dfsg-1.1ubuntu0.1 from jammy-proposed works:
$ podman run --systemd always systemd
(nothing here)

In another terminal:
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
867d113be7a3 localhost/systemd:latest /lib/systemd/syst... 2 seconds ago Up 2 seconds ago confident_mirzakhani

[Launchpad Janitor (janitor)]

This bug was fixed in the package crun - 0.17+dfsg-1.1ubuntu0.1

---------------
crun (0.17+dfsg-1.1ubuntu0.1) jammy; urgency=medium

  * lp2056442-ignore-enotsup-when-chmod-symlink.patch: Ignore ENOTSUP
    when chmod a symlink (LP: #2056442)

 -- Julian Andres Klode <email address hidden> Thu, 07 Mar 2024 14:17:06 +0100

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for crun has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.