Error: OCI runtime error: crun: chmod <path>: Operation not supported

[Jacob Godserv]

There appears to be a regression due to a "change in the kernel" in which, as of "commit 5d1f903f75a80daa4dfb3d84e114ec8ecbf29956 in the kernel, present in a release since Linux 6.6 doesn't allow anymore to change the mode of a symlink, so just ignore the failure." Quote source: https://github.com/containers/crun/pull/1309#issue-1908555328

A fix for crun was committed as of v1.9.1, and I believe should be backported: https://github.com/containers/crun/commit/be16ee75ff8574698250352302e9d5496d888d69

I am attaching the apt/history.log file, which I believe reveals the kernel update that happened since February 8th that triggered the regression.

This has been reproducible for me by using podman in combination with the "kind" tool (https://github.com/kubernetes-sigs/kind/releases/tag/v0.21.0 in this case) to attempt to start a Kubernetes cluster within a container. By running "kind create cluster" it is using the container image "kindest/node:v1.29.1" which triggers the error: "Error: OCI runtime error: crun: chmod `run/shm`: Operation not supported"

ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: crun 1.8.5-1
ProcVersionSignature: Ubuntu 6.5.0-17.17-generic 6.5.8
Uname: Linux 6.5.0-17-generic x86_64
NonfreeKernelModules: zfs
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Mon Feb 12 12:07:09 2024
InstallationDate: Installed on 2022-09-19 (511 days ago)
InstallationMedia: Ubuntu 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809.1)
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/usr/bin/zsh
 TERM=screen-256color
 XDG_RUNTIME_DIR=<set>
SourcePackage: crun
UpgradeStatus: Upgraded to mantic on 2023-11-07 (98 days ago)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robert Ross (piexil)]

I am also having this issue.
this is making it impossible to run any containers which use SystemD on kernel 6.5

[sivel (sivel)]

Also experiencing this, OP provided all the correct details as to cause and fix.

[Andre Wagner (wagnerandre85)]

Hi @all,
I was also unable to run systemd with kernel 6.5.0-21-generic, got the exactly the same problem.
I backported https://github.com/containers/crun/commit/be16ee75ff8574698250352302e9d5496d888d69 as a quilt patch for the ubuntu recent version (for ubuntu 22.04 lts) of crun 0.17+dfsg-1.1 (see attachment). After applying the patch, systemd works again as expected.

Grettings,
André

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "ignore_enotsup_when_chmod_a_symlink.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Will Szumski (willjs)]

Is there an ETA for a new package with the bugfix applied?

[Andre Wagner (wagnerandre85)]

Hi @all,
a colleague of mine also experienced the same problem with Ubuntu linux kernel version 5.15.0-101 (on Ubuntu 22.04). The patch I made (ignore_enotsup_when_chmod_a_symlink.patch) fixes also this problem.

Greetings,
André

[Jacob Godserv (fun2program8)]

As a temporary workaround, you can fetch an updated release version of crun from their GitHub and save it into /usr/local/bin.

[Andre Wagner (wagnerandre85)]

Still any news yet?
I want to emphasize that two updates to two Ubuntu LTS versions broke the one of the main features of the preferred container runtime on Ubuntu (over docker). A patch is ready and tested and after six weeks no further reaction...

Greetings,
André