systemd-nspawn error: "Failed to mount image file system: Value too large for defined data type"

[Brian Candler]

Two-line reproducer: run this on an Ubuntu 22.04 server.

sudo machinectl pull-raw http://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img jammy-rootfs
sudo machinectl start jammy-rootfs

Response:
Job for <email address hidden> failed because the control process exited with error code.

Here's the relevant output from "journalctl -xeu <email address hidden>"
--------
░░ A start job for unit <email address hidden> has begun execution.
░░
░░ The job identifier is 3223.
Oct 06 13:13:21 brian-kit systemd-nspawn[36720]: Failed to mount image file system: Value too large for defined data type
Oct 06 13:13:21 brian-kit systemd-nspawn[36688]: Short read while reading cgroup mode (0 bytes). The child is most likely dead.
Oct 06 13:13:21 brian-kit systemd[1]: <email address hidden>: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit <email address hidden> has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Oct 06 13:13:21 brian-kit systemd[1]: <email address hidden>: Failed with result 'exit-code'.
--------

This appears to be similar to https://github.com/systemd/systemd/issues/20989

If so, a fix was committed in March 2022: https://github.com/systemd/systemd/pull/22774
but maybe that was too late for 22.04? Or this is a different problem?

EDIT: the reproducer from https://github.com/systemd/systemd/issues/20989#issuecomment-997024152 also reproduces the problem.

# rm -rf /var/lib/machines/my-container/
# mkdir -p /var/lib/machines/my-container/etc
# cp /etc/os-release /var/lib/machines/my-container/etc/
# systemd-nspawn -U -b -D /var/lib/machines/my-container/
Spawning container my-container on /var/lib/machines/my-container.
Press ^] three times within 1s to kill container.
Selected user namespace base 972619776 and range 65536.
Failed to create directory at /var/lib/machines/my-container/usr: Value too large for defined data type

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: systemd-container 249.11-0ubuntu3.10
ProcVersionSignature: Ubuntu 5.15.0-84.93-generic 5.15.116
Uname: Linux 5.15.0-84-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: pass
CloudArchitecture: x86_64
CloudID: none
CloudName: none
CloudPlatform: none
CloudSubPlatform: config
Date: Fri Oct 6 13:10:41 2023
InstallationDate: Installed on 2022-11-04 (335 days ago)
InstallationMedia: Ubuntu-Server 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_GB.UTF-8
 SHELL=/bin/bash
SourcePackage: systemd
UpgradeStatus: No upgrade log present (probably fresh install)

[Brian Candler (b-candler)]

Grr... it works if I first run a dummy command within the machine:

systemd-nspawn -M jammy-rootfs --as-pid2 passwd root
machinectl start jammy-rootfs # it's working now!
machinectl login jammy-rootfs

(Even just "echo hello world" does the job)

However if you go back to a *fresh* image then it fails:

machinectl stop jammy-rootfs
machinectl remove jammy-rootfs
machinectl pull-raw http://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img jammy-rootfs
# Both of these commands consistently fail:
machinectl start jammy-rootfs
systemd-nspawn -M jammy-rootfs -U -b -n

Conclusion: this is confusing. The solution is to manually spawn any command within the image, *before* attempting to start the image with --boot.

[Brian Candler (b-candler)]

I note that the issue *doesn't* occur with 23.10 (which has systemd 253.5), tested using an lxd VM:

$ lxc launch --vm images:ubuntu/23.10/cloud mythic
$ lxc shell mythic
# apt-get install systemd-container
...
# machinectl pull-raw http://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img jammy-rootfs
...
# machinectl start jammy-rootfs
#

[Nick Rosbrook (enr0n)]

I confirmed this issue on 22.04 using the reproducer in the original description. I agree that [1] looks like the appropriate upstream bug. In fact, we currently skip a test in Jammy's autopkgtest due to this issue [2]. Unfortunately, the fix doesn't apply to v249.11 without [3], and I'm not sure that all of those changes are appropriate for an SRU.

[1] https://github.com/systemd/systemd/issues/20989
[2] https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8ca6fbb84ccd09c4e199dec9fd9536c0b3637bc5
[3] https://github.com/systemd/systemd/commit/c0c8f7180023e7c72bf9dd67f1a82d3ea611d445