systemd defaulting to tmpfs for /tmp causes enospc errors

[Paride Legovini]

[Impact]

systemd 256 changes behavior and makes /tmp a tmpfs by default. This causes some
problems with packages for which the sources weight more than the tmpfs size,
and this case often happen on the testbed because they usually don't have a lot
of RAM by default (see test case below for a concrete example).

[Test case]

(Untested yet, as it seems the current systemd in proposed is causing network
issues too, and this breaks with an unrelated problem)

autopkgtest-buildvm-ubuntu-cloud -a amd64 -r oracular
autopkgtest -BU libreoffice --apt-pocket=proposed=src:systemd -- qemu autopkgtest-oracular-amd64.img

Since the default RAM given to the QEMU VM is 2GB and the libreoffice source
take more than 4GB, this should have no trouble running in a "No space left on
device" error.

[Fix]

https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/362
This was release in autopkgtest 5.36, currently in the process of being SRU'd to
Jammy and Noble in bug #2071609.

[Regression potential]

The current SRU has a lot of unrelated changes. While this increases the
chances of regression, this also come with a broader test suite, increasing test
coverage, and so regressions should only happen in corner-cases, hopefully less
important as the current lack of this /tmp bug fix.

[Original bug report]

Debian systemd now mounts a tmpfs on /tmp by default; relevant d/changelog entry:

systemd (256~rc3-3) unstable; urgency=medium

   * Make /tmp/ a tmpfs by default. Restore the upstream default and make
    /tmp/ a tmpfs. Can be overridden with: touch
    /etc/systemd/system/tmp.mount or: systemctl mask tmp.mount

Ubuntu is aligning to this, however this means that available space under /tmp is limited to the half of the available ram. That is sometimes not enough to unpack source trees in /tmp, see for example [1].

We need to to decide how to deal with this. Possibilities I can think about:

1. Disable the tmpfs tmp.mount, as suggested in the d/changelog entry. To be done in setup-testbed. Good: easy. Bad: deviates from the Ubuntu defaults, requires rebuilding the images.

2. Make autopkgtest use /var/tmp. Good: we stick to the defaults: Bad: requires non trivial changes in src:autopkgtest, which makes assumptions on stuff being in /tmp, on /tmp being cleared on reboots, ...

This under the assumption that the switch to a tmpfs has been discussed, and we want it in Ubuntu.

[1] https://autopkgtest.ubuntu.com/results/autopkgtest-oracular/oracular/ppc64el/c/ceph/20240618_194813_d6efc@/log.gz

[Nick Rosbrook (enr0n)]

I think option (2) is the probably the "best" fix. Another alternative I can think of is shipping a tmp.mount drop-in on the autopkgtest environment to change the size to something better. E.g.,

# /etc/systemd/system/tmp.mount.d/size.conf
[Mount]
Options=size=<param>

where <param> could be a percentage of RAM (e.g. 75%%), or an explicit size (e.g. 4GB). But then again, these defaults are intentionally easy to override. If the autopkgtest environment wants to do something different, I don't think it's the end of the world, and masking tmp.mount as a first step is fine.

> This under the assumption that the switch to a tmpfs has been discussed, and we want it in Ubuntu.

This was discussed in the foundations team, and the initial consensus was to not deviate from upstream and Debian.

[Paride Legovini (paride)]

This is affecting testing of the kernel packages (linux / linux-*), as those are tested with all-proposed (they're a special case in the autopkgtest-cloud worker script), and thus the new systemd is pulled in.

[Paride Legovini (paride)]

This implements option (1): https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/362

[Nick Rosbrook (enr0n)]

My understanding is that we will leave the systemd defaults as-is for now, so marking won't fix for systemd.

[Nick Rosbrook (enr0n)]

Actually, re-opening for systemd for now so it shows up on update-excuse.

[Launchpad Janitor (janitor)]

This bug was fixed in the package autopkgtest - 5.37ubuntu2

---------------
autopkgtest (5.37ubuntu2) oracular; urgency=medium

  * New delta, all upstream cherry-picks:
    - t/autopkgtest-build-docker: use ports.ubuntu.com on non-amd64 Ubuntu
    - t/autopkgtest: test_git_source_build: skip if remote repo is unreachable
    - d/t/lxd: restrict architectures via Architecture and enable arm64
    - d/rules: don't run pre-commit at package build time

 -- Paride Legovini <email address hidden> Wed, 26 Jun 2024 13:16:53 +0200

[Timo Aaltonen (tjaalton)]

Hello Paride, or anyone else affected,

Accepted autopkgtest into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/5.37ubuntu2~24.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Paride, or anyone else affected,

Accepted autopkgtest into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/5.37ubuntu2~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Paride, or anyone else affected,

Accepted autopkgtest into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/5.37ubuntu3~24.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Paride, or anyone else affected,

Accepted autopkgtest into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/autopkgtest/5.37ubuntu3~22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted autopkgtest (5.37ubuntu3~24.04.1) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

surf/2.1+git20221016-6build2 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#autopkgtest

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!