systemd-tmpfiles-setup.service fails on btrfs

[Rene Meier]

[Impact]

 * Last security update introduced a regression on btrfs based systems, causing systemd-tmpfiles-setup.service to fail to start, resulting in degraded machines.
 * Cherrypick upstream fixes to resolve this.

[Test Case]

 * Install VM using btrfs for /
 * Boot, check that systemd-tmpfiles-setup.service is started successfully with:
$ systemctl status systemd-tmpfiles-setup.service

[Regression Potential]

 * btrfs fd doesn't support the set of flags that systemd used, with this patch, a compat set of flags is set instead, thus resolving the introduced regression. The worst case scenario is that creating subvolumes/directories is still broken (as in, the current status quo).

[Other Info]

 * Example bad output

After update to systemd 237-3ubuntu10.9 systemd-tmpfiles-setup.service fails with:

Nov 21 13:44:12 node-blc49 systemd[1]: Starting Create Volatile Files and Directories...
Nov 21 13:44:12 node-blc49 systemd-tmpfiles[1226]: Failed to create directory or subvolume "/var": Bad file descriptor
Nov 21 13:44:12 node-blc49 systemd-tmpfiles[1226]: Failed to create directory or subvolume "/home": Bad file descriptor
Nov 21 13:44:12 node-blc49 systemd-tmpfiles[1226]: Failed to create directory or subvolume "/srv": Bad file descriptor
Nov 21 13:44:12 node-blc49 systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Nov 21 13:44:12 node-blc49 systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
Nov 21 13:44:12 node-blc49 systemd[1]: Failed to start Create Volatile Files and Directories.

This happens on btrfs root filesystems in real hardware and on our virtualized servers as well. 237-3ubuntu10.6 didnt show this errors and going back to 237-3ubuntu10 removes them as well.

# lsb_release -rd
Description: Ubuntu 18.04.1 LTS
Release: 18.04

# apt-cache policy systemd
systemd:
  Installiert: 237-3ubuntu10.9
  Installationskandidat: 237-3ubuntu10.9
  Versionstabelle:
 *** 237-3ubuntu10.9 500
        500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     237-3ubuntu10 500
        500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Renato Lorenzi (renatolorenzi)]

I have a similar problem after update from 229-4ubuntu21.2 to 229-4ubuntu21.9 in arm version.

lsb_release -rd
Description: Ubuntu 16.04.4 LTS
Release: 16.04

apt-cache policy systemd

systemd:
  Installed: 229-4ubuntu21.2
  Candidate: 229-4ubuntu21.9
  Version table:
     229-4ubuntu21.9 500
        500 http://ports.ubuntu.com xenial-security/main armhf Packages
        500 http://ports.ubuntu.com xenial-updates/main armhf Packages
 *** 229-4ubuntu21.2 100
        100 /var/lib/dpkg/status
     229-4ubuntu4 500
        500 http://ports.ubuntu.com xenial/main armhf Packages

[Gannet (ken20001)]

The same shit with systemd 237-3ubuntu10.9. How to revert to previous version, could somebody explain?

[SV (smvl)]

After last update some servises failed
systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories
ssh.service loaded failed failed OpenBSD Secure Shell server

If You reboot VPS after update You can't connect via SSH, because SSH service start failed

sshd can be launched via:
1. connect via web console (if VPS)
2. mkdir /run/sshd
3. systemctl start sshd

lsb_release -rd
Description: Ubuntu 16.04.5 LTS
Release: 16.04

apt-cache policy systemd
systemd:
  Installed: 229-4ubuntu21.9
  Candidate: 229-4ubuntu21.9
  Version table:
 *** 229-4ubuntu21.9 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     229-4ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

[crush (sgrosser)]

Same issue here! No SSH-Daemon will start again! Happens after Shutdown, just a Reboot shouldn't affect the System.

received the package via unattended security updates (?)

2018-11-21 06:55:30 upgrade systemd:amd64 229-4ubuntu21.8 229-4ubuntu21.9

No relevance at all with btrfs, we use ext4

[Renato Lorenzi (renatolorenzi)]

Now I received a update from 229-4ubuntu21.9 to 229-4ubuntu21.10 and everything worked again.

[Rene Meier (meier.rene)]

Yes but only because they have reverted the changes from CVE-2018-6954. This will come back very soon. Have a look at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.10 and
https://launchpad.net/bugs/1804847 .

[Gannet (ken20001)]

In Bionic upgrading to systemd 237-3ubuntu10.10 desn't solves this issue.

[Rene Meier (meier.rene)]

I did some tests. This issue is clearly related to btrfs, because I can not reproduce this with ext4. If you have a different filesystem, you have a different issue, maybe the one in https://launchpad.net/bugs/1804847.
How to reproduce this:
1) clean install from 'ubuntu-18.04.1-live-server-amd64.iso', all settings default but change root filesystem to btrfs.
2) system starts fine after reboot
3) issue 'apt install systemd' and get the 'Bad file descriptor' error allready during installation (attached log)
4) reboot with failing systemd-tmpfiles-setup.service

I will also attach a log of the errors during apt install.

[Rene Meier (meier.rene)]

I just checked 18.10 with systemd 239-7ubuntu10.4 and it fails with the same error message.

[Dimitri John Ledkov (xnox)]

yes this is a know regression, not sure where this is tracked. But yes, we do need to fix the tmpfiles on btrfs soon.

[Stanislav Rimsha (slowpokefarm)]

Experiencing same issue on 18.04 desktop with btrfs root filesystem, not being able to boot anything except emergency mode. Is there any workaround to boot into functional system?

[Brian Murray (brian-murray)]

Hello Rene, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Rene Meier (meier.rene)]

Hi Brian,
thank you for providing this fix. It works fine on a minimal 18.10 server. I installed systemd 239-7ubuntu10.5 on a updated system.

[Gannet (ken20001)]

I'm confirming 239-7ubuntu10.5 fixed this issue on my system (Kubuntu 18.10). Please, add the fix to Bionic also. Thanks.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jurit (juritxyz)]

I have server version Ubuntu 18.04.1 LTS bionic systemd 237.
systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories

Problem started some weeks ago after I did apt-get upgrade

[Dirk Schmidtke (dirkschmidtke)]

What about a fix for 18.04 LTS. I am experiencing the same probleme for weeks on BTRFS:

journalctl -b 0 -u systemd-tmpfiles-setup.service
-- Logs begin at Fri 2018-07-06 11:06:42 CEST, end at Thu 2018-12-13 11:48:29 CET. --
Dez 13 11:33:38 Asuspro systemd[1]: Starting Create Volatile Files and Directories...
Dez 13 11:33:38 Asuspro systemd-tmpfiles[831]: Failed to create directory or subvolume "/var": Bad file descriptor
Dez 13 11:33:38 Asuspro systemd-tmpfiles[831]: Failed to create directory or subvolume "/home": Bad file descriptor
Dez 13 11:33:38 Asuspro systemd-tmpfiles[831]: Failed to create directory or subvolume "/srv": Bad file descriptor
Dez 13 11:33:38 Asuspro systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Dez 13 11:33:38 Asuspro systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
Dez 13 11:33:38 Asuspro systemd[1]: Failed to start Create Volatile Files and Directories.
Dez 13 11:33:39 Asuspro systemd[1]: Starting Create Volatile Files and Directories...
Dez 13 11:33:39 Asuspro systemd-tmpfiles[1013]: Failed to create directory or subvolume "/var": Bad file descriptor
Dez 13 11:33:39 Asuspro systemd-tmpfiles[1013]: Failed to create directory or subvolume "/home": Bad file descriptor
Dez 13 11:33:39 Asuspro systemd-tmpfiles[1013]: Failed to create directory or subvolume "/srv": Bad file descriptor
Dez 13 11:33:39 Asuspro systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Dez 13 11:33:39 Asuspro systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
Dez 13 11:33:39 Asuspro systemd[1]: Failed to start Create Volatile Files and Directories.

[fusillator (fusillo)]

Hi, same issue here in bionic 18.04 LTS on btrs subvolumes with the following systemd releases:

 *** 237-3ubuntu10.10 100
        -10 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     237-3ubuntu10.9 500
        500 http://it.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages

How can we apply the patch also in bionic?

Regards

Luca