systemd-tmpfiles-setup.service fails on btrfs

[Rene Meier]

[Impact]

 * Last security update introduced a regression on btrfs based systems, causing systemd-tmpfiles-setup.service to fail to start, resulting in degraded machines.
 * Cherrypick upstream fixes to resolve this.

[Test Case]

 * Install VM using btrfs for /
 * Boot, check that systemd-tmpfiles-setup.service is started successfully with:
$ systemctl status systemd-tmpfiles-setup.service

[Regression Potential]

 * btrfs fd doesn't support the set of flags that systemd used, with this patch, a compat set of flags is set instead, thus resolving the introduced regression. The worst case scenario is that creating subvolumes/directories is still broken (as in, the current status quo).

[Other Info]

 * Example bad output

After update to systemd 237-3ubuntu10.9 systemd-tmpfiles-setup.service fails with:

Nov 21 13:44:12 node-blc49 systemd[1]: Starting Create Volatile Files and Directories...
Nov 21 13:44:12 node-blc49 systemd-tmpfiles[1226]: Failed to create directory or subvolume "/var": Bad file descriptor
Nov 21 13:44:12 node-blc49 systemd-tmpfiles[1226]: Failed to create directory or subvolume "/home": Bad file descriptor
Nov 21 13:44:12 node-blc49 systemd-tmpfiles[1226]: Failed to create directory or subvolume "/srv": Bad file descriptor
Nov 21 13:44:12 node-blc49 systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Nov 21 13:44:12 node-blc49 systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
Nov 21 13:44:12 node-blc49 systemd[1]: Failed to start Create Volatile Files and Directories.

This happens on btrfs root filesystems in real hardware and on our virtualized servers as well. 237-3ubuntu10.6 didnt show this errors and going back to 237-3ubuntu10 removes them as well.

# lsb_release -rd
Description: Ubuntu 18.04.1 LTS
Release: 18.04

# apt-cache policy systemd
systemd:
  Installiert: 237-3ubuntu10.9
  Installationskandidat: 237-3ubuntu10.9
  Versionstabelle:
 *** 237-3ubuntu10.9 500
        500 http://de.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     237-3ubuntu10 500
        500 http://de.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Renato Lorenzi (renatolorenzi)]

I have a similar problem after update from 229-4ubuntu21.2 to 229-4ubuntu21.9 in arm version.

lsb_release -rd
Description: Ubuntu 16.04.4 LTS
Release: 16.04

apt-cache policy systemd

systemd:
  Installed: 229-4ubuntu21.2
  Candidate: 229-4ubuntu21.9
  Version table:
     229-4ubuntu21.9 500
        500 http://ports.ubuntu.com xenial-security/main armhf Packages
        500 http://ports.ubuntu.com xenial-updates/main armhf Packages
 *** 229-4ubuntu21.2 100
        100 /var/lib/dpkg/status
     229-4ubuntu4 500
        500 http://ports.ubuntu.com xenial/main armhf Packages

[Gannet (ken20001)]

The same shit with systemd 237-3ubuntu10.9. How to revert to previous version, could somebody explain?

[SV (smvl)]

After last update some servises failed
systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories
ssh.service loaded failed failed OpenBSD Secure Shell server

If You reboot VPS after update You can't connect via SSH, because SSH service start failed

sshd can be launched via:
1. connect via web console (if VPS)
2. mkdir /run/sshd
3. systemctl start sshd

lsb_release -rd
Description: Ubuntu 16.04.5 LTS
Release: 16.04

apt-cache policy systemd
systemd:
  Installed: 229-4ubuntu21.9
  Candidate: 229-4ubuntu21.9
  Version table:
 *** 229-4ubuntu21.9 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     229-4ubuntu4 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

[crush (sgrosser)]

Same issue here! No SSH-Daemon will start again! Happens after Shutdown, just a Reboot shouldn't affect the System.

received the package via unattended security updates (?)

2018-11-21 06:55:30 upgrade systemd:amd64 229-4ubuntu21.8 229-4ubuntu21.9

No relevance at all with btrfs, we use ext4

[Renato Lorenzi (renatolorenzi)]

Now I received a update from 229-4ubuntu21.9 to 229-4ubuntu21.10 and everything worked again.

[Rene Meier (meier.rene)]

Yes but only because they have reverted the changes from CVE-2018-6954. This will come back very soon. Have a look at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.10 and
https://launchpad.net/bugs/1804847 .

[Gannet (ken20001)]

In Bionic upgrading to systemd 237-3ubuntu10.10 desn't solves this issue.

[Rene Meier (meier.rene)]

I did some tests. This issue is clearly related to btrfs, because I can not reproduce this with ext4. If you have a different filesystem, you have a different issue, maybe the one in https://launchpad.net/bugs/1804847.
How to reproduce this:
1) clean install from 'ubuntu-18.04.1-live-server-amd64.iso', all settings default but change root filesystem to btrfs.
2) system starts fine after reboot
3) issue 'apt install systemd' and get the 'Bad file descriptor' error allready during installation (attached log)
4) reboot with failing systemd-tmpfiles-setup.service

I will also attach a log of the errors during apt install.

[Rene Meier (meier.rene)]

I just checked 18.10 with systemd 239-7ubuntu10.4 and it fails with the same error message.

[Dimitri John Ledkov (xnox)]

yes this is a know regression, not sure where this is tracked. But yes, we do need to fix the tmpfiles on btrfs soon.

[Stanislav Rimsha (slowpokefarm)]

Experiencing same issue on 18.04 desktop with btrfs root filesystem, not being able to boot anything except emergency mode. Is there any workaround to boot into functional system?

[Brian Murray (brian-murray)]

Hello Rene, or anyone else affected,

Accepted systemd into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/239-7ubuntu10.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Rene Meier (meier.rene)]

Hi Brian,
thank you for providing this fix. It works fine on a minimal 18.10 server. I installed systemd 239-7ubuntu10.5 on a updated system.

[Gannet (ken20001)]

I'm confirming 239-7ubuntu10.5 fixed this issue on my system (Kubuntu 18.10). Please, add the fix to Bionic also. Thanks.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jurit (juritxyz)]

I have server version Ubuntu 18.04.1 LTS bionic systemd 237.
systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories

Problem started some weeks ago after I did apt-get upgrade

[Dirk Schmidtke (dirkschmidtke)]

What about a fix for 18.04 LTS. I am experiencing the same probleme for weeks on BTRFS:

journalctl -b 0 -u systemd-tmpfiles-setup.service
-- Logs begin at Fri 2018-07-06 11:06:42 CEST, end at Thu 2018-12-13 11:48:29 CET. --
Dez 13 11:33:38 Asuspro systemd[1]: Starting Create Volatile Files and Directories...
Dez 13 11:33:38 Asuspro systemd-tmpfiles[831]: Failed to create directory or subvolume "/var": Bad file descriptor
Dez 13 11:33:38 Asuspro systemd-tmpfiles[831]: Failed to create directory or subvolume "/home": Bad file descriptor
Dez 13 11:33:38 Asuspro systemd-tmpfiles[831]: Failed to create directory or subvolume "/srv": Bad file descriptor
Dez 13 11:33:38 Asuspro systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Dez 13 11:33:38 Asuspro systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
Dez 13 11:33:38 Asuspro systemd[1]: Failed to start Create Volatile Files and Directories.
Dez 13 11:33:39 Asuspro systemd[1]: Starting Create Volatile Files and Directories...
Dez 13 11:33:39 Asuspro systemd-tmpfiles[1013]: Failed to create directory or subvolume "/var": Bad file descriptor
Dez 13 11:33:39 Asuspro systemd-tmpfiles[1013]: Failed to create directory or subvolume "/home": Bad file descriptor
Dez 13 11:33:39 Asuspro systemd-tmpfiles[1013]: Failed to create directory or subvolume "/srv": Bad file descriptor
Dez 13 11:33:39 Asuspro systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Dez 13 11:33:39 Asuspro systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.
Dez 13 11:33:39 Asuspro systemd[1]: Failed to start Create Volatile Files and Directories.

[fusillator (fusillo)]

Hi, same issue here in bionic 18.04 LTS on btrs subvolumes with the following systemd releases:

 *** 237-3ubuntu10.10 100
        -10 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     237-3ubuntu10.9 500
        500 http://it.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages

How can we apply the patch also in bionic?

Regards

Luca

[supersasho (supersasho)]

Hi everyone,

right now I've got an emergency mode only with 18.04 (kde neon) and systemd=237-3ubuntu10.9 with btrfs. Is there a patch for 18.04/bionic that could be tested out? Seems like LTS release has been forgotten about. Downgrading to 237-3ubuntu10 isn't an option as it breaks ton of dependencies.

Thanks for any suggestions.

[fusillator (fusillo)]

hi again, not sure if the most sensible approach
anyway in bionic I tried to revert the last two patches from the source of systemd_237-3ubuntu10.9.debian with the following commands:

export QUILT_PATCHES=debian/patches
export QUILT_REFRESH_ARGS="-p ab --no-timestamps --no-index"
quilt pop
quilt pop
sed 's/^CVE-2018-6954.*/#&/' debian/patches/series

Then I recompiled and installed the new package and the error disappear.
Anyway the involved patches were there to fix some issues:

$ quilt header patches/CVE-2018-6954
Description: tmpfiles: don't resolve pathnames when traversing recursively
 through directory trees

 Otherwise we can be fooled if one path component is replaced underneath us.

 The patch achieves that by always operating at file descriptor level (by using
 *at() helpers) and by making sure we do not any path resolution when traversing
 direcotry trees.

 However this is not always possible, for instance when listing the content of a
 directory or some operations don't provide the *at() helpers or others (such as
 fchmodat()) don't have the AT_EMPTY_PATH flag. In such cases we operate on
 /proc/self/fd/%i pseudo-symlink instead, which works the same for all kinds of
 objects and requires no checking of type beforehand.

 Also O_PATH flag is used when opening file objects in order to prevent
 undesired behaviors: device nodes from reacting, automounts from
 triggering, etc...

 Fixes: CVE-2018-6954

Origin: upstream, https://github.com/systemd/systemd/commit/936f6bdb803c432578e2cdcc5f93f3bfff93aff0
Bug: https://github.com/systemd/systemd/issues/7986

$ quilt header patches/CVE-2018-6954_2
Description: Make tmpfiles safe

 In addition to backporting the changesets in #8822, this also backports
 e04fc13 (test: add tests for systemd-tmpfiles), as well as empty_to_root()
 from v239.

Origin: upstream, https://github.com/systemd/systemd/pull/8822/commits
Bug: https://github.com/systemd/systemd/issues/7986

So I'm not sure if it's a secure/stable workaround
Maybe it would be better mixixing up the releases installing the patched package from cosmic-proposed... I will test on another snapshot to see what happens..
Just a curiosity: is bionic still supported?

[Gannet (ken20001)]

Just notice: Bionic is an 18.04 LTS.

[Shelby Cain (alyandon)]

Any updates on a work around or patch for this issue in 18.04?

[fusillator (fusillo)]

in the previous post I missed the inline option of sed (sorry)...
I will try to merge the patch from cosmic in the we for the sake of experience
Anyway I don't have a good comprehension of the code... and I think there will be conflicts to resolve (the patch was thought to be applied to a differente codebase..)
So if a maintainer or a skilled programmer would provide the patch for bionic it would be better.

[fusillator (fusillo)]

I didn't find any conflict or overlap applying the btrfs-util-unbreak-tmpfiles-subvol-creation on top of the other patches on the source package systemd_237-3ubuntu10.9.debian, although the codebase of src/basic/btrfs-util.c affected by the patch had different hunks regards to the file in systemd_239.
So to apply the patch conceived by Brian Murray on bionic
I only had to add the missing macro FLAGS_SET on macro.h and refreshed the patch
here's the missing part to make it compile on bionic

Index: systemd-237/src/basic/macro.h
===================================================================
--- systemd-237.orig/src/basic/macro.h
+++ systemd-237/src/basic/macro.h
@@ -351,6 +351,9 @@ static inline unsigned long ALIGN_POWER2
 #define SET_FLAG(v, flag, b) \
         (v) = (b) ? ((v) | (flag)) : ((v) & ~(flag))

+#define FLAGS_SET(v, flags) \
+ (((v) & (flags)) == (flags))
+
 #define CASE_F(X) case X:
 #define CASE_F_1(CASE, X) CASE_F(X)
 #define CASE_F_2(CASE, X, ...) CASE(X) CASE_F_1(CASE, __VA_ARGS__)

The error of tmp disappeared at the boot, and the security issues should be patched by the preceding CVE-2018-6954* patches.
Let me know if I should submit the patch. Anyway I'm not a good programmer so wait for hints of maintainers, Brian Murray or other guru.

Regards

[Reto Glauser (rglauser)]

Thanks fusillo for the report, documentation and test. I did the same thing but failed since I also tried to get coccinelle/flags-set.cocci backported, which was an upstream change: https://github.com/systemd/systemd/commit/d94a24ca2ea769755beaed0659b966b1ec75c8d4

I assumed (wrongly) this upstream change would be needed as well.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 237-3ubuntu10.11

---------------
systemd (237-3ubuntu10.11) bionic-security; urgency=medium

  * SECURITY UPDATE: memory corruption in journald via attacker controlled alloca
    - debian/patches/CVE-2018-16864.patch: journald: do not store the iovec
      entry for process commandline on the stack
    - CVE-2018-16864
  * SECURITY UPDATE: memory corruption in journald via attacker controlled alloca
    - debian/patches/CVE-2018-16865_1.patch: journald: set a limit on the
      number of fields (1k)
    - debian/patches/CVE-2018-16865_2.patch: journal-remote: set a limit on the
      number of fields in a message
    - CVE-2018-16865
  * SECURITY UPDATE: out-of-bounds read in journald
    - debian/patches/CVE-2018-16866.patch: journal: fix syslog_parse_identifier()
    - CVE-2018-16866

  * Fix LP: #1804603 - btrfs-util: unbreak tmpfiles' subvol creation
    - add debian/patches/btrfs-util-unbreak-tmpfiles-subvol-creation.patch
    - update debian/patches/series
  * Fix LP: #1804864 - test: Set executable bits on TEST-22-TMPFILES shell scripts
    - add debian/patches/test-Set-executable-bits-on-TEST-22-TMPFILES-shell-script.patch
    - update debian/patches/series

 -- Chris Coulson <email address hidden> Wed, 09 Jan 2019 15:11:53 +0000

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 239-7ubuntu10.6

---------------
systemd (239-7ubuntu10.6) cosmic-security; urgency=medium

  * SECURITY UPDATE: memory corruption in journald via attacker controlled alloca
    - debian/patches/CVE-2018-16864.patch: journald: do not store the iovec
      entry for process commandline on the stack
    - CVE-2018-16864
  * SECURITY UPDATE: memory corruption in journald via attacker controlled alloca
    - debian/patches/CVE-2018-16865_1.patch: journald: set a limit on the
      number of fields (1k)
    - debian/patches/CVE-2018-16865_2.patch: journal-remote: set a limit on the
      number of fields in a message
    - CVE-2018-16865
  * SECURITY UPDATE: out-of-bounds read in journald
    - debian/patches/CVE-2018-16866.patch: journal: fix syslog_parse_identifier()
    - CVE-2018-16866

  * Fix LP: #1804603 - btrfs-util: unbreak tmpfiles' subvol creation
    - add debian/patches/btrfs-util-unbreak-tmpfiles-subvol-creation.patch
    - update debian/patches/series
  * Fix LP: #1804864 - test: Set executable bits on TEST-22-TMPFILES shell scripts
    - add debian/patches/test-Set-executable-bits-on-TEST-22-TMPFILES-shell-script.patch
    - update debian/patches/series

 -- Chris Coulson <email address hidden> Wed, 09 Jan 2019 14:37:15 +0000

[Jurit (juritxyz)]

Does it fixes also bionic with ext4? If yes, how?
Problem starting some services since november updates

Now I installed these last upgrades with 237-3ubuntu10.11 but this didnt fix error...

● systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev
● systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories

Description: Ubuntu 18.04.1 LTS
Release: 18.04
systemd:
  Installed: 237-3ubuntu10.11
  Candidate: 237-3ubuntu10.11
  Version table:
 *** 237-3ubuntu10.11 500
        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages
        100 /var/lib/dpkg/status
     237-3ubuntu10 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

Filesystem Type 1K-blocks Used Available Use% Mounted on
udev devtmpfs 8173584 0 8173584 0% /dev
tmpfs tmpfs 1640840 2672 1638168 1% /run
/dev/sda3 ext4 102687672 5534228 91894180 6% /
tmpfs tmpfs 8204192 0 8204192 0% /dev/shm
tmpfs tmpfs 5120 0 5120 0% /run/lock
tmpfs tmpfs 8204192 0 8204192 0% /sys/fs/cgroup
/dev/md0 ext4 5325203376 4998238260 58519832 99% /phototdata
/dev/loop1 squashfs 91648 91648 0 100% /snap/core/6130
/dev/loop0 squashfs 90368 90368 0 100% /snap/core/5897
/dev/loop2 squashfs 91648 91648 0 100% /snap/core/6034
tmpfs tmpfs 1640836 0 1640836 0% /run/user/0

[supersasho (supersasho)]

237-3ubuntu10.11 fixed the issue for me on KDE Neon (18.04/Bionic). I've got btrfs on my / lv though.

Thank you all that helped to resolve this. :)

[Andreas Kar (thexmanxyz)]

To supplement the issue I will add more relevant links:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1804847
https://forum.armbian.com/topic/8852-ssh-doesnt-work-on-orange-pi-zero/
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1811580

For me the issue also appeared with 229-4ubuntu21.9. Was fixed with 229-4ubuntu21.10 and now appeared again with 229-4ubuntu21.15. I'm on Armbian with ext4 and have the issue now again.

Distribution / Kernel
Linux xxx 3.4.113-sun8i #2 SMP PREEMPT Sat Jan 12 15:54:26 CET 2019 armv7l armv7l armv7l GNU/Linux
Distributor ID: Ubuntu
Description: Ubuntu 16.04.5 LTS
Release: 16.04
Codename: xenial

Output of journalctl -b 0 -u systemd-tmpfiles-setup.service

Jän 14 11:01:51 xxx systemd[1]: Starting Create Volatile Files and Directories...
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: [/usr/lib/tmpfiles.d/var.conf:14] Duplicate line for path "/var/log", ignoring.
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/log: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/lib: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/sendsigs.omit.d: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /home: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /srv: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/lock/subsys: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/run/lighttpd: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/cache: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /var/cache/man: Bad file descriptor
Jän 14 11:01:51 xxx systemd-tmpfiles[581]: Failed to validate path /run/openvpn: Bad file descriptor
Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Main process exited, code=exited, status=1/FAILURE
Jän 14 11:01:51 xxx systemd[1]: Failed to start Create Volatile Files and Directories.
Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Unit entered failed state.
Jän 14 11:01:51 xxx systemd[1]: systemd-tmpfiles-setup.service: Failed with result 'exit-code'.

Affected services:

# dnsmasq.service loaded failed failed dnsmasq - A lightweight DHCP and caching DNS server
# lighttpd.service loaded failed failed Lighttpd Daemon
# <email address hidden> loaded failed failed OpenVPN connection to server
# ssh.service loaded failed failed OpenBSD Secure Shell server
# systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev
# systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories

[Andreas Kar (thexmanxyz)]

Upgrade to Linux pan 4.19.13-sunxi #5.70 SMP Sat Jan 12 15:43:21 CET 2019 armv7l armv7l armv7l GNU/Linux solved the issue for me.

[Jurit (juritxyz)]

Is it realistic, that Bionic also adds this Linux pan 4.19.13-sunxi to their updates? Or have I upgrade it manually, if yes, how?

[Andreas Kar (thexmanxyz)]

Jurit (juritxyz) I tested the above kernel on 16.04.05 Xenial. However it has to be said that the 4.19.13-sunxi is an ARM7 kernel (Armbian). I don't know on which platform you are but I can easily upgrade this kernel manually over the Armbian configuration application. I assume you are on x86/x64 which is the latest LTS so I assume you're already on the latest kernel supported for your distro. IDK why 4.19.13-sunxi solves it...I still consider this issue unsolved even if the latest kernel on my distro solves it...

[Andreas Kar (thexmanxyz)]

*x86/x64 18.04.01

[Jurit (juritxyz)]

Thank you, I have:
root@tery:~# uname -m
x86_64
root@tery:~# uname -r
4.15.0-43-generic
root@tery:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic

Are you sure, if I update bionic (LTS) kernel to armbian I can use all future bionic LTS updates?

[Dimitri John Ledkov (xnox)]

@jurit you mention armbian which is not an Ubuntu Project supported flavour. You show kernel version strings for x86_64 arch, instead of ARM. You should contact armbian to get better/newer kernel which is compatible with what userspace expects. If possible, I can only support and recommend to get all of your packages from the Ubuntu Project.

[Jurit (juritxyz)]

@Dmitri, Thank you
Do you have any ideas how I can fix this bug without this armbian kernel:
● systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev
● systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories

This started after november updates (apt-get update).

I have Ubuntu 18.04.01. LTS bionic,
And moreover I dont know this armbian kernel will fix it or not....

[Andreas Kar (thexmanxyz)]

@Jurit (juritxyz) Stop thinking about the Armbian (ARM) kernel, it has nothing to with your kernel and you won't be able to update to my kernel because as already said it is for ARM7-based SOCs. You are on x86_x64 which needs a proper kernel for your architecture. You can check what kernels are available for your platform and verify if there is a newer one which maybe solves the issue for you. However IDK if there is one. And no you should in no case update to Armbian kernel ... I assume it will break everything. Another option would be to downgrade to systemd 21.11 or 21.10 and hold the packages until the bug is fixed.

[Andreas Kar (thexmanxyz)]

@Jurit (juritxyz)
I was wrong you need an other versions of systemd (21.11, 21.10 is for 16.04.5) IDK which ones but you can get them from launchpad I assume you need e.g. this architecture:

https://launchpad.net/ubuntu/bionic/amd64/systemd/

You need these packages for successful downgrade get the version before the issue occured:
libpam-systemd
libsystemd0
systemd
systemd-sysv

Are you even on Armbian? What is your distribution! If you are on Armbian please proceed to this thread https://forum.armbian.com/topic/8852-ssh-doesnt-work-on-orange-pi-zero/?page=2

@Dimitri John Ledkov (xnox)
I assume he is not on Armbian because nothing indicates that but however the issue still persists on certain kernel versions, obviously.

[Kris B. (krisbee)]

Still happening for me on 18.04 LTS...

[Jurit (juritxyz)]

No, I am not on Armbian.
I have:
x86_64
4.15.0-43-generic
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic

systemd:
  Installed: 237-3ubuntu10.11
  Candidate: 237-3ubuntu10.11

And errors started after november 2018 updates:
● systemd-tmpfiles-setup-dev.service loaded failed failed Create Static Device Nodes in /dev
● systemd-tmpfiles-setup.service loaded failed failed Create Volatile Files and Directories

I never downgraded my Ubuntu system, how I can do this?

[Andreas Kar (thexmanxyz)]

Jurit (juritxyz) You need the following packages to downgrade systemd:

libpam-systemd
libsystemd0
systemd
systemd-sysv

you can get them from here

https://launchpad.net/ubuntu/bionic/amd64/libpam-systemd/
https://launchpad.net/ubuntu/bionic/amd64/libsystemd0/
https://launchpad.net/ubuntu/bionic/amd64/systemd/
https://launchpad.net/ubuntu/bionic/amd64/systemd-sysv/

download the last working revision for all of them and install it. Afterwards the issue should be gone. However you have to set the four packages to hold with apt to ensure that they won't get updated ... to prevent issues until the problem gets resolved. As you said November update broke it, it would start with 237-3ubuntu10.8 and check if this solves it. You can also try a later revision but I assume this is good to go.

[Dimitri John Ledkov (xnox)]

@Jurit

Your system should be working normally, and most likely nothing to do with this bug report. Please open a new bugreport attaching logs to figure our what is wrong with your installation.

What filesystem types do you use ($ mount)? What are the permissions of your rootfs? ($ ls -latr /)

[Jurit (juritxyz)]

user1@teki3:~# sudo mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=8173592k,nr_inodes=2043398,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1640840k,mode=755)
/dev/sda3 on / type ext4 (rw,relatime,data=ordered)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=378)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/var/lib/snapd/snaps/core_6259.snap on /snap/core/6259 type squashfs (ro,nodev,relatime,x-gdu.hide)
/var/lib/snapd/snaps/core_6130.snap on /snap/core/6130 type squashfs (ro,nodev,relatime,x-gdu.hide)
/var/lib/snapd/snaps/core_6034.snap on /snap/core/6034 type squashfs (ro,nodev,relatime,x-gdu.hide)
/dev/md0 on /aurdata type ext4 (rw,relatime,data=ordered)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=1640836k,mode=700)

user1@teki3:~# sudo ls -latr /
total 132
drwxr-xr-x 10 root root 4096 Apr 26 2018 usr
drwxr-xr-x 2 root root 4096 Apr 26 2018 srv
drwxr-xr-x 2 root root 4096 Apr 26 2018 opt
drwxr-xr-x 2 root root 4096 Apr 26...

[Jurit (juritxyz)]

Resolved, but I dont know how...

[Kris B. (krisbee)]

Srill happening on 18.04LTS with ext4fs

[Jurit (juritxyz)]

Kris B.
I think this november update changed in some way / ownership. Also I tryed to test it and it recurrenced, if I was not loged in as root, but I did as user sudo apt-get upgrade
To fix it try this, log in as root:
chown root:root /

[Kris B. (krisbee)]

Jurit - I tried that months ago as documented in the forums. Didnt fix the issue. I also have run the sudo apt-get upgrade several times as well - still not fixed for me. I also see they did push a new kernel - still occurs. Not sure what else I can do...

[Dimitri John Ledkov (xnox)]

all updates are out. there no more further updates planned for this issue.
if you have any issues on your systems, you need to debug what's wrong and open a new bug report, attaching systemd-journal details from a broken boot.
most likely something is wrong with your system, forexample unsafe ownership, unsafe permissions, broken host kernel etc.
on all up to date ubuntu systems, installed using ubuntu installers, with latest ubuntu kernel there are no issues at all.
please open a new bug report, if you have issues with your systems.