systemd journald failed unmounting var

[Adauto Serpa]

[impact]

on a system with /var and/or /var/log mounted separately from /, shutdown can't unmount /var (or /var/log) cleanly because systemd-journald is using it.

[test case]

install a system with /var mounted separately from /

shutdown, and notice:

[FAILED] Failed unmounting /var.

[regression potential]

TBD

[other info]

original description:

--

When the system is shut down or restarted the systemd-journal does not disassemble the disks correctly.

On this thread https://unix.stackexchange.com/questions/378678/why-do-i-get-the-error-failed-unmounting-var-during-shutdown

I checked what editing /etc/systemd/journald.conf
to change the Storage = line to Storage = volatile the problem is solved, but some of the logs on the shutdown are lost.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 237-3ubuntu10.3
ProcVersionSignature: Ubuntu 4.15.0-32.35-generic 4.15.18
Uname: Linux 4.15.0-32-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.2
Architecture: amd64
Date: Mon Aug 20 18:15:07 2018
ExecutablePath: /lib/systemd/systemd-journald
InstallationDate: Installed on 2018-08-20 (0 days ago)
InstallationMedia: Ubuntu-Server 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
Lsusb:
 Bus 001 Device 002: ID 80ee:0021 VirtualBox USB Tablet
 Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
MachineType: innotek GmbH VirtualBox
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-32-generic root=UUID=65cb761b-4881-4031-9113-e6bb6a1437b4 ro
SourcePackage: systemd
SystemdDelta:
 [EXTENDED] /lib/systemd/system/rc-local.service → /lib/systemd/system/rc-local.service.d/debian.conf
 [EXTENDED] /lib/systemd/system/user@.service → /lib/systemd/system/user@.service.d/timeout.conf

 2 overridden configuration files found.
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/01/2006
dmi.bios.vendor: innotek GmbH
dmi.bios.version: VirtualBox
dmi.board.name: VirtualBox
dmi.board.vendor: Oracle Corporation
dmi.board.version: 1.2
dmi.chassis.type: 1
dmi.chassis.vendor: Oracle Corporation
dmi.modalias: dmi:bvninnotekGmbH:bvrVirtualBox:bd12/01/2006:svninnotekGmbH:pnVirtualBox:pvr1.2:rvnOracleCorporation:rnVirtualBox:rvr1.2:cvnOracleCorporation:ct1:cvr:
dmi.product.family: Virtual Machine
dmi.product.name: VirtualBox
dmi.product.version: 1.2
dmi.sys.vendor: innotek GmbH
mtime.conffile..etc.systemd.journald.conf: 2018-08-20T18:00:58.586165

[Adauto Serpa (adautoserpa)]

One more discussion about systemctl does not work correctly with the separate / var partition
https://bbs.archlinux.org/viewtopic.php?pid=1204644#p1204644

Sorry English !

[Adauto Serpa (adautoserpa)]

I ended up using another solution, just do not know if it is the most correct.
Edit
sudo vim /lib/systemd/system/systemd-journald.service

Add the line "RequiresMountsFor=/var/log/journal" as below

[Unit]
...
RequiresMountsFor=/var/log/journal

systemctl daemon-reload
systemctl restart systemd-journald
# You'll now see the new setting listed at the bottom of:
systemctl cat systemd-journald.service

[Adauto Serpa (adautoserpa)]

Guys, I think it's a critical problem!
All because I created an exclusive disk for / var

[C de-Avillez (hggdh2)]

Updated with upstream bug #; this seems to be an on-going concern, and depends on some not-yet-available interfaces between kernel and systemd.

Marking Triaged/Medium. There are some bypasses discussed in the upstream bug, but not all possible effects have already been discussed (for example, setting 'storage=volatile' completely loses the journald logs at shutdown).

[Adauto Serpa (adautoserpa)]

C de-Avillez,

As I mentioned there is a solution I described in comment #3

Adding just a dependency before dismounting to /var/log/journal

Thank you for your attention!

[Dan Streetman (ddstreet)]

Just to clarify, this does not hang shutdown/reboot for anyone, right? It only fails to cleanly unmount /var, right?

[Adauto Serpa (adautoserpa)]

@Dan Streetman

That's right, this problem does not affect the reboot or shutdown of the operating system.
My concern is the loss of data due to improper unmount.

[Dan Streetman (ddstreet)]

It seems like the upstream commits to fix this may have made journalctl --flush flaky and/or introduced other regressions:
https://github.com/systemd/systemd/pull/12230#issuecomment-491593897

Additionally, the fix upstream is not a trivial backport and has the potential to cause even more problems for SRU releases.
https://github.com/systemd/systemd/pull/12230/commits

As this bug is rather limited in scope (only systems with /var or /var/log mounted separately from /) and the only potential issue I see is /var isn't cleanly unmounted, it seems like backporting the upstream fix may bring more problems than it solves.

Possibly there is some simpler workaround we could add for SRU releases, but it would take some more investigation. I'm removing myself as bug owner here, so possibly someone else can investigate, in case I'm not able to get back to this.

[Dimitri John Ledkov (xnox)]

Does installing finalrd package help?

It should pivot off root on shutdown, and unmount separate /var and /var/log cleanly.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Brian Murray (brian-murray)]

The Eoan Ermine has reached end of life, so this bug will not be fixed for that release

[Dan Streetman (ddstreet)]

please reopen if this is still an issue