systemd vacuuming does not work for remote journals

[Kai Bojens]

[impact]

attempting to vacuum journals using -D with the 'root' (in journald terms) journal directory fails to perform any vacuuming. This can lead to journal files continually increasing in size and filling up the filesystem.

[test case]

see original description below.

[regession potential]

this removes a check in the code when iterating journal directories during vacuuming, so any regression would likely occur during vacuuming, possibly causing failure to vacuum.

[scope]

this is fixed upstream by commit c488660e6edb3c1375ab62514a8df035c3d712bf which is included starting in version 244, so this is already included in Focal. This is needed for Eoan, Bionic, and Xenial.

[original description]

systemd has an issue in regard to vacuuming remote journals: https://github.com/systemd/systemd/issues/2376 - this affects 18.04 as this version still has this problem.

Steps to reproduce on "Description: Ubuntu 18.04.3 LTS":

root@1:~# du -hs /var/log/journal/remote/
8.6G /var/log/journal/remote/
root@1:~# journalctl -D /var/log/journal/remote --vacuum-size=100M
root@1:~# du -hs /var/log/journal/remote/
8.6G /var/log/journal/remote/

Would it be possible to backport this trivial patch https://github.com/systemd/systemd/pull/13924 into 18.04?

[Balint Reczey (rbalint)]

Fixed in v244, I think hits is a minor issue that does not seem to warrant a backport.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dan Streetman (ddstreet)]

I would disagree that this is minor, having log files fill up a filesystem and being unable to vacuum to reduce their size seems important to me, especially when considering the simplicity of the patch.

[Łukasz Zemczak (sil2100)]

Hello Kai, or anyone else affected,

Accepted systemd into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/242-7ubuntu3.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Kai, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.40 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (242-7ubuntu3.8) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/242-7ubuntu3.8 (ppc64el)
prometheus-postgres-exporter/unknown (armhf)
gnome-desktop3/3.34.2-2ubuntu1~19.10.1 (armhf)
openssh/1:8.0p1-6build1 (s390x, i386, ppc64el, arm64, armhf, amd64)
snapd/2.42.1+19.10 (i386)
udisks2/unknown (armhf)
umockdev/0.13.2-1 (armhf)
sks/unknown (armhf)
network-manager/1.20.4-2ubuntu2.2 (ppc64el, arm64)
asterisk/unknown (armhf)
logrotate/unknown (armhf)
lxc/3.0.4-0ubuntu1 (i386)
netplan.io/0.98-0ubuntu1 (amd64)
dbus/unknown (armhf)
gvfs/1.42.1-1ubuntu1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (237-3ubuntu10.40) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.36.1-0ubuntu1.3.3 (amd64)
prometheus-postgres-exporter/unknown (armhf)
systemd/237-3ubuntu10.40 (i386, ppc64el)
umockdev/0.11.1-1 (armhf)
linux-hwe-5.0/5.0.0-47.51~18.04.1 (armhf)
kde4libs/4:4.14.38-0ubuntu3.1 (armhf)
util-linux/unknown (armhf)
nftables/unknown (armhf)
linux-raspi2-5.3/5.3.0-1023.25~18.04.1 (armhf)
netplan.io/0.98-0ubuntu1~18.04.1 (i386)
openssh/1:7.6p1-4ubuntu0.3 (arm64, i386, armhf, ppc64el, s390x, amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Kai Bojens (starborn)]

systemd 237-3ubuntu10.40 in bonic fixed the bug for me:

"Vacuuming done, freed 19.1G of archived journals from /var/log/journal/remote."

[Łukasz Zemczak (sil2100)]

Hello Kai, or anyone else affected,

Accepted systemd into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/229-4ubuntu21.28 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted systemd (229-4ubuntu21.28) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

udisks2/2.1.7-1ubuntu1 (amd64)
multipath-tools/0.5.0+git1.656f8865-5ubuntu2.5 (armhf)
nplan/0.32~16.04.7 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dan Streetman (ddstreet)]

eoan:

ubuntu@lp1858412-e:~$ dpkg -l systemd|grep systemd
ii systemd 242-7ubuntu3.7 amd64 system and service manager
ubuntu@lp1858412-e:~$ du -sh /var/log/journal/remote
273M /var/log/journal/remote
ubuntu@lp1858412-e:~$ sudo journalctl -D /var/log/journal/remote --vacuum-size=10M
ubuntu@lp1858412-e:~$ du -sh /var/log/journal/remote
273M /var/log/journal/remote

ubuntu@lp1858412-e:~$ dpkg -l systemd|grep systemd
ii systemd 242-7ubuntu3.8 amd64 system and service manager
ubuntu@lp1858412-e:~$ du -sh /var/log/journal/remote
273M /var/log/journal/remote
ubuntu@lp1858412-e:~$ sudo journalctl -D /var/log/journal/remote --vacuum-size=10M
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000000001-0005a4f93261040f.journal (8.0M).
Deleted archived journal /var/log/journal/remote/user-1000@63c462cb3bfe42d8aae1a91b4d2711a6-0000000000000435-0005a4f935a960b5.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-00000000000013e8-0005a4fb05e4c7ef.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-00000000000013ec-0005a4fb092ed6af.journal (8.0M).
Deleted archived journal /var/log/journal/remote/user-1000@63c462cb3bfe42d8aae1a91b4d2711a6-00000000000014fe-0005a4fb271141e0.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001941-0005a4fb29d41b99.journal (8.0M).
Deleted archived journal /var/log/journal/remote/user-1000@63c462cb3bfe42d8aae1a91b4d2711a6-0000000000001993-0005a4fbc19b8de5.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001ca3-0005a4fbc267c680.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001dd8-0005a4fbc464346b.journal (8.0M).
Deleted archived journal /var/log/journal/remote/user-1000@0110cef9e4a8457ba44cf2e309034570-0000000000001de1-0005a4fbc5523608.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001df2-0005a4fbc93c71d7.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001df4-0005a4fbf7e6a339.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001df7-0005a4fc2dbe7958.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001dfe-0005a4fc2f8efc30.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001dff-0005a4fc4181be1d.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001e02-0005a4fc7bce3298.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001e06-0005a4fc7dddc828.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@106304fba7e24f4eaa633c524dee66e4-0000000000001e09-0005a4fc91a3f7ed.journal (8.0M).
Deleted archive...

[Dan Streetman (ddstreet)]

bionic:

ubuntu@lp1858412-b:~$ dpkg -l systemd|grep systemd
ii systemd 237-3ubuntu10.39 amd64 system and service manager
ubuntu@lp1858412-b:~$ du -sh /var/log/journal/remote
417M /var/log/journal/remote
ubuntu@lp1858412-b:~$ sudo journalctl -D /var/log/journal/remote/ --vacuum-size=10M
ubuntu@lp1858412-b:~$ du -sh /var/log/journal/remote
417M /var/log/journal/remote

ubuntu@lp1858412-b:~$ dpkg -l systemd|grep systemd
ii systemd 237-3ubuntu10.40 amd64 system and service manager
ubuntu@lp1858412-b:~$ du -sh /var/log/journal/remote
417M /var/log/journal/remote
ubuntu@lp1858412-b:~$ sudo journalctl -D /var/log/journal/remote/ --vacuum-size=10M
Deleted archived journal /var/log/journal/remote//remote-127.0.0.1@5a01d9e8d4804c69a55a151a95b0313b-0000000000000001-0005a4f9330068c4.journal (128.0M).
Deleted archived journal /var/log/journal/remote//remote-127.0.0.1@5a01d9e8d4804c69a55a151a95b0313b-00000000000267dd-0005a4f9337fd2bf.journal (128.0M).
Deleted archived journal /var/log/journal/remote//remote-127.0.0.1@5a01d9e8d4804c69a55a151a95b0313b-0000000000046581-0005a4fa0f7fe771.journal (128.0M).
Vacuuming done, freed 384.0M of archived journals from /var/log/journal/remote/.
ubuntu@lp1858412-b:~$ du -sh /var/log/journal/remote
33M /var/log/journal/remote

[Dan Streetman (ddstreet)]

sorry, i missed that it was already verified for bionic

[Dan Streetman (ddstreet)]

xenial:

ubuntu@lp1858412-x:~$ dpkg -l systemd|grep systemd
ii systemd 229-4ubuntu21.27 amd64 system and service manager
ubuntu@lp1858412-x:~$ du -sh /var/log/journal/remote
65M /var/log/journal/remote
ubuntu@lp1858412-x:~$ sudo journalctl -D /var/log/journal/remote --vacuum-size=10M
ubuntu@lp1858412-x:~$ du -sh /var/log/journal/remote
65M /var/log/journal/remote

ubuntu@lp1858412-x:~$ dpkg -l systemd|grep systemd
ii systemd 229-4ubuntu21.28 amd64 system and service manager
ubuntu@lp1858412-x:~$ du -sh /var/log/journal/remote
65M /var/log/journal/remote
ubuntu@lp1858412-x:~$ sudo journalctl -D /var/log/journal/remote --vacuum-size=10M
Deleted archived journal /var/log/journal/remote/system@2cd50be57af347e1a473974288f23b15-0000000000000001-0005a4fe64b79740.journal (8.0M).
Deleted archived journal /var/log/journal/remote/system@2cd50be57af347e1a473974288f23b15-000000000000047d-0005a4fe7e0e6b7b.journal (8.0M).
Vacuuming done, freed 16.0M of archived journals on disk.
ubuntu@lp1858412-x:~$ du -sh /var/log/journal/remote
49M /var/log/journal/remote

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 242-7ubuntu3.8

---------------
systemd (242-7ubuntu3.8) eoan; urgency=medium

  * d/t/logind: skip if nonexistent /sys/power/state (LP: #1862657)
  * d/p/lp1859862-network-Do-not-disable-IPv6-by-writing-to-sysctl.patch:
    - enable ipv6 when needed (LP: #1859862)
  * d/p/lp1858412-journalctl-allow-running-vacuum-on-remote-journals-t.patch:
    - allow vacuuming journal 'root' dir (LP: #1858412)
  * d/rules: use meson --print-errorlogs instead of cat testlog
    - (LP: #1870811)
  * d/p/lp1776654-test-Synchronize-journal-before-reading-from-it.patch:
    - sync journal before reading from it (LP: #1776654)
  * d/p/lp1837914-journal-do-not-trigger-assertion-when-journal_file_c.patch:
    - do not crash if NULL passted to journal destructor (LP: #1837914)
  * d/e/initramfs-tools/hooks/udev:
    - Follow symlinks when finding link files to copy into initramfs
      (LP: #1868892)

 -- Dan Streetman <email address hidden> Mon, 20 Apr 2020 09:44:51 -0400

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for systemd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 237-3ubuntu10.40

---------------
systemd (237-3ubuntu10.40) bionic; urgency=medium

  * d/t/logind: skip if nonexistent /sys/power/state (LP: #1862657)
  * d/p/lp1839290-Change-job-mode-of-manager-triggered-restarts-to-JOB.patch:
    - when restarting service after failure, replace existing queued jobs
      (LP: #1839290)
  * d/p/lp1867421-70-mouse.hwdb-Set-DPI-for-MS-Classic-IntelliMouse.patch:
    - fix resolution of IntelliMouse (LP: #1867421)
  * d/p/lp1858412-journalctl-allow-running-vacuum-on-remote-journals-t.patch:
    - allow vacuuming journal 'root' dir (LP: #1858412)
  * d/p/lp1862232/0001-network-add-more-log-messages-in-configuring-DHCP4-c.patch,
    d/p/lp1862232/0002-network-add-more-log-messages-in-configuring-DHCP6-c.patch,
    d/p/lp1862232/0003-network-also-check-that-Hostname-is-a-valid-DNS-doma.patch,
    d/p/lp1862232/0004-network-use-free_and_replace.patch,
    d/p/lp1862232/0005-network-DHCP-ignore-error-in-setting-hostname-when-i.patch,
    d/p/lp1862232/0006-man-mention-that-Hostname-for-DHCP-must-be-a-valid-D.patch,
    d/p/lp1862232/0007-resolve-fix-error-handling-of-dns_name_is_valid.patch:
    - do not fail network setup if hostname is not valid (LP: #1862232)
  * d/t/systemd-fsckd: Skip test on arm64 (LP: #1870194)
  * d/p/lp1870589-seccomp-rework-how-the-S-UG-ID-filter-is-installed.patch:
    - fix test-seccomp failure (LP: #1870589)
  * d/rules: use meson --print-errorlogs instead of cat testlog
    - (LP: #1870811)
  * d/p/lp1776654-test-Synchronize-journal-before-reading-from-it.patch:
    - sync journal before reading from it (LP: #1776654)
  * d/p/lp1837914-journal-do-not-trigger-assertion-when-journal_file_c.patch:
    - do not crash if NULL passted to journal destructor (LP: #1837914)
  * d/e/initramfs-tools/hooks/udev:
    - Follow symlinks when finding link files to copy into initramfs
      (LP: #1868892)

 -- Dan Streetman <email address hidden> Mon, 20 Apr 2020 10:12:49 -0400

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 229-4ubuntu21.28

---------------
systemd (229-4ubuntu21.28) xenial; urgency=medium

  * d/t/logind: skip if nonexistent /sys/power/state (LP: #1862657)
  * d/p/lp1839290-Change-job-mode-of-manager-triggered-restarts-to-JOB.patch:
    - when restarting service after failure, replace existing queued jobs
      (LP: #1839290)
  * d/t/storage:
    - fix buggy test (LP: #1831459)
    - without scsi_debug, skip test (LP: #1847816)
  * d/p/lp1867421-70-mouse.hwdb-Set-DPI-for-MS-Classic-IntelliMouse.patch:
    - fix resolution of IntelliMouse (LP: #1867421)
  * d/p/lp1858412-journalctl-allow-running-vacuum-on-remote-journals-t.patch:
    - allow vacuuming journal 'root' dir (LP: #1858412)
  * d/p/lp1862232-network-DHCP-ignore-error-in-setting-hostname-when-i.patch:
    - do not let invalid hostname break dhcpv4 (LP: #1862232)
  * d/t/systemd-fsckd: Skip test on arm64 (LP: #1870194)
  * d/p/lp1837914-journal-do-not-trigger-assertion-when-journal_file_c.patch:
    - do not crash if NULL passted to journal destructor (LP: #1837914)
  * d/e/initramfs-tools/hooks/udev:
    - Follow symlinks when finding link files to copy into initramfs
      (LP: #1868892)

 -- Dan Streetman <email address hidden> Mon, 20 Apr 2020 10:17:40 -0400