systemd journals take up a lot of space, and it's not obvious how much is used, and what the upper limit is.

Bug #1790205 reported by Benjamin Bach on 2018-08-31
26
This bug affects 5 people
Affects Status Importance Assigned to Milestone
systemd
New
Unknown
systemd (Ubuntu)
Wishlist
Unassigned

Bug Description

After running Bionic for 3 months, I had 2.6 GB of journals.

I would not expect from a normal desktop user that they should have to run commands like `sudo journalctl --vacuum-time=10d`.

I would nominate this command as a sane default to have running at each reboot to ensure that logs do not exceed 500 MB:

sudo journalctl --vacuum-size=500M

Supposedly, a server should by default retain more logs, so perhaps this should be implemented through a configuration package "systemd-configuration-desktop" as a dependency of the ubuntu-desktop meta package?

..... as it turns out, it's hard to see how much disk space is used, and what the upper limit is, even when it is set and respected by default.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
Emanuele (emanuc) wrote :

Same problem, I find myself 4.4GB of occupied space

Benjamin Bach (benjaoming) wrote :

A month before the release of 18.04, a change was released to make systemd journals persistent, but the setting SystemMaxUse was left unset in /etc/systemd/journald.conf

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1618188

The bug's regression analysis seems wrong, as it states: "The journald daemon has limits set for logs, meaning they will be rotated and discarded and should not cause out of disk-space errors."

It also seems likely that no testers of 18.04 would discover this issue, since it requires running out of disk space after having accumulated large systemd journals over time.

The documentation is here:

https://www.freedesktop.org/software/systemd/man/journald.conf.html#SystemMaxUse=

SystemKeepFree could be set, since it overrides the other values listed when the disk is running out of space. But I'm unsure, because if running out of disk space means that all the logs are silently truncated, then that is also a dangerous consequence on for instance a server where you could need logs to troubleshoot why disk space is running out...

Tom Reynolds (tomreyn) wrote :

$ journalctl --list-boots 2>/dev/null | head -n1 && journalctl --disk-usage 2>/dev/null
-37 64975ef449c34cdc828feb0197d7a2f5 Mon 2018-09-10 02:00:47 CEST—Mon 2018-09-10 08:05:57 CEST
Archived and active journals take up 1.8G in the file system.

My interpretation is my systems' situation is that 37 sessions were recorded, starting Sept 10, 2018 (~ three months ago), and that logs consume 1.8 GB altogether. I don't know for sure whether rotation does (not) take place, but it doesn't look kike it does, and I think 3 months of full logs are too much. If, however, it's intentional, then this would mean changed system requirements (higher storage capacity on the root file system (/)) for Ubuntu since introduction of systemd-journald (which should go into the release notes at least).

tags: added: rls-dd-incoming
Changed in systemd (Ubuntu):
importance: Undecided → High
Dimitri John Ledkov (xnox) wrote :

I disagree with the title and description of the bug report.

Journald, calculates a hard limit on logs (based on the system configuration).

$ journalctl -b -u systemd-journald.service -n 1
-- Logs begin at Mon 2018-09-17 14:21:50 BST, end at Thu 2019-03-21 15:16:25 GMT. --
Mar 21 11:27:37 ottawa systemd-journald[571]: System journal (/var/log/journal/1b8df0fa27039f0163586c6756a6d401) is 3.7G, max 4.0G, 231.6M free.

there is
$ journalctl --disk-usage
Archived and active journals take up 3.7G in the file system.

command, but that currently does not print the max size, and free, like journald does on startup. Maybe it should.

and --disk-usage is a bit odd, as it conflates the runtime and persistent journal numbers.

I guess --disk-usage should print more info, about runtime vs persistent logs; and the max sizes.

Now, do any of you have the max limits breached? Can you please paste the outputs of:
journalctl -b -u systemd-journald.service -n 1
journalctl --disk-usage

Changed in systemd (Ubuntu):
status: Confirmed → Incomplete
Tom Reynolds (tomreyn) wrote :

Not breached here:

$ journalctl -b -u systemd-journald.service -n 1
-- Logs begin at Sat 2018-09-29 05:03:28 CEST, end at Thu 2019-03-21 18:18:38 CET. --
Mar 20 13:32:03 debby2017 systemd-journald[699]: System journal (/var/log/journal/f8b692c8bb791fe2804f3d5a5905148b) is 1.8G, max 1.8G, 0B free.

$ journalctl --disk-usage
Archived and active journals take up 1.8G in the file system.

However, with systemd-journald persistence enabled, and classic logging daemons not removed (and still logging) on upgrades, you can end up with much data stored in /var/log, a lot more than there used to be in the past, which could cause problems, also (but not only) if combined with bug 1785321.

Tom Reynolds (tomreyn) wrote :

So I wasn't aware of this size hard limit on persistent storage - I just noticed that it is 'already' allocating 1.8GB, but this is indeed less than the 10% of the file system the man page says it will consume. So from my personal point of view this is a non-issue then (ideally Benjamin will also provide feedback, though) - but the new behavior may be worth discussing in release notes.

Emanuele (emanuc) wrote :

Everything ok for me too

$ journalctl -b -u systemd-journald.service -n 1
-- Logs begin at Tue 2019-02-19 00:29:40 CET, end at Thu 2019-03-21 22:33:40 CET. --
mar 21 18:33:53 emanuc systemd-journald[602]: System journal (/var/log/journal/c60f3deceebd474aa97aecbd13ebf6b9) is 744.0M, max 4.0G, 3.2G free.
$ journalctl --disk-usage
Archived and active journals take up 744.0M in the file system.

Dimitri John Ledkov (xnox) wrote :

I have filed the usability issue of --disk-usage upstream at https://github.com/systemd/systemd/issues/12063

summary: - systemd journals take up too much space, aren't vacuumed automatically
+ systemd journals take up a lot of space, and it's not obvious how much
+ is used, and what the upper limit is.
description: updated
tags: added: rls-dd-notfixing
removed: rls-dd-incoming
Changed in systemd (Ubuntu):
status: Incomplete → Confirmed
importance: High → Wishlist
Changed in systemd:
status: Unknown → New
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.