systemd journals take up a lot of space, and it's not obvious how much is used, and what the upper limit is.

Bug #1790205 reported by Benjamin Bach
38
This bug affects 8 people
Affects Status Importance Assigned to Milestone
systemd
New
Unknown
systemd (Ubuntu)
Invalid
Wishlist
Unassigned

Bug Description

After running Bionic for 3 months, I had 2.6 GB of journals.

I would not expect from a normal desktop user that they should have to run commands like `sudo journalctl --vacuum-time=10d`.

I would nominate this command as a sane default to have running at each reboot to ensure that logs do not exceed 500 MB:

sudo journalctl --vacuum-size=500M

Supposedly, a server should by default retain more logs, so perhaps this should be implemented through a configuration package "systemd-configuration-desktop" as a dependency of the ubuntu-desktop meta package?

..... as it turns out, it's hard to see how much disk space is used, and what the upper limit is, even when it is set and respected by default.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
Revision history for this message
Emanuele (emanuc) wrote :

Same problem, I find myself 4.4GB of occupied space

Revision history for this message
Benjamin Bach (benjaoming) wrote :

A month before the release of 18.04, a change was released to make systemd journals persistent, but the setting SystemMaxUse was left unset in /etc/systemd/journald.conf

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1618188

The bug's regression analysis seems wrong, as it states: "The journald daemon has limits set for logs, meaning they will be rotated and discarded and should not cause out of disk-space errors."

It also seems likely that no testers of 18.04 would discover this issue, since it requires running out of disk space after having accumulated large systemd journals over time.

The documentation is here:

https://www.freedesktop.org/software/systemd/man/journald.conf.html#SystemMaxUse=

SystemKeepFree could be set, since it overrides the other values listed when the disk is running out of space. But I'm unsure, because if running out of disk space means that all the logs are silently truncated, then that is also a dangerous consequence on for instance a server where you could need logs to troubleshoot why disk space is running out...

Revision history for this message
Tom Reynolds (tomreyn) wrote :

$ journalctl --list-boots 2>/dev/null | head -n1 && journalctl --disk-usage 2>/dev/null
-37 64975ef449c34cdc828feb0197d7a2f5 Mon 2018-09-10 02:00:47 CEST—Mon 2018-09-10 08:05:57 CEST
Archived and active journals take up 1.8G in the file system.

My interpretation is my systems' situation is that 37 sessions were recorded, starting Sept 10, 2018 (~ three months ago), and that logs consume 1.8 GB altogether. I don't know for sure whether rotation does (not) take place, but it doesn't look kike it does, and I think 3 months of full logs are too much. If, however, it's intentional, then this would mean changed system requirements (higher storage capacity on the root file system (/)) for Ubuntu since introduction of systemd-journald (which should go into the release notes at least).

tags: added: rls-dd-incoming
Changed in systemd (Ubuntu):
importance: Undecided → High
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

I disagree with the title and description of the bug report.

Journald, calculates a hard limit on logs (based on the system configuration).

$ journalctl -b -u systemd-journald.service -n 1
-- Logs begin at Mon 2018-09-17 14:21:50 BST, end at Thu 2019-03-21 15:16:25 GMT. --
Mar 21 11:27:37 ottawa systemd-journald[571]: System journal (/var/log/journal/1b8df0fa27039f0163586c6756a6d401) is 3.7G, max 4.0G, 231.6M free.

there is
$ journalctl --disk-usage
Archived and active journals take up 3.7G in the file system.

command, but that currently does not print the max size, and free, like journald does on startup. Maybe it should.

and --disk-usage is a bit odd, as it conflates the runtime and persistent journal numbers.

I guess --disk-usage should print more info, about runtime vs persistent logs; and the max sizes.

Now, do any of you have the max limits breached? Can you please paste the outputs of:
journalctl -b -u systemd-journald.service -n 1
journalctl --disk-usage

Changed in systemd (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Tom Reynolds (tomreyn) wrote :

Not breached here:

$ journalctl -b -u systemd-journald.service -n 1
-- Logs begin at Sat 2018-09-29 05:03:28 CEST, end at Thu 2019-03-21 18:18:38 CET. --
Mar 20 13:32:03 debby2017 systemd-journald[699]: System journal (/var/log/journal/f8b692c8bb791fe2804f3d5a5905148b) is 1.8G, max 1.8G, 0B free.

$ journalctl --disk-usage
Archived and active journals take up 1.8G in the file system.

However, with systemd-journald persistence enabled, and classic logging daemons not removed (and still logging) on upgrades, you can end up with much data stored in /var/log, a lot more than there used to be in the past, which could cause problems, also (but not only) if combined with bug 1785321.

Revision history for this message
Tom Reynolds (tomreyn) wrote :

So I wasn't aware of this size hard limit on persistent storage - I just noticed that it is 'already' allocating 1.8GB, but this is indeed less than the 10% of the file system the man page says it will consume. So from my personal point of view this is a non-issue then (ideally Benjamin will also provide feedback, though) - but the new behavior may be worth discussing in release notes.

Revision history for this message
Emanuele (emanuc) wrote :

Everything ok for me too

$ journalctl -b -u systemd-journald.service -n 1
-- Logs begin at Tue 2019-02-19 00:29:40 CET, end at Thu 2019-03-21 22:33:40 CET. --
mar 21 18:33:53 emanuc systemd-journald[602]: System journal (/var/log/journal/c60f3deceebd474aa97aecbd13ebf6b9) is 744.0M, max 4.0G, 3.2G free.
$ journalctl --disk-usage
Archived and active journals take up 744.0M in the file system.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

I have filed the usability issue of --disk-usage upstream at https://github.com/systemd/systemd/issues/12063

summary: - systemd journals take up too much space, aren't vacuumed automatically
+ systemd journals take up a lot of space, and it's not obvious how much
+ is used, and what the upper limit is.
description: updated
tags: added: rls-dd-notfixing
removed: rls-dd-incoming
Changed in systemd (Ubuntu):
status: Incomplete → Confirmed
importance: High → Wishlist
Changed in systemd:
status: Unknown → New
Revision history for this message
Coeur Noir (coeur-noir) wrote :
Revision history for this message
Benjamin Bach (benjaoming) wrote :

Same issue on Bionic, installed May 15 (2 months ago):

➜ ~ sudo journalctl --disk-usage
Archived and active journals take up 4.1G in the file system.

My logs are definitely clogged with endless JavaScript errors from gnome-shell. Something I would not expect to slowly consume all disk space.

What could be helpful feedback for this issue?

Revision history for this message
Tom Reynolds (tomreyn) wrote :

So, as has been previously stated here, and as is documented in section 5 of the journald.conf man page (install man-db, run 'man 5 journald.conf'), journald will, by default, limit the journals' disk allocation to either a maximum of 10% of available disk space, or to the absolute size of 4GB, whichever is lower.

Example 1: You have a 120 GB disk, 70 GB are in use, leaving 50 GB. 10% of 50 GB are 5 GB, but the absolute value of 4 GB is lower, so journald will at most take 4 GB storage on this system (and dynamically less, if the disk grows fuller).

Example 2: You have a 20 GB disk, 12 GB are in use, leaving 8 GB. 10% of 8 GB are 800 MB, which is less than 4 GB, so journald will at most take 800 MB storage on this system (and dynamically more or less, if disk allocation changes).

Benjamin is now reporting 4.1 GB allocated, which is more than 4 GB, and with the details above, this is unexpected. What could be the reason? These limits are applied both when journald starts, and when vacuuming takes place. So if Benjamins' logs have grown by 100 MB since journald started, this can easily explain this situation. They will be reduced to 4 GB (or less, if the disk runs full) again next time vacuuming takes place (such as when journald is restarted).

So unless these limits are considered a problem (why? disk space is cheap, and you can modify them as needed if you're on a special (embedded?) platform), I don't see a problem here.

Revision history for this message
Coeur Noir (coeur-noir) wrote :

disk space is cheap → this becomes an infuriating answer. Is cheap where ? For who ?
Whatever price it is, one might want to use it for something else than log ( on a desktop pc ).

Here https://answers.launchpad.net/ubuntu/+source/systemd/+question/691873 is another example of what may happen. Less than 4GB but above 10% of remaining space. This led to system complaining about storage.

Shouldn't this « journal » be set by default to free space automatically on desktop computer ?

Balint Reczey (rbalint)
Changed in systemd (Ubuntu):
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.