Controller mongodb logs aren't rotated

[Barry Price]

Modern Juju ships the mongo component as a snap.

But it doesn't look like any kind of logrotate config is set up by default.

I recently came across a Juju 2.9.32 HA controller unit (amd64, running Ubuntu 20.04) with a 71GB mongodb.log:

ubuntu@juju-789ae5-controller-0:/var/snap/juju-db/common/logs$ ls -lh
total 75G
-rw------- 1 root root 71G Nov 20 16:01 mongodb.log
ubuntu@juju-789ae5-controller-0:/var/snap/juju-db/common/logs$

Ideally these would be automatically rotated and compressed.

This is with the current latest/stable juju-db snap at time of writing:

latest/stable: 4.0.27 2021-10-07 (83) 64MB -

[Joseph Phillips (manadart)]

There is a work-around available since this patch landed:
https://github.com/juju/juju/pull/14285

kill -SIGUSR1 <PID for juju-db>

[Nick Moffitt (nick-moffitt)]

Why is this marked "Wishlist"? An overflowing log file will consume all disk and cause a crash/corruption of our juju database. It's only through extreme vigilance that we're able to spend SRE time working around this.

[Loïc Gomez (kotodama)]

This bit us again today on a 2.9.42 controller.

Logs were rotated somehow, but then a 38GB file was sitting here uncompressed:
$ sudo ls /var/snap/juju-db/common/logs/ -lhtr
total 38G
-rw-r--r-- 1 root root 38G Mar 13 01:55 mongodb.log.2023-03-13T01-55-56
-rw------- 1 root root 19M Mar 14 21:35 mongodb.log.2023-03-14T21-35-06
-rw------- 1 root root 2.8K Mar 14 21:35 mongodb.log.2023-03-14T21-35-07
-rw------- 1 root root 68M Mar 22 05:33 mongodb.log.2023-03-22T05-33-41
-rw------- 1 root root 16K Mar 22 05:35 mongodb.log

I believe a proper log rotation with compression is required, either through a logrotate setup or internal processing.

This is particularly problematic on controllers having debug enabled. When enabling debug, we usually want to troubleshoot problems, which takes a while, and getting disk space issues on top of these is not the best.

Thank you!

[Dagmawi Biru (dagbiru)]

This is a serious issue that can quickly incapacitate a controller(s) and render an environment inoperable from Juju's perspective. I don't agree with the determination that this is a wishlist but rather a critical problem that needs to be addressed as we've had many examples of environments going down due to unregulated log file sizes.

[Alan Baghumian (alanbach)]

I also see this happening...

[Victor Vavan (victorvavan)]

+1

This also happened to a customer today. Their root partition has been filled up to 100% due to a 76GB file in juju-db. Access to the controller via SSH is also intermittently affected due to this issue.

Ubuntu 20.04
Juju 2.9.44
Juju-db 4.4.18

[Nicolas Bock (nicolasbock)]

Submitted https://github.com/juju/juju-db-snap/pull/38

[Joseph Phillips (manadart)]

Assigned to Caner to test the patch and approve if OK. Looks fine to me.

[Caner Derici (cderici)]

Tested against Juju 2.9.49 with custom snap built with this change, confirming the rotation, green light from me 👍