Rotated logs are not compressed

[Jacek Nykis]

I have multiple environments where juju rotates logs but never compresses or deletes them.

For example files like this are created:
machine-0-2015-08-23T23-51-25.014.log

And the stay there forever as far as I can see (at least long enough to trigger disk space alerts).

I saw this behaviour with juju 1.24.2-precise-amd64 and 1.24.5-precise-amd64.

Once rotated logs should be compressed to save disk space and also there should be sane retention policy in place. For amount of data juju logs I think weekly rotation with 2-3 months retention is good default.

[Aaron Bentley (abentley)]

This was supposed to have been fixed in 1.24beta3, so since it's being reported in 1.24.2+, the fix must be incomplete.
https://bugs.launchpad.net/juju-core/1.24/+bug/1452285

[Aaron Bentley (abentley)]

Oh, I see. Rotation is happening, but not compression or removal.

[Anastasia (anastasia-macmood)]

This is definitely not an issue for Juju 2.x

As for Juju 1.x... As this is not considered "Critical", I am marking it as 'Won't Fix".

[Jacek Nykis (jacekn)]

Hi Anastasia,

Why do you think it's not an issue for Juju 2.x?

I can certainly see logs growing on my juju 2.0.1 controller. They are not compressed in any way:
/var/log/juju$ ls -lh logsink*
-rw------- 1 syslog syslog 300M Oct 11 07:15 logsink-2016-10-11T07-15-12.443.log
-rw------- 1 syslog syslog 300M Nov 9 08:24 logsink-2016-11-09T08-24-35.019.log
-rw------- 1 syslog syslog 287M Nov 22 09:36 logsink.log

[Ryan Beisner (1chb1n)]

I have an existing, long-running 1.25.x deployment, which has a controller that fills the disk and subsequently falls over. To me, this is a critical issue.

[Vance Morris (vmorris)]

confirming, my longest running juju 2 controller is not compressing logs:

/var/log/juju$ ls -lah
total 702M
drwxr-xr-x 2 syslog adm 4.0K Nov 21 16:15 .
drwxrwxr-x 8 root syslog 4.0K Nov 22 06:25 ..
-rw------- 1 syslog syslog 0 Sep 23 21:20 audit.log
-rw------- 1 syslog syslog 300M Nov 19 00:25 logsink-2016-11-19T00-25-07.284.log
-rw------- 1 syslog syslog 300M Nov 21 16:15 logsink-2016-11-21T16-15-26.872.log
-rw------- 1 syslog syslog 100M Nov 22 13:26 logsink.log
-rw------- 1 syslog syslog 1.4M Oct 20 18:46 machine-0.log

[Menno Finlay-Smits (menno.smits)]

The machine, unit and logsink log files are rotated at 300MB and the 2 most recent rotated copies of each are kept. This limits the maximum space for each log file and the rotated copies to 900MB.

We should look at compressing the rotated copies though.

FYI, the logsink files only exist on the controllers and contain the logs received by that particular API server from other agents. It exists as a last resort way of getting at logs in case the logs in mongodb aren't available.

[Jacek Nykis (jacekn)]

Also remember that any sane production environment will include subordinate services. In my case controller model has nrpe, ntp, livepatch, landscape and telegraf.

Every one of them creates log in /var/log/juju so if they use the same rotaion strategy it will be 5GB in total.

[Tom Haddon (mthaddon)]

There's a PR to add support for this to lumberjack here https://github.com/natefinch/lumberjack/pull/43

[Tim Penhey (thumper)]

https://github.com/juju/juju/pull/7430

[Joel Sing (jsing)]

Compression support has been merged into upstream lumberjack, so it should now just be a case of getting that pulled into Juju deps and enabled via config.