Logs were rotated somehow, but then a 38GB file was sitting here uncompressed:
$ sudo ls /var/snap/juju-db/common/logs/ -lhtr
total 38G
-rw-r--r-- 1 root root 38G Mar 13 01:55 mongodb.log.2023-03-13T01-55-56
-rw------- 1 root root 19M Mar 14 21:35 mongodb.log.2023-03-14T21-35-06
-rw------- 1 root root 2.8K Mar 14 21:35 mongodb.log.2023-03-14T21-35-07
-rw------- 1 root root 68M Mar 22 05:33 mongodb.log.2023-03-22T05-33-41
-rw------- 1 root root 16K Mar 22 05:35 mongodb.log
I believe a proper log rotation with compression is required, either through a logrotate setup or internal processing.
This is particularly problematic on controllers having debug enabled. When enabling debug, we usually want to troubleshoot problems, which takes a while, and getting disk space issues on top of these is not the best.
This bit us again today on a 2.9.42 controller.
Logs were rotated somehow, but then a 38GB file was sitting here uncompressed: juju-db/ common/ logs/ -lhtr log.2023- 03-13T01- 55-56 log.2023- 03-14T21- 35-06 log.2023- 03-14T21- 35-07 log.2023- 03-22T05- 33-41
$ sudo ls /var/snap/
total 38G
-rw-r--r-- 1 root root 38G Mar 13 01:55 mongodb.
-rw------- 1 root root 19M Mar 14 21:35 mongodb.
-rw------- 1 root root 2.8K Mar 14 21:35 mongodb.
-rw------- 1 root root 68M Mar 22 05:33 mongodb.
-rw------- 1 root root 16K Mar 22 05:35 mongodb.log
I believe a proper log rotation with compression is required, either through a logrotate setup or internal processing.
This is particularly problematic on controllers having debug enabled. When enabling debug, we usually want to troubleshoot problems, which takes a while, and getting disk space issues on top of these is not the best.
Thank you!