Amphorae need log rotation based on size

[Connor Chamberlain]

In the case of an amphora that has a significant amount of traffic, it is possible for its root disk to fill with logs before they get properly rotated, resulting in a full disk which requires a failover to resolve. It looks like current logrotate policies are acting based on elapsed time. Expanding this functionality to rotate logs when a certain log file size or space availability threshold has been reached would be useful.

In my particular case, a loadbalancer with heavy traffic saw /var/log/haproxy.log completely fill the disk in about 24 hours, before this log had been rotated even a single time.

[Michael Johnson (johnsom)]

Yes, we wanted to do this, however some distros (ubuntu and redhat for example) have disabled logrotate from running any more often than once a day, so if we switch it to "by size", cron will only trigger it once a day, which actually makes the problem worse in that it may be slighly under the size at the daily rotate, and then wait an additional 24 hours before the size is checked again.

At one point we tried just adding logrotate back into an hourly check, but we found that can impact other log rotate configurations. This may have been fixed and it may no longer be an issue, so it is probably a good time to try this again.

This dropped in priority to run down because it needs to be a very active load balancer (usually benchmarking) to fill it and as on operator told us "disk is cheap and thin provisioned, I just give the amps 20GB and forget about it.". We also implemented log offloading which allows you to disable log storage inside the amphora and simply ship them off to a central log storage location (which gives you a performance bump for the load balancer as it removes the disk io overhead).

We would welcome a patch if you are interested in working on this and testing that running logrotate more often doesn't cause any problems with the other log rotate configurations (from the distro) defined in the amphora.