[fuel-library] Incorrect logrotate configuration leads to lack of free disk space on master node

[Miroslav Anashkin]

Logs on master node may take all available free space.
We have several customer reports on such behavior

We need to add some common logrotate action, based on remained free space.

At first glance the following configuration files for logrotate on master node contain the incorrect settings:
/etc/cron.hourly/logrotate on root partition - points to not existing /etc/logrotate.d/20-fuel.conf

Same misconfiguration exists inside rsyslog container.

[Bogdan Dobrelya (bogdando)]

AFAIK, logrotate templates cannot be changed on fly out of box. I could suggest the following options to do it:
1) by CM tools (e.g. puppet agent runs) which will update /etc/logrotate.d/* as appropriate
2) by life cycle management tools, including monitoring + notifications + some triggers
3) by improving pre-deployment planning for logs storage and rotation policy as well ( there is a blueprint to address it https://blueprints.launchpad.net/fuel/+spec/manage-logs-with-free-space-consideration )

[Bogdan Dobrelya (bogdando)]

Miroslav, please elaborate what is
"At first glance the following configuration files for logrotate on master node contain the incorrect settings:
/etc/cron.hourly/logrotate on root partition - points to not existing /etc/logrotate.d/20-fuel.conf

Same misconfiguration exists inside rsyslog container."

[Bogdan Dobrelya (bogdando)]

Ok, I can see it - /etc/cron.hourly/logrotate should be fixed, indeed

[Bogdan Dobrelya (bogdando)]

20-fuel*.conf is actually a hourly scheduled rotation template. If it never called, like now, there is a chance that all free space of /var/log/ would be filled within a one day time frame - then next scheduled daily rotation is awaiting its time for execution.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/126601

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/126602

[Miroslav Anashkin (manashkin)]

Just verified - 5.0.1 has the same issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/126828

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/126601
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=d7497cfad51701717df890ea247a4b69584c14e6
Submitter: Jenkins
Branch: master

commit d7497cfad51701717df890ea247a4b69584c14e6
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Oct 7 17:29:15 2014 +0200

    Fix hourly log rotation schedule

    W/o this patch, new log rotation template for
    Fuel 5.x master node with docker (which is
    /etc/logrotate.d/20-fuel-docker.conf) would
    never be scheduled for execution.
    And that could have caused master node's /var/log/
    free space exhausted until the next planned daily
    log rotation will come.

    DocImpact
    Closes-bug: #1378327

    Change-Id: I21ba3debcfda4d94d5e25786f97996f6431e5ec3
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/126602
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=76cdbb798d26f39711d776121f49f42a2a3b068a
Submitter: Jenkins
Branch: stable/5.1

commit 76cdbb798d26f39711d776121f49f42a2a3b068a
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Oct 7 17:29:15 2014 +0200

    Fix hourly log rotation schedule

    W/o this patch, new log rotation template for
    Fuel 5.x master node with docker (which is
    /etc/logrotate.d/20-fuel-docker.conf) would
    never be scheduled for execution.
    And that could have caused master node's /var/log/
    free space exhausted until the next planned daily
    log rotation will come.

    DocImpact
    Closes-bug: #1378327

    Change-Id: I21ba3debcfda4d94d5e25786f97996f6431e5ec3
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/126828
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=a6737e68030cd92ef13bb74f176b74294139cb50
Submitter: Jenkins
Branch: stable/5.0

commit a6737e68030cd92ef13bb74f176b74294139cb50
Author: Bogdan Dobrelya <email address hidden>
Date: Tue Oct 7 17:29:15 2014 +0200

    Fix hourly log rotation schedule

    W/o this patch, new log rotation template for
    Fuel 5.x master node with docker (which is
    /etc/logrotate.d/20-fuel-docker.conf) would
    never be scheduled for execution.
    And that could have caused master node's /var/log/
    free space exhausted until the next planned daily
    log rotation will come.

    Closes-bug: #1378327

    Change-Id: I21ba3debcfda4d94d5e25786f97996f6431e5ec3
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Alexander Bozhenko (alexbozhenko)]

When running logrotate manually with this command
logrotate -f /etc/logrotate.conf
we can observe this error
error: 20-fuel-docker.conf:3 duplicate log entry for /var/log/kern.log

To avoid this:
add to /etc/logrotate.conf before(this is important) 'include /etc/logrotate.d' directive this line:
tabooext + .ignoredaily

After that
mv /etc/logrotate.d/20-fuel-docker.conf /etc/logrotate.d/20-fuel-docker.conf.ignoredaily

Now logrotate will ignore file with hourly log rotation options /etc/logrotate.d/20-fuel-docker.conf.ignoredaily, and we get rid of "duplicate log entry" message.

I think this modification should be added both to fuel master node and to rsyslog container.

[Bogdan Dobrelya (bogdando)]

Good point, thank you

[Matthew Mosesohn (raytrac3r)]

Alexander, please make a new bug with your comments in #12. This won't solve any issues with log rotation, but it will suppress some errors in logs. That makes it a low priority bug.

[Anastasia Palkina (apalkina)]

Verified on ISO #19

"build_id": "2014-11-17_21-00-23", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "19", "auth_required": true, "api": "1.0", "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1", "production": "docker", "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9", "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de", "feature_groups": ["mirantis"], "release": "5.1.1", "release_versions": {"2014.1.3-5.1.1": {"VERSION": {"build_id": "2014-11-17_21-00-23", "ostf_sha": "64cb59c681658a7a55cc2c09d079072a41beb346", "build_number": "19", "api": "1.0", "nailgun_sha": "2fcab95dc43a248ba867065e96ab764ee73882d1", "production": "docker", "fuelmain_sha": "ff22ca819e6eb7c63b6d7978fdd80ef9b84457d9", "astute_sha": "fce051a6d013b1c30aa07320d225f9af734545de", "feature_groups": ["mirantis"], "release": "5.1.1", "fuellib_sha": "add3fdd3e2af57b20dbb73a6bc53a9ccc4701c9a"}}}, "fuellib_sha": "add3fdd3e2af57b20dbb73a6bc53a9ccc4701c9a"

[Anastasia Palkina (apalkina)]

Verified on ISO #114

"build_id": "2014-11-18_22-00-23", "ostf_sha": "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068", "build_number": "114", "auth_required": true, "api": "1.0", "nailgun_sha": "b0add09c4361fee8fc70637c9a6ef42fbe738abe", "production": "docker", "fuelmain_sha": "e556f0e1b00c30ec5c4b374ca2878c047c8686c2", "astute_sha": "65eb911c38afc0e23d187772f9a05f703c685896", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-11-18_22-00-23", "ostf_sha": "82465a94eed4eff1fc8d8e1f2fb7e9993c22f068", "build_number": "114", "api": "1.0", "nailgun_sha": "b0add09c4361fee8fc70637c9a6ef42fbe738abe", "production": "docker", "fuelmain_sha": "e556f0e1b00c30ec5c4b374ca2878c047c8686c2", "astute_sha": "65eb911c38afc0e23d187772f9a05f703c685896", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "5a5275370b33ab3b9a403728a1c7ad173289e4a0"}}}, "fuellib_sha": "5a5275370b33ab3b9a403728a1c7ad173289e4a0"

[Baboune (seyvet)]

Should there be a different bug report on /var being full resulting in failed/corrupted docker containers as a result?

It seems to me that log rotation is not sufficient. At least not if there is any chance of /var getting full.

A more permanent solution would be to allocate different space for logging and containers, so that one can not corrupt the other?

[Alexander Bozhenko (alexbozhenko)]

File system corruption in containers on master node:
https://bugs.launchpad.net/fuel/+bug/1383741

[Bogdan Dobrelya (bogdando)]

@Baboune, yes, logs storage (/var) should be planned carefully and it would be nice if Fuel could address it (see related BP https://blueprints.launchpad.net/fuel/+spec/manage-logs-with-free-space-consideration).

And of cause, logs storage should be separated from any statefull / containers' data.

[Baboune (seyvet)]

How to apply this fix on a 5.1 running environment?

$ find / -name "logrotate-hourly"

/etc/puppet/2014.1.1-5.0.2/modules/anacron/files/logrotate-hourly
/etc/puppet/2014.1.1-5.1/modules/anacron/files/logrotate-hourly
/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/aaec9d27982a7dacdc6ff6a73aced11ce25105765d30e563f3f834ee2a9144aa/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/d5406e8d52cbb3611788cec6401d8fe23d4e06410be25436be0887fb105b3ed2/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/ff720279a8fcc16643946a29c22e6ea27be5c615835b0c3ed336c549b6da8233/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/56f78ed246849a774867b8fcb12c7de918f7b54e937f5982828c5c64e45eb4c8/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/13308458cdd9f1b572d21ca40f14c467c8f45db620dbaa508f442d02886b318c/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/d701157cd6614bdb122c3e42bde6786609859b537a7d1b62b6b2355598343a0d/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/1f63a20da5fda081c7d0a9a24a01cb0e5f21541b0de9b8441b133460b7e73424/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/dd861b355c2cbc706be9accf0f93749ba7a5574fce286b5d5330e8f70e8bc4a9/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/379298a33f353ba2ea33760aafc25c20ba53f4861d471b10f9043ab19cb75eb4/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/c667991c77f179982e73218121a6053eb760c392915262ed75a3a97cd0c87053/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly
/var/lib/docker/devicemapper/mnt/009535254f30cc0bb2f2819dc4cfb16d9957c91ab9bf02b1e08974e79f091775/rootfs/etc/puppet/modules/anacron/files/logrotate-hourly

[Bogdan Dobrelya (bogdando)]

@Baboune, in order to apply it manually, first, update /etc/cron.hourly/logrotate file at the master node as it is suggested in the patch. Next, make sure you have some free space at /var partition and run "/usr/sbin/logrotate -fv /etc/logrotate.d/20-fuel*.conf"