Logs from discovered but not provisioned nodes are not rotated

[Michal Skalski]

# fuel --f
api: '1.0'
astute_sha: f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
auth_required: true
build_id: 2014-09-17_21-40-34
build_number: '11'
feature_groups:
- mirantis
fuellib_sha: d9b16846e54f76c8ebe7764d2b5b8231d6b25079
fuelmain_sha: 8ef433e939425eabd1034c0b70e90bdf888b69fd
nailgun_sha: eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d
ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
production: docker
release: '5.1'
release_versions:
  2014.1.1-5.1:
    VERSION:
      api: '1.0'
      astute_sha: f5fbd89d1e0e1f22ef9ab2af26da5ffbfbf24b13
      build_id: 2014-09-17_21-40-34
      build_number: '11'
      feature_groups:
      - mirantis
      fuellib_sha: d9b16846e54f76c8ebe7764d2b5b8231d6b25079
      fuelmain_sha: 8ef433e939425eabd1034c0b70e90bdf888b69fd
      nailgun_sha: eb8f2b358ea4bb7eb0b2a0075e7ad3d3a905db0d
      ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
      production: docker
      release: '5.1'

Logs for discovered but not provisioned nodes are not rotated, and their size constantly growing if nodes don't have assigned roles. Currently pattern in logrotate configuration is:

"/var/log/remote/*-*/*.log"
"/var/log/remote/*-*/*/*.log"

which is responsible for rotating logs in catalogs with name the same as domain name of node ex. node-1.domain.tld - please see more details here: https://bugs.launchpad.net/fuel/+bug/1329991.
Rsyslog create catalogs with names the same as senders IPs. Naligun rename this catalogs to corresponding domain names but only after node have assigned role.
I propose to use pattern like this:

 "/var/log/remote/[1-9]*/*.log"
"/var/log/remote/[1-9]*/*/*.log"

this will rotate logs for bootstrapped nodes (catalogs with IP addresses in name) and symlinks (created by nailgun, also IP addresses in name) pointing to domain name catalogs. This pattern will not match domain name catalogs so rotating files in one directory more then once during logrotate run will not occur.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/138397

[Vladimir Kuklin (vkuklin)]

I would say that this issue is not so severe as bootstrapped nodes do not provide a lot of data during their life and should not waste even 10 gigabytes of space for big environments.

[Michal Skalski (mskalski)]

After 10 days:
[root@fuel bootstrap]# pwd
/var/log/remote/192.168.0.126/bootstrap
[root@fuel bootstrap]# du -sh
866M .
[root@fuel bootstrap]# ls -lh
total 866M
-rw-r----- 1 root adm 237M Dec 3 10:48 agent.log
-rw-r----- 1 root adm 22K Nov 24 20:54 mcollective.log
-rw-r----- 1 root adm 628M Dec 3 10:48 messages.log

[Bogdan Dobrelya (bogdando)]

We do not backport medium bugs, but I believe it should be addressed in 6.0.1 as well

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/138397
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=57d5e260138b166857380c0f3ae98f92b120cd11
Submitter: Jenkins
Branch: master

commit 57d5e260138b166857380c0f3ae98f92b120cd11
Author: Michal Skalski <email address hidden>
Date: Tue Dec 2 16:12:40 2014 +0100

    Rotate logs from bootsraped nodes

    Logs for discovered but not provisioned nodes were not rotated because
    of pattern used in logrotate configuration. This commits will include
    logs from this kind of nodes in rotation process and still will prevent
    from rotating files in one directory more then once during logrotate run.

    Change-Id: I4bf3e6985b5771d17a6b98650dd3b3996a35346e
    Closes-Bug: 1398418