Fuel master disk can fill because logrotate errors caused by symlinks and pattern match in /var/log/remote

[Terry Duncan]

The Fuel logrotate config /etc/logrotate.d/20-fuel.conf and 10-fuel.conf have a pattern match for the remote nodes
/var/log/remote/*/*.log

In /var/log/remote, there are symlinks created for each node
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.3 -> node-1.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.4 -> node-2.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.5 -> node-3.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.6 -> node-4.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.7 -> node-5.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.8 -> node-6.domain.tld
lrwxrwxrwx 1 root root 17 Jun 2 17:00 10.20.0.9 -> node-7.domain.tld

The pattern matches both the files in the symlink and in the actual directory producing duplicates. logrotate processes the first match in the symlinked dir and creates a log.1 file. When it comes around to the other match in the actual directory, there is already a .1 file which has not yet been compressed because of the processing in the symlinked dir. logrotate exits without completion with an error "file.1 exists". Further processing is aborted and files do not get rotated on subsequent runs. This eventually leads to a full file system.

A potential solution would be to have puppet write this pattern as /var/log/remote/{fuel-pxe-net}*/*.log substituting the network address for the pxe network.

[Terry Duncan (terry-s-duncan)]

This was seen on fuel 4.1 but I believe it is an issue in later releases as well.

[Bogdan Dobrelya (bogdando)]

The fix would be to use /var/log/remote/*-*/*.log pattern as well. To verify it, use logrotate -d <rotation_file_path>.
The fix assumes the node names in Fuel will always contain a node index separated from the "main name" by the "-" (e.g. node-100.foo.local, or mycontroller-2, anynode-24.some.name)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/100187

[Bogdan Dobrelya (bogdando)]

related bug https://bugs.launchpad.net/fuel/+bug/1316957

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/100188

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/4.1
Review: https://review.openstack.org/100189

[Bogdan Dobrelya (bogdando)]

Taking in account this https://blueprints.launchpad.net/fuel/+spec/node-naming, we should not rely for naming patterns, actually. Sadly, logrotate cannot either exclude paths nor filter them by regex. Thus, the only "long living" fix I see is to move symlynked names away from the /var/log/remote/*/ path.

Until we have this naming bp implemented, I suggest to use this *-* fix as well. And once we would have arbitrary given names for nodes - the Nailgun part should be updated in order to fix the paths for symlinks.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/100187
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5319075fb440b0880122f64af14056e3808f9d54
Submitter: Jenkins
Branch: master

commit 5319075fb440b0880122f64af14056e3808f9d54
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Jun 16 11:35:05 2014 +0300

    Fix /var/log/remote rotation template

    - Put /var/log/remote rotation into an appropriate server role template
    - Fix node names matching to exclude symlinks for IP addresses

    Closes bug: #1329991
    Closes bug: #1316957

    Change-Id: Iacd96d6fa02b0c63c63ce5256cc832eda229dcc7
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/100188
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=94cb3af924cffd4433d038280164208a41b48256
Submitter: Jenkins
Branch: stable/5.0

commit 94cb3af924cffd4433d038280164208a41b48256
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Jun 16 11:35:05 2014 +0300

    Fix /var/log/remote rotation template

    - Put /var/log/remote rotation into an appropriate server role template
    - Fix node names matching to exclude symlinks for IP addresses

    Closes bug: #1329991
    Closes bug: #1316957

    Change-Id: Iacd96d6fa02b0c63c63ce5256cc832eda229dcc7
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/100189
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=9a99b71d21717c5b278779df775cd6806a950108
Submitter: Jenkins
Branch: stable/4.1

commit 9a99b71d21717c5b278779df775cd6806a950108
Author: Bogdan Dobrelya <email address hidden>
Date: Mon Jun 16 11:41:43 2014 +0300

    Fix /var/log/remote rotation template

    Fix node names matching to exclude symlinks for IP addresses

    Closes bug: #1329991

    Change-Id: Iacd96d6fa02b0c63c63ce5256cc832eda229dcc7
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Dennis Dmitriev (ddmitriev)]

There are some more logs in the /var/log/remote/*-*/*/ subfolders that are not covered by logrotate rules at the moment, like these:

/var/log/remote/*-*/bootstrap/*.log
/var/log/remote/*-*/net/*.log
/var/log/remote/*-*/install/*.log
 and so on.

Do they need to be rotated?

[Bogdan Dobrelya (bogdando)]

I believe they don't. These logs are generated only once at bootstrap and provision stages and never get updated later

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/4.1
Review: https://review.openstack.org/103159

[Dennis Dmitriev (ddmitriev)]

Fixed:

4.1.2 : {"mirantis": "no", "ostf_sha": "a269d04abbe2daa3966577c25a30c4d13d4fe648", "nailgun_sha": "97f0589ce60e5c1f74ad81d421f0d3f614ec94e0", "production": "docker", "api": "1.0", "fuelmain_sha": "dd9434d327b1eea9fcfd060b35dd45caf446854e", "astute_sha": "694b5a55695e01e1c42185bfac9cc7a641a9bd48", "release": "4.1.2", "fuellib_sha": "b0a4dc5cea1594e8c7073f075867fe3be0292f6b"}

5.0.1 : {"build_id": "2014-06-27_03-01-14", "mirantis": "yes", "build_number": "73", "ostf_sha": "8ea3940654ff33e3e99a0182a2cf70704f836b99", "nailgun_sha": "dd7f32ab80c023a4afda70b521dd5391e5e464fd", "production": "docker", "api": "1.0", "fuelmain_sha": "ae0fd7b9f7db7ee974b0232f3d1c66d6c6f3cc49", "astute_sha": "9ab830ab3f90ede8b8bc2e0fffa663f339322079", "release": "5.0.1", "fuellib_sha": "1cf02c3a06edaae9278129f5a534d4c9d0c33784"}

5.1 : {"build_id": "2014-06-27_00-31-14", "mirantis": "yes", "build_number": "274", "ostf_sha": "a4978638de3951dbc229276608a839a19ece2b70", "nailgun_sha": "5f2944a8d5077a1c96acb076ba9194f670b818e8", "production": "docker", "api": "1.0", "fuelmain_sha": "bf8660309601cee2f8f3e1bb881d272e638dcffa", "astute_sha": "694b5a55695e01e1c42185bfac9cc7a641a9bd48", "release": "5.1", "fuellib_sha": "acc99fcd0ba9eeef0a504dc26507eb91ce757220"}

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/103159
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=23967f40ef814ed19e53e1724fc82b10462c29c7
Submitter: Jenkins
Branch: stable/4.1

commit 23967f40ef814ed19e53e1724fc82b10462c29c7
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Jun 27 17:57:19 2014 +0300

    Add missed audit logs to rotation

    The audit log is missing in rotation templates
    for stable/4.1 branch

    Related-bug: #1329991

    Change-Id: I383800f4388fabbe077f1896356ba5c49131c083
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[Meg McRoberts (dreidellhasa)]

Marked as "Fixed Issue" in 5.0.1 Release Notes.

[Bogdan Dobrelya (bogdando)]

The realted fix https://review.openstack.org/#/c/138397/ for related bug https://bugs.launchpad.net/fuel/+bug/1398418