Perl is missing in kolla containers making logrotate_crond fail its duities

[Bogdan Dobrelya]

(subject)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/553795

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/553795
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=cc82b62f44641233c515e5b2920c87dda6c4f21e
Submitter: Zuul
Branch: master

commit cc82b62f44641233c515e5b2920c87dda6c4f21e
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Mar 16 14:52:45 2018 +0100

    Replace perl with awk

    Perl is missing in kolla containers. Replace it with
    awk.
    bz: #1553077

    Closes-Bug: #1756343

    Change-Id: Ie51bd1fa08d7690ac76a01ee2c558e86fb52bb2d
    Signed-off-by: Bogdan Dobrelya <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/554599

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/554993

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/554599
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=4105d44f53af826f09f2df5ce8e4302128b0280c
Submitter: Zuul
Branch: stable/queens

commit 4105d44f53af826f09f2df5ce8e4302128b0280c
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Mar 16 14:52:45 2018 +0100

    Replace perl with awk

    Perl is missing in kolla containers. Replace it with
    awk.
    bz: #1553077

    Closes-Bug: #1756343

    Change-Id: Ie51bd1fa08d7690ac76a01ee2c558e86fb52bb2d
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit cc82b62f44641233c515e5b2920c87dda6c4f21e)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/554993
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=a525507a9248c14cd810b918dbb844545fd4c6d1
Submitter: Zuul
Branch: stable/pike

commit a525507a9248c14cd810b918dbb844545fd4c6d1
Author: Bogdan Dobrelya <email address hidden>
Date: Fri Mar 16 14:52:45 2018 +0100

    Replace perl with awk

    Perl is missing in kolla containers. Replace it with
    awk.
    bz: #1553077

    Closes-Bug: #1756343

    Change-Id: Ie51bd1fa08d7690ac76a01ee2c558e86fb52bb2d
    Signed-off-by: Bogdan Dobrelya <email address hidden>
    (cherry picked from commit cc82b62f44641233c515e5b2920c87dda6c4f21e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 7.4.11 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 8.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/puppet-tripleo 9.0.0 release.

[Bogdan Dobrelya (bogdando)]

There is an issue with awk filter:

echo "httpd 3203 48 10w REG 252,1 2717 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)" | awk '/\S+\s+[0-9]+\s.*\/var\/log\/containers\/.*\(deleted\)/ {print $2}

returns nothing, while it is expected to return '3203'. That was the case with the original perl command:

echo "httpd 3203 48 10w REG 252,1 2717 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)" | /bin/perl -ne '/\S+\s+(\d+).*\/var\/log\/.*\(deleted\)/ && print "$1\n"'
3203

but an unfortunate mistake had sneaked in!

[Bogdan Dobrelya (bogdando)]

The root cause for the regression is I5029a4b9c76268455812696290aaf82f1a0c2c23

the filter is executed inside of the logrotate-crond container, like this way:

()[root@5c78303fb8c2 /]# /sbin/lsof -nPs +L1 +D /var/log/containers 2>&1 | awk '/\S+\s+[0-9]+\s.*\/var\/log\/.*\(deleted\)/ {print}'
neutron-s 2572 neutron 5w REG 252,1 234 0 8419523 /var/log/neutron/server.log.1 (deleted)
httpd 2862 root 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
glance-ap 3026 glance 3w REG 252,1 131 0 29510206 /var/log/glance/api.log.1 (deleted)
httpd 3192 keystone 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3193 keystone 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3194 keystone 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3194 keystone 11w REG 252,1 2853 0 96570982 /var/log/keystone/keystone.log.1 (deleted)
httpd 3195 keystone 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3195 keystone 11w REG 252,1 2853 0 96570982 /var/log/keystone/keystone.log.1 (deleted)
httpd 3197 48 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3198 48 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3199 48 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3200 48 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3201 48 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3202 48 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3203 48 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
httpd 3204 48 10w REG 252,1 3287 0 549003 /var/log/httpd/keystone_wsgi_admin_access.log.1 (deleted)
nova-api- 3810 nova 3w REG 252,1 129 0 26281737 /var/log/nova/nova-api-metadata.log.1 (deleted)
neutron-s 5129 neutron 5w REG 252,1 234 0 8419523 /var/log/neutron/server.log.1 (deleted)
neutron-s 5130 neutron 5w REG 252,1 234 0 8419523 /var/log/neutron/server.log.1 (deleted)
neutron-s 5131 neutron 5w REG 252,1 234 0 8419523 /var/log/neutron/server.log.1 (deleted)
neutron-s 5132 neutron 5w REG 252,1 234 0 8419523 /var/log/neutron/server.log.1 (deleted)

So you can see that the real path should be examined for deleted (open and unlined) logs is /var/log/ and not /var/log/containers. The latter is only used to apply the logrotation over the bind-mounted host path. It cannot affect any host logs outside of /var/log/containers, so the change I5029a4b9c76268455812696290aaf82f1a0c2c2...

[Bogdan Dobrelya (bogdando)]

So we really should track this additional fix in https://bugs.launchpad.net/tripleo/+bug/1776533 and not here, closing