[swarm 10.0] /usr/bin/fuel-logrotate doesn't work correctly

[Alexandra]

Failed test:
https://product-ci.infra.mirantis.net/view/10.0_swarm/job/10.0.system_test.ubuntu.logrotate/145/testReport/(root)/test_fuel_nondaily_rotation/test_fuel_nondaily_rotation/

Scenario:

1. Revert snapshot "empty"
2. Check free disk space under /var/log, check free inodes
3. Generate 2GB /var/log/ostf-test.log size file
4. Run /usr/bin/fuel-logrotate
5. Check free disk space, check free inodes

Actual result:
Free space before rotation: 26.9G, after rotation 26.9G

[Alexandra (aallakhverdieva)]

iso 10.0 #839

[Alexandra (aallakhverdieva)]

New fail: https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.logrotate/149/testReport/(root)/test_fuel_nondaily_rotation/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/396234

[Oleksiy Molchanov (omolchanov)]

Should be fixed now, we have updated mysql package.

[Oleksiy Molchanov (omolchanov)]

Disregard my last message, it is not related to this issue.

[Oleksiy Molchanov (omolchanov)]

QA team, can you check this test case? It is not supposed to work at all:

1. We do not have a rule for rotating this filename in fuel.nodaily
2. Rotation not always leads to increasing of free space

[Dmitry Belyaninov (dbelyaninov)]

fuel-logrotate works fine on 9.x
[root@nailgun log]# ll -h | grep ostf
drwxr-xr-x. 2 root root 4.0K Dec 21 09:16 ostf
-rw-r--r--. 1 root root 43K Dec 21 09:37 ostf.log
-rw-r--r--. 1 root root 2.0G Dec 22 07:53 ostf-test.log
[root@nailgun log]#
[root@nailgun log]# /usr/bin/fuel-logrotate
[root@nailgun log]#
[root@nailgun log]# ll -h | grep ostf
drwxr-xr-x. 2 root root 4.0K Dec 21 09:16 ostf
-rw-r--r--. 1 root root 43K Dec 21 09:37 ostf.log
-rw-r--r--. 1 root root 0 Dec 22 07:53 ostf-test.log
-rw-r--r--. 1 root root 2.0M Dec 22 07:53 ostf-test.log.1.gz
[root@nailgun log]#

but not on 10
[root@nailgun log]# ll -h | grep ostf
drwxr-xr-x. 2 root root 4.0K Dec 21 14:40 ostf
-rw-r--r--. 1 root root 22K Dec 21 14:40 ostf.log
-rw-r--r--. 1 root root 2.0G Dec 21 14:52 ostf-test.log
[root@nailgun log]#
[root@nailgun log]# /usr/bin/fuel-logrotate
[root@nailgun log]#
[root@nailgun log]# ll -h | grep ostf
drwxr-xr-x. 2 root root 4.0K Dec 21 14:40 ostf
-rw-r--r--. 1 root root 22K Dec 21 14:40 ostf.log
-rw-r--r--. 1 root root 2.0G Dec 21 14:52 ostf-test.log
[root@nailgun log]#

As I can see there are no test differencies 9.x / 10

[Dmitry Belyaninov (dbelyaninov)]

The reason is missed ostf key in fuel.nodaily

9.x
[root@nailgun log]# grep ostf /etc/logrotate.d/fuel.nodaily
"/var/log/docker-logs/ostf*.log"
"/var/log/docker-logs/ostf/*.log"
"/var/log/ostf*.log"
[root@nailgun log]#

10
[root@nailgun log]# grep ostf /etc/logrotate.d/fuel.nodaily
[root@nailgun log]#

[Alexey Stupnikov (astupnikov)]

ostf logs are not the only one affected, we need to restore some logrotate targets removed at https://review.openstack.org/#/c/304735 step.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/424278

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/424278
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=f023c0bb23a2c134feea654a968b0cc06dce78ca
Submitter: Jenkins
Branch: master

commit f023c0bb23a2c134feea654a968b0cc06dce78ca
Author: Alexey Stupnikov <email address hidden>
Date: Mon Jan 23 21:12:37 2017 +0300

    Fix logrotate configuration on master node

    A docker-related code and confguration was removed in patch
    https://review.openstack.org/#/c/304735/. At this point a single
    configuration template is used to configure fuel-logrotate job
    on both master and slaves; old master configuration template
    was removed. As a result, som log files are not rotated.

    Here is a short list of log files to configure:
    - /var/log/astute/*.log
    - /var/log/dnsmasq.log
    - /var/log/mcollective.log (changed to /var/log/mcollective*.log)
    - /var/log/ostf*.log
    - /var/log/ostf/*.log
    - /var/log/rsync.log
    - /var/log/fuelmenu.log
    - /var/log/nailgun/*.log (we use separate template to configure
      nailgun)

    Change-Id: Ie5dfe086ece06421045063505199681abb9f6504
    Closes-bug: #1631917

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/429662

[Dmitry Belyaninov (dbelyaninov)]

@Alexey- any problems with review?

[Alexey Stupnikov (astupnikov)]

@dbelyaninov looks like everything is ok. Fuel team is too busy at the moment, I will ask them to check it again.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/429662
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4e1ee9f0234c8602bf96bb94f80d5e826894237e
Submitter: Jenkins
Branch: stable/newton

commit 4e1ee9f0234c8602bf96bb94f80d5e826894237e
Author: Alexey Stupnikov <email address hidden>
Date: Mon Jan 23 21:12:37 2017 +0300

    Fix logrotate configuration on master node

    A docker-related code and confguration was removed in patch
    https://review.openstack.org/#/c/304735/. At this point a single
    configuration template is used to configure fuel-logrotate job
    on both master and slaves; old master configuration template
    was removed. As a result, som log files are not rotated.

    Here is a short list of log files to configure:
    - /var/log/astute/*.log
    - /var/log/dnsmasq.log
    - /var/log/mcollective.log (changed to /var/log/mcollective*.log)
    - /var/log/ostf*.log
    - /var/log/ostf/*.log
    - /var/log/rsync.log
    - /var/log/fuelmenu.log
    - /var/log/nailgun/*.log (we use separate template to configure
      nailgun)

    Change-Id: Ie5dfe086ece06421045063505199681abb9f6504
    Closes-bug: #1631917
    (cherry picked from commit f023c0bb23a2c134feea654a968b0cc06dce78ca)

[Vladimir Khlyunev (vkhlyunev)]

https://product-ci.infra.mirantis.net/job/10.0.system_test.ubuntu.logrotate/198/testReport/ still fails

File was rotated but not compressed:
[root@nailgun ~]# ll /var/log/ostf-test.log-20170302-1488492686 -h
-rw-r--r--. 1 root root 2.0G Mar 2 22:11 /var/log/ostf-test.log-20170302-1488492686

[Alexey Stupnikov (astupnikov)]

@vkhlyunev it is not a fuel issue, since developers added 'delaycompress' statement to fuel.nodaily during newton cycle. It looks like fuel-qa code should be changed to test this settings correctly. I suggest you to open another bug because of this issue, since original logrotate-related issue was solved.

[Vladimir Khlyunev (vkhlyunev)]

Verified at 1575 build

[Alexey Stupnikov (astupnikov)]

The same issue came up in Fuel9: services were moved from docker containers, but they still had incorrect logrotate targets.

https://github.com/openstack/fuel-library/blob/stable/mitaka/deployment/puppet/openstack/templates/10-fuel-docker.conf.erb#L13-L25

[Alexey Stupnikov (astupnikov)]

stable/mitaka patch: https://review.openstack.org/#/c/459387/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/459387
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=0f27fc81a1cfab49647b09e9ff50f5d7d9f006c9
Submitter: Jenkins
Branch: stable/mitaka

commit 0f27fc81a1cfab49647b09e9ff50f5d7d9f006c9
Author: Alexey Stupnikov <email address hidden>
Date: Mon Apr 24 18:47:08 2017 +0300

    Remove duplicate logrotate targets

    The following duplicate logrotate targets were removed:

    - cobbler logs were removed from fuel.nodaily template since
      they fully duplicate targets shipped with cobbler package
    - httpd: same as above
    - nginx: same as above
    - rabbitmq: same as above
    - nailgun: a puppet template used to create separate logrotate
      configuration file for nailgun logs was modified to include
      all nailgun logs. Duplicate log target was removed from
      fuel.nodaily template.

    I have also removed old docker-logs targets to clean up the mess.

    Change-Id: Ie4129d1fd7dc35716530dfc3a78019e28c9d96d0
    Closes-bug: #1656305
    Closes-bug: #1631917

[Vladimir Jigulin (vjigulin)]

Verified on 9.2 + proposed

Without patch:
[root@nailgun ~]# grep -r docker /etc/logrotate.d
/etc/logrotate.d/fuel.nodaily:"/var/lib/docker/containers/*/*.log"
/etc/logrotate.d/fuel.nodaily:"/var/log/docker"
/etc/logrotate.d/fuel.nodaily:"/var/log/docker-*.log"
/etc/logrotate.d/fuel.nodaily:"/var/log/docker-logs/astute/*.log"
/etc/logrotate.d/fuel.nodaily:"/var/log/docker-logs/cobbler/*.log"
/etc/logrotate.d/fuel.nodaily:"/var/log/docker-logs/dnsmasq.log"
/etc/logrotate.d/fuel.nodaily:"/var/log/docker-logs/httpd/*.log"
/etc/logrotate.d/fuel.nodaily:"/var/log/docker-logs/mcollective.log"
/etc/logrotate.d/fuel.nodaily:"/var/log/docker-logs/messages"
............>

With patch:
[root@nailgun ~]# grep -r docker /etc/logrotate.d
/etc/logrotate.d/fuel.nodaily:"/var/lib/docker/containers/*/*.log"
/etc/logrotate.d/fuel.nodaily:"/var/log/docker"
/etc/logrotate.d/fuel.nodaily:"/var/log/docker-*.log"