Fuel for OpenStack

/etc/logrotate.d/fuel.nodaily is not using dateext directive, /etc/logrotate.conf is not using dateformat, which leads to broken log rotation, and filled up disk

  1. Series 8.0.x
  2. Bug #1618487
Bug #1618487 reported by Alexander Bozhenko
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Rodion Tikunov
Fuel for OpenStack 11.0
Nominated for Ocata by Oleksiy Molchanov
6.1.x
In Progress
High
Oleksiy Molchanov
Fuel for OpenStack 6.1-updates
7.0.x
In Progress
High
Oleksiy Molchanov
Fuel for OpenStack 7.0-updates
8.0.x
In Progress
High
Oleksiy Molchanov
Fuel for OpenStack 8.0-updates
Mitaka
Fix Released
High
Oleksiy Molchanov
Fuel for OpenStack 9.2
Newton
Fix Released
High
Oleksiy Molchanov
Fuel for OpenStack 10.1

Bug Description

In /etc/logrotate.d/fuel.nodaily we have this:

  # logrotate allows to use only year, month, day and unix epoch
  dateformat -%Y%m%d-%s

But dateext directive from /etc/logrotate.conf is not inherited.
So dateformat doesn't work.
Because of this we still see on customer's installations errors like these:
error: error creating output file /var/log/remote/10.189.168.10/nailgun-agent.log.1: File exists
error: error creating output file /var/log/remote/10.189.168.19/CRON.log.1: File exists

To avoid it we MUST HAVE both deteext and dateformat /etc/logrotate.d/fuel.nodaily and in /etc/logrotate.conf both on fuel master and slave nodes:

  # logrotate allows to use only year, month, day and unix epoch
  dateext
  dateformat -%Y%m%d-%s

All versions 6.1-9.0 are affected.

See original description
Alexander Bozhenko (alexbozhenko)
summary: - Fuel logrotate configuration is not using date suffix
+ fuel.nodaily logrotate configuration is not using date suffix
Revision history for this message
Maksim Malchuk (mmalchuk) wrote : Re: fuel.nodaily logrotate configuration is not using date suffix

the errors like:
>error: error creating output file /var/log/remote/10.189.168.19/CRON.log.1: File exists
 appears only from the previously failed run for example on the lack of free disk space for example.

Changed in fuel:
assignee: Fuel for Openstack (fuel) → Fuel Sustaining (fuel-sustaining-team)
importance: Undecided → High
status: New → Confirmed
milestone: none → 10.0
tags: added: area-library
Changed in fuel:
importance: High → Medium
status: Confirmed → Triaged
Revision history for this message
Alexander Bozhenko (alexbozhenko) wrote :

I also propose to add the same
dateext
dateformat -%Y%m%d-%s

to /etc/logrotate.conf both, on fuel master and slave nodes, to make it:
1. more universal
2. prevent 'File exists' errors

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/381067

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Richard Berwald (rberwald)
status: Triaged → In Progress
Alexander Bozhenko (alexbozhenko)
summary: - fuel.nodaily logrotate configuration is not using date suffix
+ fuel.nodaily logrotate configuration is not using date suffix, which
+ leads to broken log rotation, and filled up disk
Alexander Bozhenko (alexbozhenko)
description: updated
summary: - fuel.nodaily logrotate configuration is not using date suffix, which
- leads to broken log rotation, and filled up disk
+ /etc/logrotate.d/fuel.nodaily is not using dateext directive,
+ /etc/logrotate.conf is not using dateformat, which leads to broken log
+ rotation, and filled up disk
Revision history for this message
Alexander Bozhenko (alexbozhenko) wrote :

Related https://bugs.launchpad.net/fuel/+bug/1486690

Revision history for this message
Alexander Bozhenko (alexbozhenko) wrote :

I confirm that even without Sharedscripts config directive, on fuel 9.0 large deployment, logrotate is still broken with errors
error: error creating output file /path/to/file File exists

So this fix to include
  dateext
  dateformat -%Y%m%d-%s
to both /etc/logrotate.conf and /etc/logrotate.d/fuel.nodaily on both fuel master and slave nodes is very much needed.

Revision history for this message
Alexander Bozhenko (alexbozhenko) wrote :

Looks like Richard, author of the original patch, is no longer with the company(at least he is not in directory)

Changed in fuel:
assignee: Richard Berwald (rberwald) → Fuel Sustaining (fuel-sustaining-team)
Rodion Tikunov (rtikunov)
Changed in fuel:
status: In Progress → Confirmed
status: Confirmed → In Progress
Rodion Tikunov (rtikunov)
Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Rodion Tikunov (rtikunov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Rodion Tikunov (rtikunov) → Oleksiy Molchanov (omolchanov)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Sergii Golovatiuk (sgolovatiuk)
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

I'd raise that to high as this is a major UX bug (disks may be filled up by non rotated logs)

Changed in fuel:
importance: Medium → High
assignee: Sergii Golovatiuk (sgolovatiuk) → Richard Berwald (rberwald)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/381067
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=90419d9f8a413d19c3ceb1c1e8baebb94a92ee37
Submitter: Jenkins
Branch: master

commit 90419d9f8a413d19c3ceb1c1e8baebb94a92ee37
Author: R. Berwald <email address hidden>
Date: Mon Oct 3 13:59:05 2016 +0200

    Adjust logrotate configuration to use date suffix

    Change-Id: I268bf01030b2bfa5fc998d2479f90fe11212443a
    Closes-bug: #1618487

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/401186

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/401912

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/401186
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=fdcf1dd6369aeb3937e41eb80396ac1450560392
Submitter: Jenkins
Branch: stable/mitaka

commit fdcf1dd6369aeb3937e41eb80396ac1450560392
Author: R. Berwald <email address hidden>
Date: Mon Oct 3 13:59:05 2016 +0200

    Adjust logrotate configuration to use date suffix

    Change-Id: I268bf01030b2bfa5fc998d2479f90fe11212443a
    Closes-bug: #1618487
    (cherry picked from commit 90419d9f8a413d19c3ceb1c1e8baebb94a92ee37)

TatyanaGladysheva (tgladysheva)
tags: added: on-verification
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on 9.2 snapshot #572.

Steps to verify:
Launch 'logrotate -f /etc/logrotate.d/fuel.nodaily' a couple of times and check that archived logs in /var/log/ contain date suffix:
For example, for cinder-all.log:
-rw-r----- 1 syslog adm 459 Nov 30 07:32 cinder-all.log
-rw-r----- 1 syslog adm 192719 Nov 30 07:30 cinder-all.log-20161130-1480491010.gz
-rw-r----- 1 syslog adm 5049 Nov 30 07:32 cinder-all.log-20161130-1480491125

tags: removed: on-verification
Revision history for this message
Alexander Bozhenko (alexbozhenko) wrote :

@Rodion,

Tatyana send me files from 9.2 env with the fix, and here are my observations:
/etc/logrotate.conf on target nodes is missing dateext
/etc/logrotate.d/fuel.nodaily is missing dateext

Please update the patch to include both
dateext
dateformat -%Y%m%d-%s

to all files.
Thanks!

Oleksiy Molchanov (omolchanov)
Changed in fuel:
status: Fix Committed → Confirmed
assignee: Richard Berwald (rberwald) → Oleksiy Molchanov (omolchanov)
Revision history for this message
Oleksiy Molchanov (omolchanov) wrote :

Really, it is missing dateext in logrotate.conf, marking as Confirmed.

no longer affects: fuel/future
Changed in fuel:
assignee: Oleksiy Molchanov (omolchanov) → Rodion Tikunov (rtikunov)
milestone: 10.0 → 11.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/401912
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=349182fa7e91f6f2f74689f6d4b91950e9579aa0
Submitter: Jenkins
Branch: stable/newton

commit 349182fa7e91f6f2f74689f6d4b91950e9579aa0
Author: R. Berwald <email address hidden>
Date: Mon Oct 3 13:59:05 2016 +0200

    Adjust logrotate configuration to use date suffix

    Change-Id: I268bf01030b2bfa5fc998d2479f90fe11212443a
    Closes-bug: #1618487
    (cherry picked from commit 90419d9f8a413d19c3ceb1c1e8baebb94a92ee37)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/408505

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/408506

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/408508

Changed in fuel:
status: Confirmed → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/406889
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=bcb01593f7adb51aebbe9bab817fe2185b3cdd63
Submitter: Jenkins
Branch: master

commit bcb01593f7adb51aebbe9bab817fe2185b3cdd63
Author: R. Berwald <email address hidden>
Date: Mon Oct 3 13:59:05 2016 +0200

    Adjust logrotate configuration to use date suffix

    Change-Id: I7747b5b190a1509d7024ca7c3030c18eb419491d
    Closes-bug: #1618487

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/409100

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/409101

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/newton)

Reviewed: https://review.openstack.org/408505
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=6068c2bda19aab4fee5b0655c08e84127346f67a
Submitter: Jenkins
Branch: stable/newton

commit 6068c2bda19aab4fee5b0655c08e84127346f67a
Author: R. Berwald <email address hidden>
Date: Mon Oct 3 13:59:05 2016 +0200

    Adjust logrotate configuration to use date suffix

    Change-Id: I7747b5b190a1509d7024ca7c3030c18eb419491d
    Closes-bug: #1618487

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/408506
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5fb4c77dfc990ffd9dc35da6e0471ac602917290
Submitter: Jenkins
Branch: stable/mitaka

commit 5fb4c77dfc990ffd9dc35da6e0471ac602917290
Author: R. Berwald <email address hidden>
Date: Mon Oct 3 13:59:05 2016 +0200

    Adjust logrotate configuration to use date suffix

    Change-Id: I7747b5b190a1509d7024ca7c3030c18eb419491d
    Closes-bug: #1618487

TatyanaGladysheva (tgladysheva)
tags: added: on-verification
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Issue is reproduced on 9.2 snapshot #627:
/etc/logrotate.d/fuel.nodaily is missing dateext on fuel master node.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/411321

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/8.0)

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: stable/8.0
Review: https://review.openstack.org/408508

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/7.0)

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/409100

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/6.1)

Change abandoned by Oleksiy Molchanov (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/409101

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/411323

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/7.0)

Fix proposed to branch: stable/7.0
Review: https://review.openstack.org/411324

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/6.1)

Fix proposed to branch: stable/6.1
Review: https://review.openstack.org/411326

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/411321
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=5985f8a0328acce9ca107c43c74263eafc0ed0e2
Submitter: Jenkins
Branch: stable/mitaka

commit 5985f8a0328acce9ca107c43c74263eafc0ed0e2
Author: R. Berwald <email address hidden>
Date: Mon Oct 3 13:59:05 2016 +0200

    Adjust logrotate configuration to use date suffix

    Change-Id: I0766797954cc51cdd7dd1016d76f2059870137b7
    Closes-bug: #1618487

Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on 9.2 snapshot #651.

Actual results:
There are dateext and dateformat in both config files - /etc/logrotate.d/fuel.nodaily and in /etc/logrotate.conf on fuel master and slave nodes.

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/fuel-library 11.0.0.0rc1

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.

TatyanaGladysheva (tgladysheva)
tags: added: on-verification
Revision history for this message
TatyanaGladysheva (tgladysheva) wrote :

Verified on 10.0 build #1569.

Actual results:
There are dateext and dateformat in both config files - /etc/logrotate.d/fuel.nodaily and in /etc/logrotate.conf on fuel master and slave nodes.

tags: removed: on-verification
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/6.1)

Change abandoned by Tony Breeds (<email address hidden>) on branch: stable/6.1
Review: https://review.openstack.org/411326
Reason: This branch (stable/6.1) is at End Of Life

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/7.0)

Change abandoned by Tony Breeds (<email address hidden>) on branch: stable/7.0
Review: https://review.openstack.org/411324
Reason: This branch (stable/7.0) is at End Of Life

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/fuel-library (stable/8.0)

Fix proposed to branch: stable/8.0
Change author: Richard Berwald <email address hidden>
Review: https://review.fuel-infra.org/37265

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/8.0)

Change abandoned by Andreas Jaeger (<email address hidden>) on branch: stable/8.0
Review: https://review.opendev.org/411323
Reason: This repo is retired now, no further work will get merged.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.