logrotate doesn't work exited abnormally with [1] (0 was expected)

Bug #1455104 reported by Leontiy Istomin on 2015-05-14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Bartłomiej Piotrowski
Swann Croiset
Swann Croiset
Swann Croiset
Swann Croiset

Bug Description

api: '1.0'
astute_sha: 055b2d82fe8499b27c7047295e2e36a7a2c5d430
auth_required: true
build_id: 2015-04-16_21-30-10
build_number: '317'
- mirantis
fuellib_sha: db5f39e96e7ab9f79691202755e547bf8242661f
fuelmain_sha: 0de2d2039e76839d339f977df45b111bef7200d6
nailgun_sha: 52d92c86e68602fb5dd2f3b8870a773d20a387ef
openstack_version: 2014.2-6.1
ostf_sha: b0991dbad159f53d335efa5d31cb94016ad5312e
production: docker
python-fuelclient_sha: 279ffc358e40dbdc162cfe76898cbd0874529f1f
release: '6.1'

Successfully deployed the following configuration:
Baremetal,Ubuntu,IBP,HA, Neutron-vlan,Ceph-all,Nova-debug,Nova-quotas, 6.1-317
Controllers:3 Computes:47

We have found that there is no archived logs on controller nodes:
root@node-1:~# ls -lah /var/log/*-all.log*
-rw-r----- 1 syslog syslog 279M May 14 13:59 /var/log/cinder-all.log
-rw-r----- 1 syslog syslog 356M May 14 13:59 /var/log/glance-all.log
-rw-r----- 1 syslog syslog 157M May 14 13:59 /var/log/heat-all.log
-rw-r----- 1 syslog syslog 1.8G May 14 13:59 /var/log/keystone-all.log
-rw-r----- 1 syslog syslog 3.3G May 14 13:59 /var/log/neutron-all.log
-rw-r----- 1 syslog syslog 718M May 14 13:59 /var/log/nova-all.log

For example cinder-all log hasn't been rotated:
root@node-1:~# head -1 /var/log/cinder-all.log
<159>May 12 22:41:11 node-1 cinder-manage backend <module 'cinder.db.sqlalchemy.migration' from '/usr/lib/python2.7/dist-packages/cinder/db/sqlalchemy/migration.pyc'>
root@node-1:~# tail -1 /var/log/cinder-all.log
<158>May 14 14:00:28 node-1 cinder-api - - [14/May/2015 14:00:28] "OPTIONS / HTTP/1.0" 200 422 0.003460

root@node-1:~# ls -la /var/log/cinder/
total 416764
drwxr-x--- 2 cinder adm 4096 May 12 22:41 .
drwxrwxr-x 25 root syslog 4096 May 14 12:30 ..
-rw-r--r-- 1 cinder cinder 300518822 May 14 14:00 cinder-api.log
-rw-r--r-- 1 cinder cinder 709235 May 14 14:00 cinder-backup.log
-rw-r--r-- 1 cinder cinder 23014 May 12 22:41 cinder-manage.log
-rw-r--r-- 1 cinder cinder 29438943 May 14 14:00 cinder-scheduler.log
-rw-r--r-- 1 cinder cinder 96045179 May 14 14:00 cinder-volume.log

from messages log file:
<13>May 14 13:50:01 node-1 logrotate: ALERT exited abnormally with [1] (0 was expected)
We can see this alert every 10 minutes

Diagnostic snapshot is here http://mos-scale-share.mirantis.com/fuel-snapshot-2015-05-14_08-14-00.tar.xz

no longer affects: mos
Changed in fuel:
assignee: nobody → Bartlomiej Piotrowski (bpiotrowski)
milestone: none → 6.1
importance: Undecided → High

It's a really old iso. There were a few changes to logrotate configuration in the meantime:


If you think that the issue is still present, please post here the logs from logrotate runs, because I don't think those are included in diagnostic snapshot.

1. logrotate -v /etc/logrotate.conf
2. Edit our logrotate shell wrapper to include 'set -x' before anything gets executed and attach the output of run.

Changed in fuel:
status: New → Incomplete
Changed in fuel:
status: Incomplete → Invalid
Serg Lystopad (slystopad) wrote :

We still see the bug in Fuel 8.0
cat /etc/fuel/version.yaml
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "570"
  build_id: "570"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "c2a335b5b725f1b994f78d4c78723d29fa44685a"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

Changed in fuel:
status: Invalid → Opinion
tags: added: customer-found
Stanislav Kolenkin (skolenkin) wrote :

The crontab contain the following (crontab -l) on compute nodes:
# HEADER: This file was autogenerated at 2016-07-26 22:11:14 +0000 by puppet.
# HEADER: While it can still be managed manually, it is definitely not recommended.
# HEADER: Note particularly that the comments starting with 'Puppet Name' should
# HEADER: not be deleted, as doing so could cause duplicate cron jobs.
# Puppet Name: fuel-logrotate
*/30 * * * * /usr/bin/fuel-logrotate
# Puppet Name: lma_collector logrotate
*/30 * * * * /usr/local/bin/logrotate_lma_collector

Following workaround may help here.
Please, update crontab file "/var/spool/cron/crontabs/root” as follows.
Substitute these lines:
*/30 * * * * /usr/bin/fuel-logrotate
# Puppet Name: lma_collector logrotate
*/30 * * * * /usr/local/bin/logrotate_lma_collector
25 * * * * /usr/bin/fuel-logrotate
# Puppet Name: lma_collector logrotate
40 * * * * /usr/local/bin/logrotate_lma_collector

Swann Croiset (swann-w) on 2016-08-16
Changed in lma-toolchain:
importance: Undecided → High
assignee: nobody → LMA-Toolchain Fuel Plugins (mos-lma-toolchain)
status: New → Confirmed
Swann Croiset (swann-w) wrote :

This is the root cause when concurrent execution of logrotate occur:

error: error renaming temp state file /var/lib/logrotate/status.tmp

Fix proposed to branch: master
Review: https://review.openstack.org/355949

Changed in lma-toolchain:
assignee: LMA-Toolchain Fuel Plugins (mos-lma-toolchain) → Swann Croiset (swann-w)
status: Confirmed → In Progress
Dmitry Guryanov (dguryanov) wrote :

Are there any fixes needed on the fuel side?

Reviewed: https://review.openstack.org/355949
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=99508dbc3919863330d9a879529b330fdd1d5dff
Submitter: Jenkins
Branch: master

commit 99508dbc3919863330d9a879529b330fdd1d5dff
Author: Swann Croiset <email address hidden>
Date: Tue Aug 16 15:50:55 2016 +0200

    Fixes the concurrent execution of logrotate

    Change-Id: I1546744d3c4cab8fc359b6571d99aabde3c3ff2d
    Closes-bug: #1455104

Changed in lma-toolchain:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/356047
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=43a6ad6a79c0644724693541a6cc2d74c2e3dadd
Submitter: Jenkins
Branch: stable/0.10

commit 43a6ad6a79c0644724693541a6cc2d74c2e3dadd
Author: Swann Croiset <email address hidden>
Date: Tue Aug 16 15:50:55 2016 +0200

    Fixes the concurrent execution of logrotate

    Change-Id: I1546744d3c4cab8fc359b6571d99aabde3c3ff2d
    Closes-bug: #1455104
    (cherry picked from commit 99508dbc3919863330d9a879529b330fdd1d5dff)

Reviewed: https://review.openstack.org/356048
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=7ee4a5b53b8a0f167076fba8351035a492f2e66c
Submitter: Jenkins
Branch: stable/0.9

commit 7ee4a5b53b8a0f167076fba8351035a492f2e66c
Author: Swann Croiset <email address hidden>
Date: Tue Aug 16 15:50:55 2016 +0200

    Fixes the concurrent execution of logrotate

    Change-Id: I1546744d3c4cab8fc359b6571d99aabde3c3ff2d
    Closes-bug: #1455104
    (cherry picked from commit 99508dbc3919863330d9a879529b330fdd1d5dff)

Change abandoned by Swann Croiset (<email address hidden>) on branch: stable/0.8
Review: https://review.openstack.org/356050
Reason: no body wants it :/

no longer affects: lma-toolchain/1.0
Changed in lma-toolchain:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers