Logs on controllers are not rotated: /var/log/murano/ directory has insecure permissions

Bug #1376209 reported by Artem Panchenko
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Critical
Dmitry Teselkin
5.1.x
Fix Released
Critical
Dmitry Teselkin
6.0.x
Fix Released
Critical
Dmitry Teselkin

Bug Description

Fuel version - http://paste.openstack.org/show/117417/

After 2 hours under high load (Rally testing on bare metal) MySQL crashed on primary controller and Nova API began to return errors to all requests (HAProxy continued to forward requests to it), because there was no free disk space on root partition. This issue was caused by broken logs rotation:

http://paste.openstack.org/show/117423/

As you can see logs rotation worked fine after I fixed permissions for directory.

Recently there was another but similar issue with logs rotation: https://bugs.launchpad.net/fuel/+bug/1367234 which also caused failing of MySQL and breakage of OpenStack services. I think such issue could occur again, because new services are continuously added on nodes and they may create new log files. In my opinion it's much safer to create a separate partition for logs, especially if there is enough disk space.

Tags: murano library
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :
Changed in fuel:
importance: Undecided → Critical
assignee: Fuel Library Team (fuel-library) → Bogdan Dobrelya (bogdando)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/125316

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.0)

Fix proposed to branch: stable/5.0
Review: https://review.openstack.org/125318

Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

I verified the packages for murano and centos one should be fixed to provide 0766 -> -0755 for murano-ali rpm
 Ubuntu: murano-api, murano-dashboard
  /var/log/murano (drwxr-xr-x)
 Centos: murano-api
  /var/log/murano (drwxrw-rw-)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (master)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: master
Review: https://review.openstack.org/125316

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-library (stable/5.0)

Change abandoned by Bogdan Dobrelya (<email address hidden>) on branch: stable/5.0
Review: https://review.openstack.org/125318

tags: added: murano
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/129628

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/129630

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/129628
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=14cb8c26cc48106e7053153cd82798430c17de91
Submitter: Jenkins
Branch: master

commit 14cb8c26cc48106e7053153cd82798430c17de91
Author: Dmitry Teselkin <email address hidden>
Date: Mon Oct 20 18:12:14 2014 +0400

    Fix murano logdir permissions

    Change-Id: I67c4356f52f93ad345dc18d1868c48199798fc6d
    Relates-bug: 1376209

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/129630
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=c95fd96dd80fd5816b409250dcdfcd0bf41995a0
Submitter: Jenkins
Branch: stable/5.1

commit c95fd96dd80fd5816b409250dcdfcd0bf41995a0
Author: Dmitry Teselkin <email address hidden>
Date: Mon Oct 20 18:18:43 2014 +0400

    Fix murano logdir permissions

    Relates-bug: 1376209

    Change-Id: I67c4356f52f93ad345dc18d1868c48199798fc6d

Revision history for this message
Artem Panchenko (apanchenko-8) wrote :

verified on Fuel 5.1.1:

api: '1.0'
astute_sha: ef8aa0fd0e3ce20709612906f1f0551b5682a6ce
auth_required: true
build_id: 2014-12-03_01-07-36
build_number: '48'
feature_groups:
- mirantis
- experimental
fuellib_sha: a3043477337b4a0a8fd166dc83d6cd5d504f5da8
fuelmain_sha: 7626c5aeedcde77ad22fc081c25768944697d404
nailgun_sha: 500e36d08a45dbb389bf2bd97673d9bff48ee84d
ostf_sha: 64cb59c681658a7a55cc2c09d079072a41beb346
production: docker
release: 5.1.1

Revision history for this message
Mykhaylo Slobodyan (mykhaylo-slobodyan) wrote :

Verified correct permissions on 6.0 ISO 49:

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.0"
  api: "1.0"
  build_number: "49"
  build_id: "2014-12-09_22-41-06"
  astute_sha: "16b252d93be6aaa73030b8100cf8c5ca6a970a91"
  fuellib_sha: "2c99931072d951301d395ebd5bf45c8d401301bb"
  ostf_sha: "a9afb68710d809570460c29d6c3293219d3624d4"
  nailgun_sha: "22bd43b89a17843f9199f92d61fc86cb0f8772f1"
  fuelmain_sha: "3aab16667f47dd8384904e27f70f7a87ba15f4

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.