Permission denied: '/var/log/nova/nova-manage.log'

Bug #1671681 reported by Diana Clarke on 2017-03-10
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
puppet-nova
Undecided
Oliver Walsh

Bug Description

The following traceback has been reported multiple times downstream of puppet-nova (here are a few examples).

    https://bugzilla.redhat.com/show_bug.cgi?id=1044155
    https://bugzilla.redhat.com/show_bug.cgi?id=1289761
    https://bugzilla.redhat.com/show_bug.cgi?id=1368588
    https://bugzilla.redhat.com/show_bug.cgi?id=1428572

Traceback (most recent call last):
  File "/bin/nova-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1517, in main
    logging.setup(CONF, "nova")
  File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 243, in setup
    _setup_logging_from_conf(conf, product_name, version)
  File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 319, in _setup_logging_from_conf
    filelog = file_handler(logpath)
  File "/usr/lib64/python2.7/logging/handlers.py", line 392, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'

When you execute nova-manage commands, oslo logs to the following location (the log file name is dynamically created based on the command name).

    /var/log/nova/nova-manage.log

Because puppet-nova is executing these commands as root, nova-manage.log is owned by root:root, preventing the 'nova-manage db archive_deleted_rows' entry in nova's crontab from later executing successfully.

    $ sudo crontab -u nova -l
    PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
    1 0 * * * nova-manage db archive_deleted_rows --max_rows 100 >>/dev/null 2>&1

This log file is also an outlier, as all other log files in /var/log/nova/ are owned by nova:nova. For example:

    $ sudo ls -la /var/log/nova/
    total 102700
    drwxr-x---. 2 nova root 252 Mar 3 03:25 .
    drwxr-xr-x. 40 root root 4096 Mar 2 02:58 ..
    -rw-r--r--. 1 nova nova 49046149 Mar 3 20:21 nova-api.log
    -rw-r--r--. 1 nova nova 3419964 Mar 3 03:25 nova-api.log-20170303.gz
    -rw-r--r--. 1 nova nova 178353 Mar 2 03:07 nova-cert.log
    -rw-r--r--. 1 nova nova 25453688 Mar 3 20:21 nova-compute.log
    -rw-r--r--. 1 nova nova 2528760 Mar 3 03:25 nova-compute.log-20170303.gz
    -rw-r--r--. 1 nova nova 14367109 Mar 3 20:21 nova-conductor.log
    -rw-r--r--. 1 nova nova 1806036 Mar 3 03:25 nova-conductor.log-20170303.gz
    -rw-r--r--. 1 root root 215298 Mar 2 03:07 nova-manage.log
    -rw-r--r--. 1 nova nova 2666163 Mar 3 20:21 nova-scheduler.log

** Heads Up! **

Note that a fix for this bug should consider the upgrades case where that log file is already owned by root:root.

PS. Here's an example of a past workaround:

    https://github.com/redhat-openstack/openstack-utils/pull/8/files

Changed in puppet-nova:
assignee: nobody → Diana Clarke (diana-clarke)
Diana Clarke (diana-clarke) wrote :

Here's a work in progress attempt at this bug:

- Run nova-manage commands as the nova user
- https://review.openstack.org/#/c/442713/

It does not yet address the upgrades case.

Christopher Brown (snecklifter) wrote :

Hi Diana,

Any interest in reviving the abandoned patch here and trying to get this merged?

Diana Clarke (diana-clarke) wrote :

Hi Christopher,

I'm not working on OpenStack anymore. I'll un-assign myself from the bug in case someone else wants to give it a shot.

Have a great week!

Changed in puppet-nova:
assignee: Diana Clarke (diana-clarke) → nobody
Oliver Walsh (owalsh) wrote :

I've proposed a fix in the nova RPM spec https://review.rdoproject.org/r/7474

Changed in puppet-nova:
assignee: nobody → Oliver Walsh (owalsh)
status: New → In Progress
Maciej Kucia (maciejkucia) wrote :

This is happening on Ocata as well.

Fix proposed to branch: master
Review: https://review.openstack.org/498723

Changed in puppet-nova:
assignee: Oliver Walsh (owalsh) → Maciej Kucia (maciejkucia)
Changed in puppet-nova:
assignee: Maciej Kucia (maciejkucia) → Oliver Walsh (owalsh)

Reviewed: https://review.openstack.org/498723
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=e128ba6538eb806589ebc01c16ee5e298bb0ee86
Submitter: Zuul
Branch: master

commit e128ba6538eb806589ebc01c16ee5e298bb0ee86
Author: Diana Clarke <email address hidden>
Date: Tue Mar 7 14:07:16 2017 -0500

    Correct permissions on the nova logfiles

    When you execute nova-manage commands, oslo logs to the following
    location (file name is dynamically created based on command name).

        /var/log/nova/nova-manage.log

    Because puppet-nova is executing these commands as root,
    nova-manage.log is owned by root, preventing the 'nova-manage
    db archive_deleted_rows' entry in nova's crontab from executing.

        Permission denied: '/var/log/nova/nova-manage.log'

    This log file is also an outlier, as all other log files in
    /var/log/nova/ are owned by nova:nova.

    Similar issues are possible for other nova logs, if for example
    a nova services is initially started manually as root, so the
    ownership of all nova logs is corrected before configuring nova.

    Co-Authored-By: Oliver Walsh <email address hidden>
    Co-Authored-By: Diana Clarke <email address hidden>
    Co-Authored-By: Maciej Kucia <email address hidden>
    Closes-Bug: #1671681
    Change-Id: I0ca0110cbf9139c79074cf603dcab9135f96e765

Changed in puppet-nova:
status: In Progress → Fix Released

This issue was fixed in the openstack/puppet-nova 12.2.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers