puppet-nova

Permission denied: '/var/log/nova/nova-manage.log'

Bug #1671681 reported by Diana Clarke on 2017-03-10

This bug affects 3 people

Affects		Status	Importance	Assigned to	Milestone
	puppet-nova	Fix Released	Undecided	Oliver Walsh

Bug Description

The following traceback has been reported multiple times downstream of puppet-nova (here are a few examples).

    https://bugzilla.redhat.com/show_bug.cgi?id=1044155
    https://bugzilla.redhat.com/show_bug.cgi?id=1289761
    https://bugzilla.redhat.com/show_bug.cgi?id=1368588
    https://bugzilla.redhat.com/show_bug.cgi?id=1428572

Traceback (most recent call last):
  File "/bin/nova-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1517, in main
    logging.setup(CONF, "nova")
  File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 243, in setup
    _setup_logging_from_conf(conf, product_name, version)
  File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 319, in _setup_logging_from_conf
    filelog = file_handler(logpath)
  File "/usr/lib64/python2.7/logging/handlers.py", line 392, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'

When you execute nova-manage commands, oslo logs to the following location (the log file name is dynamically created based on the command name).

/var/log/nova/nova-manage.log

Because puppet-nova is executing these commands as root, nova-manage.log is owned by root:root, preventing the 'nova-manage db archive_deleted_rows' entry in nova's crontab from later executing successfully.

    $ sudo crontab -u nova -l
    PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
    1 0 * * * nova-manage db archive_deleted_rows --max_rows 100 >>/dev/null 2>&1

This log file is also an outlier, as all other log files in /var/log/nova/ are owned by nova:nova. For example:

    $ sudo ls -la /var/log/nova/
    total 102700
    drwxr-x---. 2 nova root 252 Mar 3 03:25 .
    drwxr-xr-x. 40 root root 4096 Mar 2 02:58 ..
    -rw-r--r--. 1 nova nova 49046149 Mar 3 20:21 nova-api.log
    -rw-r--r--. 1 nova nova 3419964 Mar 3 03:25 nova-api.log-20170303.gz
    -rw-r--r--. 1 nova nova 178353 Mar 2 03:07 nova-cert.log
    -rw-r--r--. 1 nova nova 25453688 Mar 3 20:21 nova-compute.log
    -rw-r--r--. 1 nova nova 2528760 Mar 3 03:25 nova-compute.log-20170303.gz
    -rw-r--r--. 1 nova nova 14367109 Mar 3 20:21 nova-conductor.log
    -rw-r--r--. 1 nova nova 1806036 Mar 3 03:25 nova-conductor.log-20170303.gz
    -rw-r--r--. 1 root root 215298 Mar 2 03:07 nova-manage.log
    -rw-r--r--. 1 nova nova 2666163 Mar 3 20:21 nova-scheduler.log

** Heads Up! **

Note that a fix for this bug should consider the upgrades case where that log file is already owned by root:root.

PS. Here's an example of a past workaround:

https://github.com/redhat-openstack/openstack-utils/pull/8/files

Tags:

Diana Clarke (diana-clarke) on 2017-03-10

Changed in puppet-nova:
assignee:	nobody → Diana Clarke (diana-clarke)

Revision history for this message

Diana Clarke (diana-clarke) wrote on 2017-03-10:

Here's a work in progress attempt at this bug:

- Run nova-manage commands as the nova user
- https://review.openstack.org/#/c/442713/

It does not yet address the upgrades case.

Revision history for this message

Christopher Brown (snecklifter) wrote on 2017-06-29:

Hi Diana,

Any interest in reviving the abandoned patch here and trying to get this merged?

Revision history for this message

Diana Clarke (diana-clarke) wrote on 2017-06-29:

Hi Christopher,

I'm not working on OpenStack anymore. I'll un-assign myself from the bug in case someone else wants to give it a shot.

Have a great week!

Changed in puppet-nova:
assignee:	Diana Clarke (diana-clarke) → nobody

Revision history for this message

Oliver Walsh (owalsh) wrote on 2017-07-07:

I've proposed a fix in the nova RPM spec https://review.rdoproject.org/r/7474

Changed in puppet-nova:
assignee:	nobody → Oliver Walsh (owalsh)
status:	New → In Progress

Revision history for this message

Maciej Kucia (maciejkucia) wrote on 2017-08-29:

This is happening on Ocata as well.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-29: Fix proposed to puppet-nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/498723

Changed in puppet-nova:
assignee:	Oliver Walsh (owalsh) → Maciej Kucia (maciejkucia)

OpenStack Infra (hudson-openstack) on 2017-12-12

Changed in puppet-nova:
assignee:	Maciej Kucia (maciejkucia) → Oliver Walsh (owalsh)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-12-20: Fix merged to puppet-nova (master)

Reviewed: https://review.openstack.org/498723
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=e128ba6538eb806589ebc01c16ee5e298bb0ee86
Submitter: Zuul
Branch: master

commit e128ba6538eb806589ebc01c16ee5e298bb0ee86
Author: Diana Clarke <email address hidden>
Date: Tue Mar 7 14:07:16 2017 -0500

Correct permissions on the nova logfiles

When you execute nova-manage commands, oslo logs to the following
location (file name is dynamically created based on command name).

/var/log/nova/nova-manage.log

    Because puppet-nova is executing these commands as root,
    nova-manage.log is owned by root, preventing the 'nova-manage
    db archive_deleted_rows' entry in nova's crontab from executing.

Permission denied: '/var/log/nova/nova-manage.log'

This log file is also an outlier, as all other log files in
/var/log/nova/ are owned by nova:nova.

    Similar issues are possible for other nova logs, if for example
    a nova services is initially started manually as root, so the
    ownership of all nova logs is corrected before configuring nova.

Changed in puppet-nova:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-25: Fix included in openstack/puppet-nova 12.2.0

This issue was fixed in the openstack/puppet-nova 12.2.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-04-16: Fix proposed to puppet-nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/561695

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-04-16: Fix proposed to puppet-nova (stable/ocata)

#10

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/561742

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-04-17: Fix merged to puppet-nova (stable/pike)

#11

Reviewed: https://review.openstack.org/561695
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=0d6f9e10749959b0fe88c876319ca03ddac1b34e
Submitter: Zuul
Branch: stable/pike

commit 0d6f9e10749959b0fe88c876319ca03ddac1b34e
Author: Diana Clarke <email address hidden>
Date: Tue Mar 7 14:07:16 2017 -0500

Correct permissions on the nova logfiles

When you execute nova-manage commands, oslo logs to the following
location (file name is dynamically created based on command name).

/var/log/nova/nova-manage.log

    Because puppet-nova is executing these commands as root,
    nova-manage.log is owned by root, preventing the 'nova-manage
    db archive_deleted_rows' entry in nova's crontab from executing.

Permission denied: '/var/log/nova/nova-manage.log'

This log file is also an outlier, as all other log files in
/var/log/nova/ are owned by nova:nova.

    Similar issues are possible for other nova logs, if for example
    a nova services is initially started manually as root, so the
    ownership of all nova logs is corrected before configuring nova.

    Co-Authored-By: Oliver Walsh <email address hidden>
    Co-Authored-By: Diana Clarke <email address hidden>
    Co-Authored-By: Maciej Kucia <email address hidden>
    Closes-Bug: #1671681
    Change-Id: I0ca0110cbf9139c79074cf603dcab9135f96e765
    (cherry picked from commit e128ba6538eb806589ebc01c16ee5e298bb0ee86)

tags:

added: in-stable-pike

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-04-18: Fix merged to puppet-nova (stable/ocata)

#12

Reviewed: https://review.openstack.org/561742
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=482bf9695fb49c9bc59785510978cd2f973b3bba
Submitter: Zuul
Branch: stable/ocata

commit 482bf9695fb49c9bc59785510978cd2f973b3bba
Author: Diana Clarke <email address hidden>
Date: Tue Mar 7 14:07:16 2017 -0500

Correct permissions on the nova logfiles

When you execute nova-manage commands, oslo logs to the following
location (file name is dynamically created based on command name).

/var/log/nova/nova-manage.log

    Because puppet-nova is executing these commands as root,
    nova-manage.log is owned by root, preventing the 'nova-manage
    db archive_deleted_rows' entry in nova's crontab from executing.

Permission denied: '/var/log/nova/nova-manage.log'

This log file is also an outlier, as all other log files in
/var/log/nova/ are owned by nova:nova.

    Similar issues are possible for other nova logs, if for example
    a nova services is initially started manually as root, so the
    ownership of all nova logs is corrected before configuring nova.

tags:

added: in-stable-ocata

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-16: Fix included in openstack/puppet-nova 11.6.0

#13

This issue was fixed in the openstack/puppet-nova 11.6.0 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.