SysLogHandler doesn't work properly with the UDP socket

Bug #1135539 reported by Haiwei Xu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Haiwei Xu
oslo-incubator
Fix Released
High
Haiwei Xu
Grizzly
Fix Released
High
Haiwei Xu

Bug Description

When "nova dns-create-private-domain" command failed due to the privileges,
the error message is different between the syslog configuration of "unix socket" and "UDP socket".

When it is configured in "unix socket" mode, it outputs a forbidden error(HTTP 403) which is expected.
However it occurs an internal server error(HTTP 500) in the "UDP socket" mode, and users can't get the
reason of the error.

I configured the /etc/nova/logging-nova-network.conf like below.

...
[handler_syslog]
args=('/dev/log',handlers.SysLogHandler.LOG_LOCAL1) #unix socket
...

$ nova dns-create-private-domain demo (demo user doesn't have the admin authority)
ERROR: User does not have admin privileges (HTTP 403) (Request-ID: req-85e1d369-f22f-45cf-bb1b-f3740d535e8f)

But when I changed the "args" to "UDP socket" mode,
...
[handler_syslog]
args=(('localhost',handlers.SYSLOG_UDP_PORT),handlers.SysLogHandler.LOG_LOCAL1)
...

I got an internal server error(HTTP 500):
$ nova dns-create-private-domain demo
ERROR: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-7b4ce110-dd1d-45a1-bc38-afcd52d05e46)

And the error message in the log file is :

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl
    func(*args, **kwargs)
  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 294, in _process_data
    connection_pool=self.connection_pool)
  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 191, in reply
    ending, log_failure)
  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 161, in msg_reply
    log_failure)
  File "/opt/stack/nova/nova/openstack/common/rpc/common.py", line 273, in serialize_remote_exception
    'args': failure.args,
AttributeError: 'NoneType' object has no attribute 'args'

I also found the same problem in "nova dns-create-public-domain" command.

Haiwei Xu (xu-haiwei)
Changed in nova:
assignee: nobody → Haiwei Xu (xu-haiwei)
Revision history for this message
Russell Bryant (russellb) wrote :

If there is a syslog problem, it's probably in the log module from oslo, so I added oslo to this bug.

Changed in nova:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/23669

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to oslo-incubator (master)

Fix proposed to branch: master
Review: https://review.openstack.org/23894

Changed in oslo:
assignee: nobody → Haiwei Xu (xu-haiwei)
status: New → In Progress
Mark McLoughlin (markmc)
Changed in oslo:
milestone: none → grizzly-rc1
importance: Undecided → High
Thierry Carrez (ttx)
Changed in nova:
status: In Progress → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo-incubator (master)

Reviewed: https://review.openstack.org/23894
Committed: http://github.com/openstack/oslo-incubator/commit/b85e8303347b4ff16951e55e2be2d9b2d39d9678
Submitter: Jenkins
Branch: master

commit b85e8303347b4ff16951e55e2be2d9b2d39d9678
Author: Haiwei Xu <email address hidden>
Date: Thu Mar 7 10:37:14 2013 +0900

    Keep exc_info() to prevent an internal error

    Fixes bug 1135539
    When SysLogHandler is configured in UDP socket mode, the user
    who doesn't have the admin privilege got the unexpecting internal
    server error(HTTP 500) in "nova dns-create-private-domain" command.
    This bug occurs as the call graph drew below:

    |--ProxyCallback._process_data():amqp.py
                              |--LOG.exception()
                              | :
                              | |--BaseHub.switch():hub.py
                              | |-- clear_sys_exc_info()
                              |
                              |--ctxt.reply(None, sys.exc_info(),..)
                                           |
                                           :
                                           |
                             |--serialize_remote_exception():common.py
                                           |--failure_info#(None,None,None)

    The sys.exc_info() is cleared by the clear_sys_exc_info(), but it is
    called again by the ctxt.reply().
    This patch prevents the internal error and returns the details of
    the forbidden error(HTTP 403) which is expected.

    Change-Id: I549b4b2365c53c401edc80bbe30bcde249d27b85

Changed in oslo:
status: In Progress → Fix Committed
Changed in nova:
status: Triaged → In Progress
Thierry Carrez (ttx)
Changed in oslo:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/23669
Committed: http://github.com/openstack/nova/commit/2784fff6300d79d7100e6413f0c3120970134b86
Submitter: Jenkins
Branch: master

commit 2784fff6300d79d7100e6413f0c3120970134b86
Author: Haiwei Xu <email address hidden>
Date: Tue Mar 5 03:52:37 2013 +0000

    Return error details to users in "dns-create-private-domain"

    Fixes bug 1135539
    When SysLogHandler is configured in UDP socket mode, the user
    who doesn't have the admin privilege got the unexpecting internal
    server error(HTTP 500) in "nova dns-create-private-domain" command.
    This bug occurs as the call graph drew below:

    |--ProxyCallback._process_data():amqp.py
                              |--LOG.exception()
                              | :
                              | |--BaseHub.switch():hub.py
                              | |-- clear_sys_exc_info()
                              |
                              |--ctxt.reply(None, sys.exc_info(),..)
                                           |
                                           :
                                           |
                             |--serialize_remote_exception():common.py
                                           |--failure_info#(None,None,None)

    The sys.exc_info() is cleared by the clear_sys_exc_info(), but it is
    called again by the ctxt.reply().
    This patch prevents the internal error and returns the details of
    the forbidden error(HTTP 403) which is expected.

    Change-Id: I76d4a938e66bfaf44423f7586e00db2caf9f5dde

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.