AttributeError: _oslo_messaging_localcontext errors found in neutron l3-agent logs

[Swaminathan Vasudevan]

This TRACE is seen in many places in the neutron l3-agent logs from the jenkins logs.

2015-02-02 23:29:13.916 ERROR oslo_messaging.rpc.dispatcher [req-ce57a6b0-04fc-41dd-a114-5b69c0ebcf6d FloatingIPsNegativeTestJSON-267704990 FloatingIPsNegativeTestJSON-594738290] Exception during message handling: _oslo_messaging_localcontext_fdc8b1dcad1246c49200370f4281e5d5
2015-02-02 23:29:13.916 3051 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-02-02 23:29:13.916 3051 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-02-02 23:29:13.916 3051 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-02-02 23:29:13.916 3051 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 188, in _dispatch
2015-02-02 23:29:13.916 3051 TRACE oslo_messaging.rpc.dispatcher localcontext.clear_local_context()
2015-02-02 23:29:13.916 3051 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/localcontext.py", line 55, in clear_local_context
2015-02-02 23:29:13.916 3051 TRACE oslo_messaging.rpc.dispatcher delattr(_STORE, _KEY)
2015-02-02 23:29:13.916 3051 TRACE oslo_messaging.rpc.dispatcher AttributeError: _oslo_messaging_localcontext_fdc8b1dcad1246c49200370f4281e5d5
2015-02-02 23:29:13.916 3051 TRACE oslo_messaging.rpc.dispatcher

I am not sure if there is any other similar bugs that have been reported.

[venkata anil (anil-venkata)]

@swaminathan - please free to assign to your self if you are already working on it.

[venkata anil (anil-venkata)]

Also see trace in neutron server. I think it is a similar issue

2015-02-03 10:12:55.119 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Endpoint does not support RPC method security_group_info_for_devices
2015-02-03 10:12:55.119 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-02-03 10:12:55.119 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 137, in _dispatch_and_reply
2015-02-03 10:12:55.119 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-02-03 10:12:55.119 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 197, in _dispatch
2015-02-03 10:12:55.119 TRACE oslo.messaging.rpc.dispatcher raise NoSuchMethod(method)
2015-02-03 10:12:55.119 TRACE oslo.messaging.rpc.dispatcher NoSuchMethod: Endpoint does not support RPC method security_group_info_for_devices
2015-02-03 10:12:55.119 TRACE oslo.messaging.rpc.dispatcher
2015-02-03 10:12:55.121 ERROR oslo.messaging._drivers.common [-] Returning exception Endpoint does not support RPC method security_group_info_for_devices to caller

[venkata anil (anil-venkata)]

Now not seeing the issue. Can you please share the complete logs (or logstash/jenkins link)

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Here is the logstash link.

http://logs.openstack.org/84/150284/7/check/check-tempest-dsvm-neutron-dvr/3271052/logs/screen-q-vpn.txt.gz?level=TRACE

In this trace you can see these failures.

2015-02-04 04:09:28.990 ERROR oslo_messaging.rpc.dispatcher [req-9ab07a7a-250a-4342-a3f4-a9764a865be2 admin admin] Exception during message handling: _oslo_messaging_localcontext_dd897471657b4ad988739822c560034a
2015-02-04 04:09:28.990 2812 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-02-04 04:09:28.990 2812 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-02-04 04:09:28.990 2812 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-02-04 04:09:28.990 2812 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 188, in _dispatch
2015-02-04 04:09:28.990 2812 TRACE oslo_messaging.rpc.dispatcher localcontext.clear_local_context()
2015-02-04 04:09:28.990 2812 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/localcontext.py", line 55, in clear_local_context
2015-02-04 04:09:28.990 2812 TRACE oslo_messaging.rpc.dispatcher delattr(_STORE, _KEY)
2015-02-04 04:09:28.990 2812 TRACE oslo_messaging.rpc.dispatcher AttributeError: _oslo_messaging_localcontext_dd897471657b4ad988739822c560034a
2015-02-04 04:09:28.990 2812 TRACE oslo_messaging.rpc.dispatcher

[Hua Zhang (zhhuabj)]

hit the same issue today,

2015-02-07 21:09:40.620 INFO oslo_messaging._drivers.impl_qpid [req-65511803-e393-4a9a-9102-a4d25d274d4c None None] Connected to AMQP server on 172.16.1.1:5672
2015-02-07 21:09:40.630 INFO neutron.agent.l3.agent [req-65511803-e393-4a9a-9102-a4d25d274d4c None None] L3 agent started
2015-02-07 21:11:20.307 ERROR oslo_messaging.rpc.dispatcher [req-d4d814b3-c121-4abb-83c5-c96cac91a91a TestNetworkVPNaaS-85735279 TestNetworkVPNaaS-656156572] Exception during message handling: _oslo_messaging_localcontext_8737cc42c3ff4ffeaa0c5bc63c0031f8
2015-02-07 21:11:20.307 22108 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-02-07 21:11:20.307 22108 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-02-07 21:11:20.307 22108 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-02-07 21:11:20.307 22108 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 188, in _dispatch
2015-02-07 21:11:20.307 22108 TRACE oslo_messaging.rpc.dispatcher localcontext.clear_local_context()
2015-02-07 21:11:20.307 22108 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/localcontext.py", line 55, in clear_local_context
2015-02-07 21:11:20.307 22108 TRACE oslo_messaging.rpc.dispatcher delattr(_STORE, _KEY)
2015-02-07 21:11:20.307 22108 TRACE oslo_messaging.rpc.dispatcher AttributeError: _oslo_messaging_localcontext_8737cc42c3ff4ffeaa0c5bc63c0031f8
2015-02-07 21:11:20.307 22108 TRACE oslo_messaging.rpc.dispatcher

[Armando Migliaccio (armando-migliaccio)]

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiaW4gY2xlYXJfbG9jYWxfY29udGV4dFwiIEFORCBtZXNzYWdlOlwiQXR0cmlidXRlRXJyb3JcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiODY0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDIzNTA5OTM1MDE5fQ==

[Armando Migliaccio (armando-migliaccio)]

This looks related: https://bugs.launchpad.net/mistral/+bug/1337705

[Armando Migliaccio (armando-migliaccio)]

It looks like Terry's change:

https://review.openstack.org/#/c/153699/

Does not manifest the traces:

http://logs.openstack.org/99/153699/4/check/check-tempest-dsvm-neutron-full/9d322ba/logs/screen-q-vpn.txt.gz

So I'd be tempted to raise's Terry's change awareness

[Ihar Hrachyshka (ihar-hrachyshka)]

This is aregression of https://github.com/openstack/neutron/commit/6ca8cb84fd8f703367e1bd8ee1a2f26071116725

[Armando Migliaccio (armando-migliaccio)]

Terry's change is about to be reverted. We need another plan of attack.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/156942

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156942
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=37115731515895ea224b1dafd71250d0b002ad40
Submitter: Jenkins
Branch: master

commit 37115731515895ea224b1dafd71250d0b002ad40
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Feb 18 11:42:43 2015 +0100

    Monkey patch all the code inside neutron/cmd/eventlet/...

    The directory is initially empty.

    We are going to maintain entry points for all services and agents that
    run in eventlet mode in this directory, and monkey patch them from
    there, instead of spreading monkey_patch() calls throughout the library.

    This will guarantee us that all the services that are maintained in this
    part of the tree monkey patch stdlib properly, before doing any other
    imports.

    This is also useful to track which parts of the project require
    eventlet. This will later help to migrate services one by one out of
    eventlet to real threads in case we decide to move this direction.

    Related-Bug: #1417386
    Related-Bug: #1418541
    Change-Id: I2bc16ca4422c01d64e9fac4910214dbb0d0326ff