rpc call error from powervc neutron agent

[Terry Yao]

I saw several rpc call error from powervc neutron agent:
2014-11-03 17:18:26.687 3600 INFO powervc.neutron.api.powervc_rpc [req-22c031c8-3bb5-4692-a769-c50dc5529c28 None] Neutron Agent RPC: get_pvc_port_uuid:
2014-11-03 17:18:26.687 3600 INFO powervc.neutron.api.powervc_rpc [req-22c031c8-3bb5-4692-a769-c50dc5529c28 None] - local_port_id: 61245c6b-dbf7-405f-8e22-0991d31cdbf3
2014-11-03 17:18:26.691 3600 INFO powervc.neutron.api.powervc_rpc [req-22c031c8-3bb5-4692-a769-c50dc5529c28 None] - pvc_port_id: None
2014-11-03 17:18:26.730 3600 INFO powervc.neutron.api.powervc_rpc [req-732b37f6-9b8a-454e-9637-fc389c554d4a None] Neutron Agent RPC: get_pvc_port_uuid:
2014-11-03 17:18:26.730 3600 INFO powervc.neutron.api.powervc_rpc [req-732b37f6-9b8a-454e-9637-fc389c554d4a None] - local_port_id: a244d8c8-8ec7-4d27-8fe7-47063c74fa86
2014-11-03 17:18:26.735 3600 INFO powervc.neutron.api.powervc_rpc [req-732b37f6-9b8a-454e-9637-fc389c554d4a None] - pvc_port_id: None
2014-11-03 17:18:26.744 3600 INFO powervc.neutron.api.powervc_rpc [req-146283a8-5d7b-4796-8e14-172b81447f2a None] Neutron Agent RPC: get_pvc_port_uuid:
2014-11-03 17:18:26.745 3600 INFO powervc.neutron.api.powervc_rpc [req-146283a8-5d7b-4796-8e14-172b81447f2a None] - local_port_id: 48a4dec2-74e3-47d4-a227-769b7c1cc816
2014-11-03 17:18:26.749 3600 INFO powervc.neutron.api.powervc_rpc [req-5189c2ac-0a76-4ac5-82f0-6c58ae8e38e5 None] Neutron Agent RPC: get_pvc_port_uuid:
2014-11-03 17:18:26.750 3600 INFO powervc.neutron.api.powervc_rpc [req-5189c2ac-0a76-4ac5-82f0-6c58ae8e38e5 None] - local_port_id: ab3f8a9f-03b0-4473-845a-1f997015d71d
2014-11-03 17:18:26.755 3600 INFO powervc.neutron.api.powervc_rpc [req-5189c2ac-0a76-4ac5-82f0-6c58ae8e38e5 None] - pvc_port_id: None
2014-11-03 17:18:26.757 3600 INFO powervc.neutron.api.powervc_rpc [req-5189c2ac-0a76-4ac5-82f0-6c58ae8e38e5 None] - pvc_port_id: None
2014-11-03 17:18:26.757 3600 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: _oslo_messaging_localcontext_a5bb97ccf6064400ac9107698d48b016
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 179, in _dispatch
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher localcontext.clear_local_context()
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/localcontext.py", line 55, in clear_local_context
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher delattr(_STORE, _KEY)
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher AttributeError: _oslo_messaging_localcontext_a5bb97ccf6064400ac9107698d48b016
2014-11-03 17:18:26.757 3600 TRACE oslo.messaging.rpc.dispatcher
2014-11-03 17:18:26.758 3600 ERROR oslo.messaging._drivers.common [-] Returning exception _oslo_messaging_localcontext_a5bb97ccf6064400ac9107698d48b016 to caller

There is some known community bugs for such error:
https://bugs.launchpad.net/mistral/+bug/1337705
https://bugs.launchpad.net/oslo.messaging/+bug/1288878

people said:
monkey_patch thread is needed when using oslo.message. the behavior is unknown and in risk if thread is set to False.

and I saw they are fixing them using:
eventlet.monkey_patch(socket=True, select=True)

From other neutron agent/service, they are directly using
event.monkey_patch(),

but for current powervc neutron agent:
import eventlet
eventlet.patcher.monkey_patch(os=False, thread=False)

It's explicitly un-patch the thread module... I think that's wrong from oslo.messaging rpc call, we need to monkey_patch the eventlet like neutron..

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/132756
Committed: https://git.openstack.org/cgit/stackforge/powervc-driver/commit/?id=64139a45b79f1a4bf70ee8c5cd1e84c13fec0a15
Submitter: Jenkins
Branch: master

commit 64139a45b79f1a4bf70ee8c5cd1e84c13fec0a15
Author: terryyao <email address hidden>
Date: Wed Nov 5 14:54:25 2014 +0800

    fix eventlet monkey_patch for rpc call

    In PowerVC neutron agent, currently, it's monkey_patch eventlet with
    setting thread=False, that will result in oslo.messaging RPC call failed
    in concurrent execution. Just refer neutron code call monkey_patch() to
    patch all modules

    Closes-Bug: #1389468
    Change-Id: Ia669c37a555bec1d3d2e125c8d088b4d6ef1b40e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/132938

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/132938
Committed: https://git.openstack.org/cgit/stackforge/powervc-driver/commit/?id=443c40efe4d5d0a94f9ca313e8cf3b58efc286e2
Submitter: Jenkins
Branch: stable/juno

commit 443c40efe4d5d0a94f9ca313e8cf3b58efc286e2
Author: terryyao <email address hidden>
Date: Wed Nov 5 14:54:25 2014 +0800

    fix eventlet monkey_patch for rpc call

    In PowerVC neutron agent, currently, it's monkey_patch eventlet with
    setting thread=False, that will result in oslo.messaging RPC call failed
    in concurrent execution. Just refer neutron code call monkey_patch() to
    patch all modules

    Closes-Bug: #1389468
    Change-Id: Ia669c37a555bec1d3d2e125c8d088b4d6ef1b40e
    (cherry picked from commit 64139a45b79f1a4bf70ee8c5cd1e84c13fec0a15)