Can't establish qpid connection with impl_qpid.py

[Lance Bragstad]

The notifier module from Oslo has been sync'd with Keystone and is being used to deliver notifications using the log_notifier and rpc_notifier. By default the rpc_backend is impl_kombu.py and I can verify the notifications arrive on the queue. When using impl_qpid as the rpc_backend and sending a notification, the process hangs attempting to establish a connection with qpid.

Steps to reproduce:
1.) Update keystone.conf to send notifications
# ======================= NOTIFICATIONS =================================
notification_driver = keystone.openstack.common.notifier.rpc_notifier
rpc_backend = keystone.openstack.common.rpc.impl_qpid
2.) restart Keystone
3.) create/update/delete project or a user (both resources will send a notification)
$ keystone tenant-create --name my_project

After setting a break point I have followed the call in to qpid code before it hangs. I am not sure if the problem here lies in impl_qpid or possibly something with qpid code. I have been able to reproduce this on qpid 0.20 and 0.22.

The following is a trace with pdb included:

http://paste.openstack.org/show/46886/

I've tried this using config options for qpid username and password and leaving them as default, neither work for establishing a connection.

[Lance Bragstad (lbragstad)]

I have followed the trace farther into qpid code and have this stack trace:

http://pastebin.com/k2ZKHN0C

I put a pdb.set_trace() in /usr/lib/python2.6/site-packages/qpid/compat.py(125)fileno and qpid seems to hang on:

          rd, wr, ex = select(self.reading, self.writing, (), timeout)

[Lance Bragstad (lbragstad)]

I did a local sync of Oslo-incubator's notifier module to Keystone, pulling in the latest notifier changes including the following commit:

https://review.openstack.org/#/c/32187/

Dims suggested I try with the latest notifier module. I set 'qpid_topology_version' to both 1 and 2 and observed the same outcome as before, qpid still hangs calling qpid_messaging.Connection(broker). It seems Glance had similar issues in Grizzly, according to the following bug:

https://bugs.launchpad.net/glance/+bug/1100317

Even though Glance doesn't use the notifier module (or the dependent rpc module) provided by Oslo-incubator I thought it might help shed some light on a possible solution for this as well.

[Lance Bragstad (lbragstad)]

Found the issue. Earlier in the dev cycle a commit was merged to start isolating eventlet code in Keystone (https://github.com/openstack/keystone/blob/master/keystone/common/environment/__init__.py). If a module in Keystone needs to be monkey patched, it is carried out here, when the service starts:

https://github.com/openstack/keystone/blob/master/keystone/common/environment/__init__.py#L81

Before, select was never being monkey patched. As documented in commit #1, qpid would hang on a select call, monkey patching select within Keystone allows select to execute in a monkey patched environment. This can be resolved be explicitly saying what to monkey patch and what to not in keystone/common/environment/__init__.py. This is helpful since it will avoid future issues similar to this and also makes the monkey patching code more readable. The following is a link to the patch:

http://paste.openstack.org/show/47270/

I understand I opened this under the wrong project and I will reassign to Keystone and push up a patch shortly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/47444

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/47444
Committed: http://github.com/openstack/keystone/commit/c0c61ed60b356d581a5618a753f5a5a8939ab844
Submitter: Jenkins
Branch: master

commit c0c61ed60b356d581a5618a753f5a5a8939ab844
Author: Lance Bragstad <email address hidden>
Date: Thu Sep 19 18:22:03 2013 +0000

    Monkey patch select in environment

    Explicitly say what needs to be monkey patched in
    environment/__init__.py. By doing this we are making the code easier to
    read and select needs to be monkey patched in order for notifications to
    work with keystone/openstack/common/rpc/impl_qpid.py. Without monkey
    patching select, keystone will hang on sending a notification since
    qpid_messaging won't operate properly when establishing a connection
    with the qpid broker.

    Change-Id: I4cbdf29924473a76ce459d969e5e7077aac5beb8
    Closes-bug: #1224565