Can't establish qpid connection with impl_qpid.py

Bug #1224565 reported by Lance Bragstad on 2013-09-12
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
High
Lance Bragstad

Bug Description

The notifier module from Oslo has been sync'd with Keystone and is being used to deliver notifications using the log_notifier and rpc_notifier. By default the rpc_backend is impl_kombu.py and I can verify the notifications arrive on the queue. When using impl_qpid as the rpc_backend and sending a notification, the process hangs attempting to establish a connection with qpid.

Steps to reproduce:
1.) Update keystone.conf to send notifications
# ======================= NOTIFICATIONS =================================
notification_driver = keystone.openstack.common.notifier.rpc_notifier
rpc_backend = keystone.openstack.common.rpc.impl_qpid
2.) restart Keystone
3.) create/update/delete project or a user (both resources will send a notification)
$ keystone tenant-create --name my_project

After setting a break point I have followed the call in to qpid code before it hangs. I am not sure if the problem here lies in impl_qpid or possibly something with qpid code. I have been able to reproduce this on qpid 0.20 and 0.22.

The following is a trace with pdb included:

http://paste.openstack.org/show/46886/

I've tried this using config options for qpid username and password and leaving them as default, neither work for establishing a connection.

description: updated
Lance Bragstad (lbragstad) wrote :

I have followed the trace farther into qpid code and have this stack trace:

http://pastebin.com/k2ZKHN0C

I put a pdb.set_trace() in /usr/lib/python2.6/site-packages/qpid/compat.py(125)fileno and qpid seems to hang on:

          rd, wr, ex = select(self.reading, self.writing, (), timeout)

Lance Bragstad (lbragstad) wrote :

I did a local sync of Oslo-incubator's notifier module to Keystone, pulling in the latest notifier changes including the following commit:

https://review.openstack.org/#/c/32187/

Dims suggested I try with the latest notifier module. I set 'qpid_topology_version' to both 1 and 2 and observed the same outcome as before, qpid still hangs calling qpid_messaging.Connection(broker). It seems Glance had similar issues in Grizzly, according to the following bug:

https://bugs.launchpad.net/glance/+bug/1100317

Even though Glance doesn't use the notifier module (or the dependent rpc module) provided by Oslo-incubator I thought it might help shed some light on a possible solution for this as well.

Lance Bragstad (lbragstad) wrote :

Found the issue. Earlier in the dev cycle a commit was merged to start isolating eventlet code in Keystone (https://github.com/openstack/keystone/blob/master/keystone/common/environment/__init__.py). If a module in Keystone needs to be monkey patched, it is carried out here, when the service starts:

https://github.com/openstack/keystone/blob/master/keystone/common/environment/__init__.py#L81

Before, select was never being monkey patched. As documented in commit #1, qpid would hang on a select call, monkey patching select within Keystone allows select to execute in a monkey patched environment. This can be resolved be explicitly saying what to monkey patch and what to not in keystone/common/environment/__init__.py. This is helpful since it will avoid future issues similar to this and also makes the monkey patching code more readable. The following is a link to the patch:

http://paste.openstack.org/show/47270/

I understand I opened this under the wrong project and I will reassign to Keystone and push up a patch shortly.

no longer affects: oslo

Fix proposed to branch: master
Review: https://review.openstack.org/47444

Changed in keystone:
assignee: nobody → Lance Bragstad (ldbragst)
status: New → In Progress
Dolph Mathews (dolph) on 2013-09-20
Changed in keystone:
importance: Undecided → High
milestone: none → havana-rc1

Reviewed: https://review.openstack.org/47444
Committed: http://github.com/openstack/keystone/commit/c0c61ed60b356d581a5618a753f5a5a8939ab844
Submitter: Jenkins
Branch: master

commit c0c61ed60b356d581a5618a753f5a5a8939ab844
Author: Lance Bragstad <email address hidden>
Date: Thu Sep 19 18:22:03 2013 +0000

    Monkey patch select in environment

    Explicitly say what needs to be monkey patched in
    environment/__init__.py. By doing this we are making the code easier to
    read and select needs to be monkey patched in order for notifications to
    work with keystone/openstack/common/rpc/impl_qpid.py. Without monkey
    patching select, keystone will hang on sending a notification since
    qpid_messaging won't operate properly when establishing a connection
    with the qpid broker.

    Change-Id: I4cbdf29924473a76ce459d969e5e7077aac5beb8
    Closes-bug: #1224565

Changed in keystone:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2013-10-02
Changed in keystone:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-10-17
Changed in keystone:
milestone: havana-rc1 → 2013.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers