notifier endless loops in is_primitive

[Chris Behrens]

The recent change for rpc connection pool stuff (https://review.openstack.org/#change,3893) unfortunately broke notifications in the manager when exceptions are raised.

The notifier payload includes the 'context' used for the manager call. Unfortunately this is typically an RpcContext which now includes the rpc connection_pool instance.

Debug output:

working on {'connection_cls': <class 'nova.rpc.impl_kombu.Connection'>, 'current_size': 1, 'order_as_stack': True, 'free_items': deque([<nova.rpc.impl_kombu.Connection object at 0x389aa50>]), 'min_size': 0, 'max_size': 30, 'channel': <LightQueue at 0x3592a90 maxsize=0>} (level=2)

This is the connection pool. unfortunately 'deque' has a list of connections... and kombu's Connection class has an attribute 'consumer_num' which is: itertools.count(). When to_primitive() reaches this.... it spins forever iterating on it.

I think there are multiple problems here.

1) to_primitive tries to recurse only so many levels... and there are a number of recursive calls that do not increase the 'level'.
2) We probably shouldn't store the connection_pool in the rpc context.

[Russell Bryant (russellb)]

Since I introduced the bug, I'll assign this to myself and work on it first thing this week.

[Chris Behrens (cbehrens)]

Thanks! My initial instinct was to not store connection_pool in the context but to just pass it as an arg to msg_reply.... however... that wouldn't work for the case where you can do a context.reply() in a manager. Now... we don't use that behavior currently, though... and I wonder how useful it is. If we can ditch the context.reply() support, the fix is rather easy.

Maybe you can come up with something different.

[Chris Behrens (cbehrens)]

Er, s/msg_reply/ctxt.reply() in ProxyCallback.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/4356

[Monsyne Dragon (mdragon)]

it should also be noted that there is no reason to include the context in the notifications. the fact it does was pretty much accidental.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/4356
Committed: http://github.com/openstack/nova/commit/2a9b66c3dba5f7fb13c7b4e7442eae2bc5dbc130
Submitter: Jenkins
Branch: master

commit 2a9b66c3dba5f7fb13c7b4e7442eae2bc5dbc130
Author: Russell Bryant <email address hidden>
Date: Tue Feb 21 11:33:51 2012 -0500

    Don't store connection pool in RpcContext.

    Fix bug 934575.

    As Chris pointed out in the bug, there is a downside to this approach to
    fixing the bug, in that a manager will no longer be able to use
    context.reply(). However, it wasn't being used at all, and it's no loss
    in functionality. A remote method can still return multiple values (in
    response to a multicall()) by using yield.

    Change-Id: I0e5aff2e8a40ffd8390c0e19d89dd17e60a74130