impl_rabbit can not recover from the "NoneType" error.

[sunzuohua]

When the master-slave switching of AMQP servers occurs, the service log(octavia) is increasing crazy. The log is such as " Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'".
It seems that the channels is setted to None.

2017-04-20 15:53:23.701 57808 ERROR root return infunc(*args, **kwargs)
2017-04-20 15:53:23.701 57808 ERROR root File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/oslo_messaging/_drivers/base.py", line 302, in _runner
2017-04-20 15:53:23.701 57808 ERROR root batch_size=self.batch_size, batch_timeout=self.batch_timeout)
2017-04-20 15:53:23.701 57808 ERROR root File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/oslo_messaging/_drivers/base.py", line 52, in wrapper
2017-04-20 15:53:23.701 57808 ERROR root msg = func(in_self, timeout=timeout_watch.leftover(True))
2017-04-20 15:53:23.701 57808 ERROR root File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in poll
2017-04-20 15:53:23.701 57808 ERROR root self.conn.consume(timeout=min(self._current_timeout, left))
2017-04-20 15:53:23.701 57808 ERROR root File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1106, in consume
2017-04-20 15:53:23.701 57808 ERROR root error_callback=_error_callback)
2017-04-20 15:53:23.701 57808 ERROR root File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 825, in ensure
2017-04-20 15:53:23.701 57808 ERROR root ret, channel = autoretry_method()
2017-04-20 15:53:23.701 57808 ERROR root File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/kombu/connection.py", line 506, in _ensured
2017-04-20 15:53:23.701 57808 ERROR root self.collect()
2017-04-20 15:53:23.701 57808 ERROR root File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/kombu/connection.py", line 350, in collect
2017-04-20 15:53:23.701 57808 ERROR root gc_transport(self._connection)
2017-04-20 15:53:23.701 57808 ERROR root File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 107, in _collect
2017-04-20 15:53:23.701 57808 ERROR root connection.collect()
2017-04-20 15:53:23.701 57808 ERROR root File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/amqp/connection.py", line 423, in collect
2017-04-20 15:53:23.701 57808 ERROR root temp_list = [x for x in values(self.channels) if x is not self]
2017-04-20 15:53:23.701 57808 ERROR root File "/opt/openstack/octavia/octavia-env/lib/python2.7/site-packages/vine/five.py", line 218, in values
2017-04-20 15:53:23.701 57808 ERROR root return d.itervalues()
2017-04-20 15:53:23.701 57808 ERROR root AttributeError: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:23.701 57808 ERROR root
2017-04-20 15:53:24.709 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:25.712 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:26.714 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:27.716 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:28.718 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:29.721 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:30.723 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:31.726 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:32.729 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:33.730 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:34.732 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:35.733 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:36.735 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues'
2017-04-20 15:53:37.736 57808 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute 'itervalues

Add "self._set_current_channel(None)" at [1] and [2] to trigger creating a new connection and a new channel will repair this problem.
[1]https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L826
[2]https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L829

[Daniel Alvarez (dalvarezs)]

Why was this set as invalid? Could you please clarify? I'm seeing the same messages in one setup.

[shen.zhixing (fooy5460)]

This problem may be solved :

https://github.com/celery/py-amqp/commit/b66bdcbff98047819359b64648f6ba3f7e2cdcd0#diff-2a6a0da3750a5417c3a6341ad912f551