metadata-agent fills up log-files with " TypeError: 'NoneType' object has no attribute '__getitem__'"

[L00nix]

Sience Monday, the log file of metadata-agent are getting full with the following error / trace:

2015-09-17 11:06:00.187 8277 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute '__getitem__'
2015-09-17 11:06:00.187 8276 ERROR oslo_messaging._drivers.impl_rabbit [-] Failed to consume message from queue: 'NoneType' object has no attribute '__getitem__'
2015-09-17 11:06:00.186 8267 ERROR oslo_messaging._drivers.amqpdriver [-] Failed to process incoming message, retrying...
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver Traceback (most recent call last):
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 197, in poll
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver self.conn.consume(limit=1)
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1172, in consume
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver six.next(it)
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1083, in iterconsume
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver error_callback=_error_callback)
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 870, in ensure
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver ret, channel = autoretry_method()
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 453, in _ensured
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver return fun(*args, **kwargs)
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 520, in __call__^C
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver self.revive(create_channel())
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 251, in channel
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver chan = self.transport.create_channel(self.connection)^C
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 91, in create_channel
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver return connection.channel()
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 279, in channel
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver return self.channels[channel_id]
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver TypeError: 'NoneType' object has no attribute '__getitem__'
2015-09-17 11:06:00.186 8267 TRACE oslo_messaging._drivers.amqpdriver

There was no change in the configuration before, i'm not exactly sure why it does that, the configuration looks fine and all good.

I have also a 100% load on the server.

[Silvan Kaiser (2-silvan)]

We have the same issue on a kilo-release RDO installation, except for the high load.

[Kevin Benton (kevinbenton)]

It looks like a bug in oslo.messaging. The self.channels var is set to None somehow.

[Gary Kotton (garyk)]

This was a result of the change https://review.openstack.org/#q,7267d75fdd3f90af759d71e9490cd41d41ba6d98,n,z
If you are not running a plugin that is in the main tree you need to add the following:
        self.conn.create_consumer(topics.REPORTS,
                                  [agents_db.AgentExtRpcCallback()],
                                  fanout=False)

Does this bug still affect you?

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]