Package Versions
================
python-oslo.messaging/bionic,now 5.35.0-0ubuntu1
python-amqp/bionic,now 2.2.2-1
Description
===========
We have a customer who keeps experiencing issues with Heat stack creations/deletions. After investigation, we found that Openstack services (neutron, nova, heat itself, etc) can encounter an error with RMQ message passing, where the service involved gets a MessageDeliveryFailure [0] and the stack fails. This failure can manifest at any point of the creation/deletion and can lead to instances being stuck forever in the BUILD state and other oddities.
From the error output, it looks a lot like a previously fixed issue for py-amqp where a connection closure causes channel issues [1].
Workaround
==========
None found so far other than retrying the Heat stack creation/deletion.
[0] from a nova-cloud-controller unit in nova-conductor.log:
2021-06-24 13:09:14.102 1252 ERROR oslo.messaging._drivers.impl_rabbit [req-5f22f6b9-7b96-4caa-80e2-780084603616 bd514b9cc0dd4395b8c4ea0aa251cf60 d7c94233a5144a4c96c1c78c06384616 - 7744427f95d7445bb90798103ac11564 7744427f95d7445bb90798103ac11564] Failed to publish message to topic 'nova': 'NoneType' object has no attribute '__getitem__'
2021-06-24 13:09:14.103 1252 ERROR oslo.messaging._drivers.impl_rabbit [req-5f22f6b9-7b96-4caa-80e2-780084603616 bd514b9cc0dd4395b8c4ea0aa251cf60 d7c94233a5144a4c96c1c78c06384616 - 7744427f95d7445bb90798103ac11564 7744427f95d7445bb90798103ac11564] Unable to connect to AMQP server on 172.29.101.62:5672 after None tries: 'NoneType' object has no attribute '__getitem__'
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server [req-5f22f6b9-7b96-4caa-80e2-780084603616 bd514b9cc0dd4395b8c4ea0aa251cf60 d7c94233a5144a4c96c1c78c06384616 - 7744427f95d7445bb90798103ac11564 7744427f95d7445bb90798103ac11564] Exception during message handling: MessageDeliveryFailure: Unable to connect to AMQP server on 172.29.101.62:5672 after None tries: 'NoneType' object has no attribute '__getitem__'
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 1302, in schedule_and_build_instances
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server limits=host.limits, host_list=host_list)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 1234, in build_and_run_instance
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server cctxt.cast(ctxt, 'build_and_run_instance', **kwargs)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 152, in cast
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server self.transport._send(self.target, msg_ctxt, msg, retry=self.retry)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 131, in _send
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server timeout=timeout, retry=retry)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server retry=retry)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 545, in _send
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server msg=msg, timeout=timeout, retry=retry)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1272, in topic_send
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server retry=retry)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1155, in _ensure_publishing
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server self.ensure(method, retry=retry, error_callback=_error_callback)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 825, in ensure
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server raise exceptions.MessageDeliveryFailure(msg)
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server MessageDeliveryFailure: Unable to connect to AMQP server on 172.29.101.62:5672 after None tries: 'NoneType' object has no attribute '__getitem__'
2021-06-24 13:09:14.103 1252 ERROR oslo_messaging.rpc.server
[1] https://github.com/celery/py-amqp/pull/289
