Comment 3 for bug 1993149

Revision history for this message
Przemyslaw Hausman (phausman) wrote :

I'm attaching output of `rabbitmqctl cluster_status` from all rabbitmq-server units before and after shutting down one unit.

juju-crushdump and juju bundle contain sensitive information, so I'm not sharing it publicly: https://docs.google.com/document/d/1yKPw3vgQWy1rkamQ5YYaoFnWh7PO_TiUEl_r2cnlGaQ/

Steps I used to reproduce:

1. Poweroff rabbitmq-server/0

2. Create VM.

Building the VM fails:

```
$ openstack server show vm-2 -c fault -f value
{'code': 500, 'created': '2022-10-18T14:50:31Z', 'message': 'MessagingTimeout', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get\n return self._queues[msg_id].get(block=True, timeout=timeout)\n File "/usr/lib/python3/dist-packages/eventlet/queue.py", line 322, in get\n return waiter.wait()\n File "/usr/lib/python3/dist-packages/eventlet/queue.py", line 141, in wait\n return get_hub().switch()\n File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 313, in switch\n return self.greenlet.switch()\n_queue.Empty\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1548, in schedule_and_build_instances\n host_lists = self._schedule_instances(context, request_specs[0],\n File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 908, in _schedule_instances\n host_lists = self.query_client.select_destinations(\n File "/usr/lib/python3/dist-packages/nova/scheduler/client/query.py", line 41, in select_destinations\n return self.scheduler_rpcapi.select_destinations(context, spec_obj,\n File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/client.py", line 189, in call\n result = self.transport._send(\n File "/usr/lib/python3/dist-packages/oslo_messaging/transport.py", line 123, in _send\n return self._driver.send(target, ctxt, message,\n File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout,\n File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send\n result = self._waiter.wait(msg_id, timeout,\n File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait\n message = self.waiters.get(msg_id, timeout=timeout)\n File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get\n raise oslo_messaging.MessagingTimeout(\noslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 0bdb814301504f479abcafea2ee3f7b6\n'}
```