Steps to reproduce:
1. Deploy MOS environment in HA mode with several controllers
2. Shut down one of the controllers, either gracefully or not
3. Wait for MySQL, RabbitMQ and OpenStack to failover (several minutes)
4. Try to use OpenStack API which invokes internal messaging via Rabbit MQ. For instance, you can view console log of an instance, create instances, networks, volumes, etc.
Some requests sent in step #4 might fail with timeout. In the logs of the affected service the following message could be seen:
"Queue not found:Basic.consume: (404) NOT_FOUND - no queue 'reply_f7cac1a2428d414bb8b9e0a612"
Conditions for reproduction:
The issue occurs rather infrequently, though we don't have exact date. So far we have only 4 reproductions reported during last 2-3 weeks.
User impact:
While the issue exists, some requests to OpenStack might fail (those, which are processed on the affected controller/service). The issue does not disappear with time without manual fix (see workaround below).
Workaround:
The workaround is to restart the affected service. After restart, the service will immediately become operational.
Current plan:
We are planning to fix the issue in updates for 6.1. Right now we are reproducing it with additional logging enabled to understand the root cause.
Detailed analisys by Roman Podoliaka
==========================================================================
Reply queues created by oslo.messaging are not durable (i.e. they are gone after restart of the last RabbitMQ in the cluster). The problem is that after successful failover of RabbitMQ OpenStack services will correctly reconnect, but RPC calls will be broken until we restart the affected service: a reply queue is not recreated, which means no reply can be received for a given call, and the call will eventually fail with TimeoutError.
As it can be seen in the output of commands below, this particular reply queue of nova-conductor first migrated from one RabbitMQ node to another, then saw death of another mirror, and after RabbitMQ server on node-16 was restarted the queue was gone, still nova-conductor RPC client tried to consume messages from it.
rabbitmqctl list_queues: http://xsnippet.org/360736/raw/
root@node-16:~# grep reply_f7cac1a2428d414bb8b9e0a61291a468 -P3 /<email address hidden>: http://xsnippet.org/360737/raw/
This wouldn't be a problem, if a new reply queue was created for new RPC calls, but currently this makes RPC client unusable unless we restart the whole process.
Note: description of the original error in nova-conductor has been put below.
Initial description by Artem Panchenko
==========================================================================
Fuel version info (6.1 build #521 RC1): http://paste.openstack.org/show/277715/
After shutting down of primary controller OSTF tests which create Nova instances fail, because all new booted instances have ERROR state:
http://paste.openstack.org/show/281028/
Here is a part of nova-conductor.log (node-16):
http://paste.openstack.org/show/281014/
RabbitMQ cluster status looks good:
[root@fuel-lab-cz5558 ~]# runc 2 rabbitmqctl cluster_status
DEPRECATION WARNING: /etc/fuel/client/config.yaml exists and will be used as the source for settings. This behavior is deprecated. Please specify the path to your custom settings file in the FUELCLIENT_CUSTOM_SETTINGS environment variable.
node-16.mirantis.com
Cluster status of node 'rabbit@node-16' ...
[{nodes,[{disc,['rabbit@node-16','rabbit@node-7']}]},
{running_nodes,['rabbit@node-7','rabbit@node-16']},
{cluster_name,<<"<email address hidden>">>},
{partitions,[]}]
...done.
node-7.mirantis.com
Cluster status of node 'rabbit@node-7' ...
[{nodes,[{disc,['rabbit@node-16','rabbit@node-7']}]},
{running_nodes,['rabbit@node-16','rabbit@node-7']},
{cluster_name,<<"<email address hidden>">>},
{partitions,[]}]
...done.
Here is AMQP queues info:
http://paste.openstack.org/show/281029/
Steps to reproduce:
1. Create environment: Ubuntu, NeutronGRE, Ceph, Sahara, Ceilometer
2. Add 1 controller, 2 controller+ceph, 1 compute and 3 mongo nodes
3. Deploy changes.
4. Run OSTF
5. Shutdown primary controller (gracefully using `poweroff` command)
6. Run OSTF
Expected result:
- all tests passed except 'Check that required services are running'
Actual:
- all tests which create Nova instances fail
Also, I didn't find why, but all API requests to Nova take a long time, for example `nova list` simple command execution takes 17 seconds:
http://paste.openstack.org/show/281031/
Diagnostic snapshot (environment ID - 2, nodes: 5,16,7,6,11,13,14): https://drive.google.com/file/d/0BzaZINLQ8-xkNDFrX2RKRS1GOWs/view?usp=sharing
This issue can be resolved using 'amqp_durable_