Queue not found:Basic.consume in Nova compute logs

[Ilya Shakhat]

User-visible effect:
HTTP 504 is returned while trying to get instance log data. Other functions like booting instance, working with Neutron objects, Heat stacks work.

Client request as seen by haproxy:
2015-05-27T15:17:47.255551+00:00 info: 172.18.161.232:33094 [27/May/2015:15:16:47.248] nova-api-2 nova-api-2/node-1 2/0/0/-1/60004 504 194 - - sH-- 82/0/0/0/0 0/0 "POST /v2/4f86e1df1a9841a1bfc0f7c44b5c98ea/servers/c34fefa3-e07d-4744-b160-e34ffa3e0742/action HTTP/1.1"

Nova compute log on one of controllers:
2015-05-27 15:16:47.258 5705 DEBUG nova.api.openstack.wsgi [req-afdd1729-0382-47ac-8c97-ed121406483c None] Action: 'action', calling method: <bound method ConsoleOutputController.get_console_output of <nova.api.openstack.compute.contrib.console_output.ConsoleOutputController object at 0x7fe4edb758d0>>, body: {"os-getConsoleOutput": {"length": null}} _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:934
2015-05-27 15:16:47.473 5705 WARNING oslo.messaging._drivers.impl_rabbit [req-afdd1729-0382-47ac-8c97-ed121406483c ] Queue not found:Basic.consume: (404) NOT_FOUND - no queue 'reply_1d33c1c7e7fc40cfb0bd56baec94f9e4' in vhost '/'. Waiting.
2015-05-27 15:16:48.479 5705 WARNING oslo.messaging._drivers.impl_rabbit [req-afdd1729-0382-47ac-8c97-ed121406483c ] Queue not found:Basic.consume: (404) NOT_FOUND - no queue 'reply_1d33c1c7e7fc40cfb0bd56baec94f9e4' in vhost '/'. Waiting.
....
2015-05-27 15:17:46.583 5705 WARNING oslo.messaging._drivers.impl_rabbit [req-afdd1729-0382-47ac-8c97-ed121406483c ] Queue not found:Basic.consume: (404) NOT_FOUND - no queue 'reply_1d33c1c7e7fc40cfb0bd56baec94f9e4' in vhost '/'. Waiting.
2015-05-27 15:17:47.585 5705 ERROR nova.api.openstack [req-afdd1729-0382-47ac-8c97-ed121406483c None] Caught error: Timed out waiting for a reply to message ID a536055214504e1fa8894ec47396ab49.
2015-05-27 15:17:47.595 5705 INFO nova.api.openstack [req-afdd1729-0382-47ac-8c97-ed121406483c None] http://172.18.161.230:8774/v2/4f86e1df1a9841a1bfc0f7c44b5c98ea/servers/c34fefa3-e07d-4744-b160-e34ffa3e0742/action returned with HTTP 500
2015-05-27 15:17:47.600 5705 DEBUG nova.api.openstack.wsgi [req-afdd1729-0382-47ac-8c97-ed121406483c None] Returning 500 to user: The server has either erred or is incapable of performing the requested operation. __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1201
2015-05-27 15:17:47.608 5705 INFO nova.osapi_compute.wsgi.server [req-afdd1729-0382-47ac-8c97-ed121406483c None] Traceback (most recent call last):
2015-05-27 15:17:47.613 5705 INFO nova.osapi_compute.wsgi.server [req-afdd1729-0382-47ac-8c97-ed121406483c None] 192.168.0.2 "POST /v2/4f86e1df1a9841a1bfc0f7c44b5c98ea/servers/c34fefa3-e07d-4744-b160-e34ffa3e0742/action HTTP/1.1" status: 500 len: 0 time: 60.3618622

[Ilya Shakhat (shakhat)]

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "462"
  build_id: "2015-05-24_15-51-50"
  nailgun_sha: "76441596e4fe6420cc7819427662fa244e150177"
  python-fuelclient_sha: "e19f1b65792f84c4a18b5a9473f85ef3ba172fce"
  astute_sha: "0bd72c72369e743376864e8e8dabfe873d40450a"
  fuel-library_sha: "889c2534ceadf8afd5d1540c1cadbd913c0c8c14"
  fuel-ostf_sha: "9a5f55602c260d6c840c8333d8f32ec8cfa65c1f"
  fuelmain_sha: "5c8ebddf64ea93000af2de3ccdb4aa8bb766ce93"

Deployment: 3 controllers + 1 compute/cinder + 1 compute, Neutron GRE

[Ilya Shakhat (shakhat)]

Note that the issue appeared on cluster that survived several controller reboots

[Ilya Shakhat (shakhat)]

Snapshot: https://drive.google.com/a/mirantis.com/file/d/0B1mJ0eQoi7tEQ3UxVVVoSXh0aHc/view

[Viktor Serhieiev (vsergeyev)]

Can this bug be reproduced on the last ISO build?

[Ilya Shakhat (shakhat)]

@Victor - the issue is reproduced on ISO 511