Cinder fails to communicate with rabbitmq

[Ashish]

I was playing with OS newton release on ubuntu 16.04.

I 've a OS installation with Horizon, compute, Neutron, Heat. All of them were working nicely. I thought of starting with Cinder but it's not working.

Below are the details from my node. I 've installed cinder on one of my compute node.
root@compute2:~# cinder --version
1.9.0

alepo@compute1:~$ cinder service-list
+------------------+--------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+--------------+------+---------+-------+----------------------------+-----------------+
| cinder-scheduler | controller | nova | enabled | up | 2016-11-25T08:37:56.000000 | - |
| cinder-volume | compute2@lvm | nova | enabled | down | 2016-11-25T06:49:52.000000 | - |
+------------------+--------------+------+---------+-------+----------------------------+-----------------+

Logs from compute2 node.

2016-11-25 14:20:43.218 9613 ERROR oslo_service.service return amqp_method(self, args)
2016-11-25 14:20:43.218 9613 ERROR oslo_service.service File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 243, in _close
2016-11-25 14:20:43.218 9613 ERROR oslo_service.service reply_code, reply_text, (class_id, method_id), ChannelError,
2016-11-25 14:20:43.218 9613 ERROR oslo_service.service PreconditionFailed: Exchange.declare: (406) PRECONDITION_FAILED - inequivalent arg 'durable' for exchange 'openstack' in vhost '/': received 'false' but current is 'true'
2016-11-25 14:20:43.218 9613 ERROR oslo_service.service
2016-11-25 14:21:13.225 9613 WARNING oslo_messaging.server [req-a97a2bd0-dc0b-454b-937c-7b8cd34eb5c4 - - - - -] Possible hang: wait is waiting for stop to complete

Logs from AMQ server

=ERROR REPORT==== 25-Nov-2016::14:17:22 ===
Channel error on connection <0.941.1> (172.16.110.209:37836 -> 172.16.110.254:5672, vhost: '/', user: 'openstack'), channel 1:
{amqp_error,precondition_failed,
            "inequivalent arg 'durable' for exchange 'openstack' in vhost '/': received 'false' but current is 'true'",
            'exchange.declare'}

[Ashish (ashish235)]

Well I could fix the problem by adding a config in the cinder.conf file under default section.

rabbit_durable_queues = true.

After I added the conf parameter I can see the connection is established. If this is a generic configuration it should be documented.

[Pratik Pravin Bandarkar (pratik-bandarkar)]

Hi,

I am unable to reproduce the issue with same cinder version:
# cinder --version
1.9.0

<snip>
# Use durable queues in AMQP. (boolean value)
# Deprecated group/name - [DEFAULT]/amqp_durable_queues
# Deprecated group/name - [DEFAULT]/rabbit_durable_queues
#amqp_durable_queues = false

# cinder service-list
+------------------+------------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+------------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup | xxxx | nova | enabled | up | 2016-12-08T12:33:22.000000 | - |
| cinder-scheduler | xxxx | nova | enabled | up | 2016-12-08T12:33:26.000000 | - |
| cinder-volume | xxxx | nova | enabled | up | 2016-12-08T12:33:26.000000 | - |
+------------------+------------------------------+------+---------+-------+----------------------------+-----------------+
</snip>

[jun923.gu (gujun1989)]

I think I encounter the same problem, and I can reproduce the the problem. I had submitted a bug in oslo.messaging. The link is: https://bugs.launchpad.net/oslo.messaging/+bug/1773070. From reply of Ken Giusti (kgiusti), I think the bug maybe belong cinder.

[jun923.gu (gujun1989)]

This bug should be fixed. The link is: https://review.openstack.org/#/c/576327/

[Christian Sarrasin (sxc731)]

FYIW we just experienced that symptom on two out of three cinder-scheduler instances that came up in a brand new deployment (running latest Queens Cinder version 12.0.4 as deployed by OSA 17.0.8).

In addition to the symptomatic error msg ("WARNING oslo_messaging.server [req-5ea6475e-e930-4ff0-9d17-20189aa85ba1 - - - - -] Possible hang: wait is waiting for stop to complete") cinder-scheduler was gobbling up 100% CPU. A bounce of both hung processes cured the issue.