Cisco Openstack

nova-conductor fails in full_ha

Bug #1269975 reported by Chris Ricker on 2014-01-16

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cisco Openstack	New	Undecided	Unassigned

Bug Description

With full_ha, I see nova-conductor failing on all my control nodes:

root@ci-os-con1:~# nova-manage service list | grep XXX
nova-conductor ci-os-con1 internal enabled XXX None
nova-conductor ci-os-con2 internal enabled XXX None
nova-conductor ci-os-con3 internal enabled XXX None
root@ci-os-con1:~#

log shows a trace accessing a rabbit queue:

2014-01-16 22:28:36.615 19963 ERROR nova.openstack.common.threadgroup [-] 406: (PRECONDITION_FAILED - inequivalent arg 'x-ha-policy'for queue 'conductor' in vhost '/': received the value 'all' of type 'longstr' but current is none, (50, 10), None)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 117, in wait
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup x.wait()
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 49, in wait
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 65, in run_service
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup service.start()
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/service.py", line 176, in start
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup self.conn.create_consumer(self.topic, rpc_dispatcher, fanout=False)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 163, in create_consumer
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup self.connection.create_consumer(topic, proxy, fanout)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 761, in create_consumer
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup self.declare_topic_consumer(topic, proxy_cb)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 708, in declare_topic_consumer
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup topic, callback)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 632, in declare_consumer
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup return self.ensure(_connect_error, _declare_consumer)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 577, in ensure
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup return method(*args, **kwargs)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 628, in _declare_consumer
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup self.consumer_num.next())
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 270, in __init__
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup **options)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 134, in __init__
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup self.reconnect(channel)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 141, in reconnect
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup self.queue.declare()
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 493, in declare
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup self.queue_declare(nowait, passive=False)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 519, in queue_declare
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup nowait=nowait)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 1241, in queue_declare
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup (50, 11), # Channel.queue_declare_ok
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 71, in wait
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup return self.dispatch_method(method_sig, args, content)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 88, in dispatch_method
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup return amqp_method(self, args)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 224, in _close
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup raise ChannelError(reply_code, reply_text, (class_id, method_id))
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup ChannelError: 406: (PRECONDITION_FAILED - inequivalent arg 'x-ha-policy'for queue 'conductor' in vhost '/': received the value 'all' of type 'longstr' but current is none, (50, 10), None)
2014-01-16 22:28:36.615 19963 TRACE nova.openstack.common.threadgroup

Revision history for this message

Chris Ricker (chris-ricker) wrote on 2014-01-16:

purging queues resolves:

# on one controller
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app

# on all 3 controllers
service nova-conductor start

so need to check how we're doing mirrored queues

Revision history for this message

Chris Ricker (chris-ricker) wrote on 2014-01-17:

I don't see this on a second build

Revision history for this message

Mark T. Voelker (mvoelker) wrote on 2014-01-17:

Setting incomplete until we get confirmation that his wasn't a one-off during development.

Changed in openstack-cisco:
status:	New → Incomplete

Mark T. Voelker (mvoelker) on 2014-01-24

Changed in openstack-cisco:
milestone:	h.1 → none

Revision history for this message

Chris Ricker (chris-ricker) wrote on 2014-01-31:

hit the same underlying issue on a current coi-dev full_ha, this time with nova-consoleauth as the unlucky victim of rabbit shared queue borkage:

# nova-manage service list | grep XXX
nova-consoleauth ci-os-con1 internal enabled XXX None
nova-consoleauth ci-os-con2 internal enabled XXX None
nova-consoleauth ci-os-con3 internal enabled XXX None
#

2014-01-31 17:33:32.684 9948 TRACE nova.openstack.common.threadgroup ChannelError: 406: (PRECONDITION_FAILED - inequivalent arg 'x-ha-policy'for queue 'consoleauth' in vhost '/': received the value 'all' of type 'longstr' but current is none, (50, 10), None)

in nova-consoleauth.log

as with the earlier nova-conductor case, resetting the queue resolved:

rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app

Changed in openstack-cisco:
status:	Incomplete → New

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.