Ubuntu: wrong ceilometer startup sequence

[Aleksandr Didenko]

{"build_id": "2013-12-27_00-24-14", "ostf_sha": "83ada35fec2664089e07fdc0d34861ae2a4d948a", "build_number": "214", "nailgun_sha": "af1598bcc9faf468d4d9265cc5c51fa8cea53136", "fuelmain_sha": "17eed776b30886851ae0042fa7a30184f5cd8eb6", "astute_sha": "6ce36837882399e0d3bb1ffdb2c3b2d8dcb84b54", "release": "4.0", "fuellib_sha": "eebe07913ee09311c8e7c9231f6785081327dc0e"}

Steps to reproduce:
1. Deploy simple Ubuntu environment (1 controller, 1 cinder, 1 compute), neutron + gre, with savanna, murano and ceilometer enabled.
2. Run OSTF

Result:
Platform test "Ceilometer create, update, check, delete alarm" fails with error:
"Time limit exceeded while waiting for metrics created to finish. Please refer to OpenStack logs for more details."

Diagnostic snapshot attached.

Part of /var/log/ceilometer/ceilometer-collector.log from controller node:

2013-12-27 14:07:35 ERROR ceilometer.openstack.common.threadgroup [-] 406: (PRECONDITION_FAILED - inequivalent arg 'x-ha-policy'for queue 'ceilometer.collector' in vhost '/': received the value 'all' of type 'longstr' but current is none, (50, 10), None)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup Traceback (most recent call last):
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/threadgroup.py", line 117, in wait
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup x.wait()
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/threadgroup.py", line 49, in wait
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup return self.thread.wait()
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup return self._exit_event.wait()
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup return hubs.get_hub().switch()
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup return self.greenlet.switch()
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup result = function(*args, **kwargs)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/service.py", line 448, in run_service
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup service.start()
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/collector/service.py", line 138, in start
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup super(CollectorService, self).start()
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/service.py", line 56, in start
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup self.conn.create_consumer(self.topic, dispatcher, fanout=False)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/amqp.py", line 163, in create_consumer
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup self.connection.create_consumer(topic, proxy, fanout)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 761, in create_consumer
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup self.declare_topic_consumer(topic, proxy_cb)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 708, in declare_topic_consumer
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup topic, callback)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 632, in declare_consumer
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup return self.ensure(_connect_error, _declare_consumer)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 577, in ensure
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup return method(*args, **kwargs)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 628, in _declare_consumer
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup self.consumer_num.next())
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 270, in __init__
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup **options)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 134, in __init__
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup self.reconnect(channel)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 141, in reconnect
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup self.queue.declare()
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 493, in declare
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup self.queue_declare(nowait, passive=False)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/kombu/entity.py", line 519, in queue_declare
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup nowait=nowait)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 1241, in queue_declare
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup (50, 11), # Channel.queue_declare_ok
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 71, in wait
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup return self.dispatch_method(method_sig, args, content)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 88, in dispatch_method
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup return amqp_method(self, args)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup File "/usr/lib/python2.7/dist-packages/amqp/channel.py", line 224, in _close
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup raise ChannelError(reply_code, reply_text, (class_id, method_id))
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup ChannelError: 406: (PRECONDITION_FAILED - inequivalent arg 'x-ha-policy'for queue 'ceilometer.collector' in vhost '/': received the value 'all' of type 'longstr' but current is none, (50, 10), None)
2013-12-27 14:07:35.300 20548 TRACE ceilometer.openstack.common.threadgroup

[Aleksandr Didenko (adidenko)]

Archive with /var/log/ceilometer/ directory from controller node attached.

[Vladimir Kuklin (vkuklin)]

This issue should happen only on Ubuntu as it starts packages on installation with default ha policy and queues created without ha policies. Thus, we need to make ceilometer start only after configurations files are filled.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/74965

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/74965
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6e28f561b0aa138a2b0931c9928e932feaa809ed
Submitter: Jenkins
Branch: master

commit 6e28f561b0aa138a2b0931c9928e932feaa809ed
Author: Dmitry Ilyin <email address hidden>
Date: Thu Feb 20 12:52:36 2014 +0400

    Add overrides for services

    To prevent their autorun at Ubuntu deployment
    Closes-Bug: 1264530

    Change-Id: I25019d9c9f5e5a24276f8ad5ddd551904c823869

[Tatyanka (tatyana-leontovich)]

verified on 214 iso