Steps to reproduce:
1. Create new cluster with the following configuration:
- Ubuntu Simple;
- nova network VLAN with tagged networks;
- Cinder LVM for storage;
- ceilometer and savanna as additional services;
2. Add the following nodes:
- 1 controller;
- 1 compute;
- 1 cinder;
3. Deploy cluster.
4. Check cluster was deployed successfully and run the OSTF tests.
Expected result:
All OSTF tests pass.
Actual result:
OSTF test "Ceilometer create, update, check, delete alarm." fails. There are no metrics created, which was verified by executing command "ceilometer meter-list" By investigating ceilometer logs the following errors were found:
Errors from ceilometer-alarm-notifier.log:
2013-12-27 12:54:30.412 10044 ERROR ceilometer.openstack.common.rpc.common [-] Failed to consume message from queue: 320: (CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', (0, 0), None)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common Traceback (most recent call last):
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 577, in ensure
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common return method(*args, **kwargs)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/ceilometer/openstack/common/rpc/impl_kombu.py", line 657, in _consume
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common return self.connection.drain_events(timeout=timeout)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 281, in drain_events
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common return connection.drain_events(**kwargs)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 286, in drain_events
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common return amqp_method(channel, args)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 491, in _close
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common raise ConnectionError(reply_code, reply_text, (class_id, method_id))
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common ConnectionError: 320: (CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', (0, 0), None)
2013-12-27 12:54:30.412 10044 TRACE ceilometer.openstack.common.rpc.common
Errors from ceilometer-alarm-evaluator.log:
2013-12-27 12:56:35.616 10012 ERROR ceilometer.alarm.service [-] alarm evaluation cycle failed
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service Traceback (most recent call last):
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 95, in _evaluate_assigned_alarms
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service alarms = self._assigned_alarms()
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 137, in _assigned_alarms
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service return self._client.alarms.list(q=[{'field': 'enabled',
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 90, in _client
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service self.api_client = ceiloclient.get_client(2, **creds)
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 77, in get_client
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service _ksclient = _get_ksclient(**ks_kwargs)
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 33, in _get_ksclient
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service insecure=kwargs.get('insecure'))
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 139, in __init__
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service self.authenticate()
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 467, in authenticate
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service resp, body = self.get_raw_token_from_identity_service(**kwargs)
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 166, in get_raw_token_from_identity_service
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service "%s" % e)
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service AuthorizationFailure: Authorization Failed: An unexpected error prevented the server from fulfilling your request. (ProgrammingError) (1146, "Table 'keystone.domain' doesn't exist") 'SELECT domain.id AS domain_id, domain.name AS domain_name, domain.enabled AS domain_enabled, domain.extra AS domain_extra \nFROM domain \nWHERE domain.id = %s' ('default',) (HTTP 500)
2013-12-27 12:56:35.616 10012 TRACE ceilometer.alarm.service
2013-12-27 12:57:35.618 10012 ERROR ceilometer.alarm.service [-] alarm evaluation cycle failed
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service Traceback (most recent call last):
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 95, in _evaluate_assigned_alarms
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service alarms = self._assigned_alarms()
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 137, in _assigned_alarms
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service return self._client.alarms.list(q=[{'field': 'enabled',
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometer/alarm/service.py", line 90, in _client
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service self.api_client = ceiloclient.get_client(2, **creds)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 77, in get_client
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service _ksclient = _get_ksclient(**ks_kwargs)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/ceilometerclient/client.py", line 33, in _get_ksclient
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service insecure=kwargs.get('insecure'))
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 139, in __init__
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service self.authenticate()
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 467, in authenticate
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service resp, body = self.get_raw_token_from_identity_service(**kwargs)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 160, in get_raw_token_from_identity_service
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service token=token)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/v2_0/client.py", line 191, in _base_authN
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service resp, body = self.request(url, 'POST', body=params, headers=headers)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 609, in request
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service **request_kwargs)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service File "/usr/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 123, in request
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service raise exceptions.from_response(resp, method, url)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service Unauthorized: The request you have made requires authentication. (HTTP 401)
2013-12-27 12:57:35.618 10012 TRACE ceilometer.alarm.service
Reproduced on 27_01-17- 41", 18d1df8c6597588 b853d02de4c" , 58d1078e467d764 4855549879a" , 35459ef9062a1ab ce66a60f79e" , e9c3bf8d8ad6cd6 a13caa6e4ac" , 2719e3af9a91d65 9c1956ee610"
"build_id": "2014-01-
"ostf_sha": "338ddf840c2299
"build_number": "63",
"nailgun_sha": "a77e67aa73bf7e
"fuelmain_sha": "93cb99fa035cea
"astute_sha": "f967bf7d643c35
"release": "4.1",
"fuellib_sha": "b118aefcc71b1a
1. Create new environment (Ubuntu, simple mode)
2. Chose VLAN segmentation
3. Choose installing Ceph for images and Ceilometer
4. Add controller, compute, cinder and 2 ceph nodes
5. Untag Management network and move it to other interface
6. Start deployment. It was successful
7. Start OSTF tests
8. Ceilometer tests has failed on step with creating metrics
Errors from ceilometer- alarm-notifier. log:
2014-01-27 08:34:11.134 10760 ERROR ceilometer. openstack. common. rpc.common [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 9 seconds. openstack. common. rpc.common [-] AMQP server on localhost:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 11 seconds. openstack. common. rpc.common [-] Failed to consume message from queue: 320: (CONNECTION_FORCED - broker forced connection closure with reason 'shutdown', (0, 0), None) openstack. common. rpc.common Traceback (most recent call last):
2014-01-27 08:34:20.148 10760 ERROR ceilometer.
2014-01-27 08:37:26.674 10760 ERROR ceilometer.
2014-01-27 08:37:26.674 10760 TRACE ceilometer.
Errors from ceilometer- alarm-evaluator .log:
2014-01-27 08:34:54.843 10728 ERROR ceilometer. alarm.service [-] alarm evaluation cycle failed alarm.service Traceback (most recent call last): alarm.service File "/usr/lib/ python2. 7/dist- packages/ ceilometer/ alarm/service. py", line 95, in _evaluate_ assigned_ alarms alarm.service alarms = self._assigned_ alarms( ) alarm.service File "/usr/lib/ python2. 7/dist- packages/ ceilometer/ alarm/service. py", line 137, in _assigned_alarms alarm.service return self._client. alarms. list(q= [{'field' : 'enabled', alarm.service File "/usr/lib/ python2. 7/dist- packages/ ceilometer/ alarm/service. py", line 90, in _client alarm.service self.api_client = ceiloclient. get_client( 2, **creds) alarm.service File "/usr/lib/ python2. 7/dist- packages/ ceilometerclien t/client. py", line 77, in get_client alarm.service _ksclient = _get_ksclient( **ks_kwargs) alarm.service File "/usr/lib/ python2. 7/dist- packages/ ceilometerclien t/client. py", line 33, in _get_ksclient alarm.service insecure= kwargs. get('insecure' )) alarm.service File "/usr/lib/ python2. 7/dist- packages/ keystoneclient/ v2_0/client. py", line 139, in __init__
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2014-01-27 08:34:54.843 10728 TRACE ceilometer.
2...