CeilometerAlarmTest fails with resources.alarm: Expecting to find domain in project

[Rabi Mishra]

http://logs.openstack.org/16/261216/1/check/gate-heat-dsvm-functional-orig-mysql/da98451/console.html#_2015-12-24_08_51_05_504

http://logs.openstack.org/16/261216/1/check/gate-heat-dsvm-functional-orig-mysql/da98451/logs/screen-h-eng.txt.gz?level=ERROR#_2015-12-24_08_48_41_792

Possibly looks like it's due to the recent devstack change.

https://review.openstack.org/#/c/254755/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/261272

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/261272
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=7ae4b3212d04728079ff7882d84a6a6c0318de6c
Submitter: Jenkins
Branch: master

commit 7ae4b3212d04728079ff7882d84a6a6c0318de6c
Author: Rabi Mishra <email address hidden>
Date: Thu Dec 24 16:33:20 2015 +0530

    Disable CeilometerAlarmTest.test_alarm

    Gate seems to be broken due to some recent change. Possibly
    https://review.openstack.org/#/c/254755/. Disabling the test
    till it's sorted.

    Change-Id: Iaae3305b3844be8c20c0652f544abe6a4844fe00
    Partial-Bug: #1529058

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/262129

[Zi Lian Ji (jizilian)]

It seems to lead to the failure of ceilometer jenkins.

http://logs.openstack.org/60/262660/2/check/gate-ceilometer-dsvm-integration/e7b4e07//logs/devstack-gate-post_test_hook.txt.gz

[Zi Lian Ji (jizilian)]

The exception is thrown by the code:
https://github.com/openstack/keystone/blob/master/keystone/auth/controllers.py#L199

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/262129
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=5eec4c080f5a4dd7fc81eaeba9a734427c789255
Submitter: Jenkins
Branch: master

commit 5eec4c080f5a4dd7fc81eaeba9a734427c789255
Author: Rabi Mishra <email address hidden>
Date: Tue Dec 29 10:14:11 2015 +0530

    Pass domain id to ceilometer client

    Pass user/project domain id as `user_domain_id/project_domain_id`
    to ceilometer client.

    Change-Id: Ibbd363dc1d2e56fda98a5bd158a403c2f0941e85
    Co-Authored-By: huangtianhua <email address hidden>
    Closes-Bug: #1529058

[Zi Lian Ji (jizilian)]

Hi Rabi

When heat initializes the ceilometer client, it transfers the parameter 'project_name' to ceilometer client and its value is not null.

https://github.com/openstack/heat/blob/master/heat/engine/clients/os/ceilometer.py#L38

but it doesn't transfer the detailed info about domain of the project. I think domain is important to keystone v3. So it leads the exception is thrown by the following code.

https://github.com/openstack/keystone/blob/master/keystone/auth/controllers.py#L199

Do you have any suggestions about it?

Thanks

[Zi Lian Ji (jizilian)]

Hi Liu Sheng

Do you have any suggestions about it ?

Thanks

The call stack is:

2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource alarm = self.client().alarms.create(**props)
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/opt/stack/new/heat/heat/engine/resource.py", line 543, in client
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource return self.stack.clients.client(client_name)
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/opt/stack/new/heat/heat/engine/clients/__init__.py", line 70, in client
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource return client_plugin.client()
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/opt/stack/new/heat/heat/engine/clients/client_plugin.py", line 136, in client
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource self._client = self._create()
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/opt/stack/new/heat/heat/engine/clients/os/ceilometer.py", line 48, in _create
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource return cc.get_client('2', **args)
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 419, in get_client
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource return Client(version, endpoint, **kwargs)
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 366, in Client
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource return client_class(*args, **client_kwargs)
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 66, in __init__
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource self.alarm_client, aodh_enabled = self._get_alarm_client(**kwargs)
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/v2/client.py", line 120, in _get_alarm_client
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource kwargs.get('timeout'))
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 276, in redirect_to_aodh_endpoint
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource self.opts['endpoint'] = _get_endpoint(ks_session, **ks_kwargs)
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/ceilometerclient/client.py", line 206, in _get_endpoint
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource region_name=kwargs.get('region_name'))
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 664, in get_endpoint
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource return auth.get_endpoint(self, **kwargs)
2015-12-24 08:48:41.792 27481 ERROR heat.engine.resource File "/usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/base.py", line 324, in get_endpoint
2015-12-24 08:48:41.792 27481 E...

[gordon chung (chungg)]

this is resolved by heat

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/263749

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/heat 6.0.0.0b2 development milestone.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/heat 6.0.0.0b2 development milestone.

[Pallavi (pallavi-nagboth)]

We are also facing the same problem. We are using openstack liberty on ubuntu. We have a heat template where we try to create OS::Ceilometer::Alarm. This resource creation is failing with the following error in heat-engine.log:

2016-01-20 10:40:45.143 17336 ERROR heat.engine.resource BadRequest: Expecting to find domain in project - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-7d9b53cb-e44b-42d6-96e4-08dd19a32915)

We fixed the bug by adding the following line inheat/engine/clients/os/ceilometer.py:

args = {
            'auth_url': con.auth_url,
            'service_type': self.METERING,
            'project_name': con.tenant,
            'project_id': con.tenant_id, -----------------> Added this line
            'token': lambda: self.auth_token,
            'endpoint_type': endpoint_type,
            'os_endpoint': endpoint,
            'cacert': self._get_client_option('ceilometer', 'ca_file'),
            'cert_file': self._get_client_option('ceilometer', 'cert_file'),
            'key_file': self._get_client_option('ceilometer', 'key_file'),
            'insecure': self._get_client_option('ceilometer', 'insecure')
        }

ubuntu@controller0:/usr/lib/python2.7/dist-packages/heat/engine/clients/os$ heat --version
0.8.0
ubuntu@controller0:/usr/lib/python2.7/dist-packages/heat/engine/clients/os$ keystone --version
/usr/lib/python2.7/dist-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
1.7.1
ubuntu@controller0:/usr/lib/python2.7/dist-packages/heat/engine/clients/os$ ceilometer --version
1.5.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/284125

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/284127

[Prateek khushalani (prateek-khushalani)]

Any workaround for this bug ???

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergey Kraynev (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/284125
Reason: We abandon this patch due to EOL for liberty and broken gate state.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergey Kraynev (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/284127
Reason: According decision on weekly team meeting, we need to abandon this patch due to EOL for liberty and broken gate state.