Create Ceilometer alarm from Heat no authorized

[magicboiz]

After a fresh install of MOS 8.0, a regular user cannot deploy heat templates which create autoscaling alarms.

A simple template like this: http://pastebin.com/D7cUxVuC
gets this error: Resource CREATE failed: Forbidden: resources.cpu_alarm_high: You are not authorized to perform the requested action. (HTTP 403) (Request-ID:........)

The heat resource which fires the error is this:

  cpu_alarm_high:
    type: OS::Ceilometer::Alarm
    properties:
      description: Scale-up if the average CPU > 50% for 1 minute
      meter_name: cpu_util
      statistic: avg
      period: 60
      evaluation_periods: 1
      threshold: 50
      alarm_actions:
        - {get_attr: [scale_up_policy, alarm_url]}
      matching_metadata: {"metadata.user_metadata.stack": {get_param: "OS::stack_id"}}
      comparison_operator: gt

Nevertheless, a regular user can create the alarm through the ceilometer CLI:

Nova instance:

[user1@linux]$ nova list
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------+
| efd35884-a3ff-4977-a172-d9b693d17807 | t1 | ACTIVE | - | Running | red01=10.0.1.142 |
+--------------------------------------+------+--------+------------+-------------+------------------+

Check meter/samples for the instance:

[user1@linux]$ ceilometer sample-list -q resource_id=efd35884-a3ff-4977-a172-d9b693d17807 -m cpu_util | head -n 5
+--------------------------------------+----------+-------+--------------------+------+----------------------------+
| Resource ID | Name | Type | Volume | Unit | Timestamp |
+--------------------------------------+----------+-------+--------------------+------+----------------------------+
| efd35884-a3ff-4977-a172-d9b693d17807 | cpu_util | gauge | 5.861807444732659 | % | 2016-05-18T14:52:32.770000 |
| efd35884-a3ff-4977-a172-d9b693d17807 | cpu_util | gauge | 5.8543203133394 | % | 2016-05-18T14:47:31.534000 |

Create the alarm:

[user1@linux]$ ceilometer alarm-threshold-create --name cpu_high_alarm --description 'CPU usage high' --meter-name cpu_util --threshold 5 --comparison-operator gt --statistic avg --period 300 --evaluation-periods 1 --query resource_id=efd35884-a3ff-4977-a172-d9b693d17807
+---------------------------+---------------------------------------------------------+
| Property | Value |
+---------------------------+---------------------------------------------------------+
| alarm_actions | [] |
| alarm_id | 339ead8e-f772-4653-b797-21962a29f9c7 |
| comparison_operator | gt |
| description | CPU usage high |
| enabled | True |
| evaluation_periods | 1 |
| exclude_outliers | False |
| insufficient_data_actions | [] |
| meter_name | cpu_util |
| name | cpu_high_alarm |
| ok_actions | [] |
| period | 300 |
| project_id | 65f8d3591f2c478e8373cbabcdcd2302 |
| query | resource_id == efd35884-a3ff-4977-a172-d9b693d17807 AND |
| | project_id == 65f8d3591f2c478e8373cbabcdcd2302 |
| repeat_actions | False |
| severity | low |
| state | insufficient data |
| statistic | avg |
| threshold | 5.0 |
| type | threshold |
| user_id | 202e3502ad9d486b9b403ffe60dc2bf1 |
+---------------------------+---------------------------------------------------------+

And check the alarm status:

[user1@linux]$ ceilometer alarm-show 339ead8e-f772-4653-b797-21962a29f9c7
+---------------------------+---------------------------------------------------------+
| Property | Value |
+---------------------------+---------------------------------------------------------+
| alarm_actions | [] |
| alarm_id | 339ead8e-f772-4653-b797-21962a29f9c7 |
| comparison_operator | gt |
| description | CPU usage high |
| enabled | True |
| evaluation_periods | 1 |
| exclude_outliers | False |
| insufficient_data_actions | [] |
| meter_name | cpu_util |
| name | cpu_high_alarm |
| ok_actions | [] |
| period | 300 |
| project_id | 65f8d3591f2c478e8373cbabcdcd2302 |
| query | resource_id == efd35884-a3ff-4977-a172-d9b693d17807 AND |
| | project_id == 65f8d3591f2c478e8373cbabcdcd2302 |
| repeat_actions | False |
| severity | low |
| state | insufficient data |
| statistic | avg |
| threshold | 5.0 |
| type | threshold |
| user_id | 202e3502ad9d486b9b403ffe60dc2bf1 |
+---------------------------+---------------------------------------------------------+

In short: the user can create the alarm with ceilometer CLI, but through heat there's an authorization problem.

[Peter Razumovsky (prazumovsky)]

It cannot be reproduced in Mitaka release. If this issue will appear in Mitaka, reopen bug for Mitaka.

[magicboiz (magicboiz)]

Hi,

is there any patch available for Liberty/MOS8.0?

[Oleksii Chuprykov (ochuprykov)]

Hi, magicboiz. I can't reproduce it on my MOS8.0 setup. Probably you have MOS8.0 with Sahara on board?
Do you have reauthentication_auth_method=trusts in heat.conf?

[magicboiz (magicboiz)]

Hi

yes, I have Sahara deployed, and yes, I have /etc/heat/heat.conf:reauthentication_auth_method=trusts

is there any incompatibility between heat and sahara?

[Oleksii Chuprykov (ochuprykov)]

Heat is deployed with reauthentication_auth_method=trusts if Sahara is enabled. There is an incompatibility with this option and ceilometer client plugin. We've already fixed it in mitaka/9.0, but didn't backport it to liberty.
Could you please try applying this patch http://paste.openstack.org/show/508621/ ?
Fix in upstream here: https://review.openstack.org/#/c/326405/

[magicboiz (magicboiz)]

ok, something has changed. This is the path log result:

patching file heat/engine/clients/os/ceilometer.py
Using Plan A...
Hunk #1 succeeded at 22.
Hunk #2 succeeded at 30.
Hunk #3 succeeded at 42.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/heat/engine/stack.py b/heat/engine/stack.py
|index 744c99a..c7862e9 100644
|--- a/heat/engine/stack.py
|+++ b/heat/engine/stack.py
--------------------------
patching file heat/engine/stack.py
Using Plan A...
Hunk #1 succeeded at 190.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/heat/tests/fakes.py b/heat/tests/fakes.py
|index 585875a..e94115d 100644
|--- a/heat/tests/fakes.py
|+++ b/heat/tests/fakes.py
--------------------------
patching file heat/tests/fakes.py
Using Plan A...
Hunk #1 succeeded at 99.
Hunk #2 succeeded at 113.
Hunk #3 succeeded at 202.
done

And now, when I try to create the same heat template, instead of Authorization problem, I get this other error:

EndpointNotFound: resources.cpu_alarm_high: publicURL endpoint for alarming service in RegionOne region not found

[magicboiz (magicboiz)]

I a lab environment, I've just redeployed MOS8.0 (with sahara enabled), applied patchset https://review.openstack.org/#/c/326405/ and again I'm gettting the same error creating the stack from heat:

cpu_alarm_low
  OS::Ceilometer::Alarm 3 minutes Create Failed EndpointNotFound: resources.cpu_alarm_low: publicURL endpoint for alarming service in RegionOne region not found

[Oleksii Chuprykov (ochuprykov)]

magicboiz, I'm working on it now. One possible quick fix here is to remove reauthentication_auth_method=trusts from your heat.conf.

[magicboiz (magicboiz)]

Oleksii, removing "reauthentication_auth_method=trusts" from heat.conf (and reversing patchset https://review.openstack.org/#/c/326405/ ) seems to do the job, but now my question would be: any implications for Sahara? Why fuel does enable "reauthentication_auth_method=trusts" when deploying Sahara?

[Oleksii Chuprykov (ochuprykov)]

I'll upload new fix ASAP.
>>would be: any implications for Sahara?
This option was added to support reauthentication. If your stack is deployed more than 1hr(default value), it will fail, because of token expiration. In other situations it should work as usual.

[magicboiz (magicboiz)]

Sahara and Murano stacks usually live longer than 1hr.....isn't?

[Oleksii Chuprykov (ochuprykov)]

New patch: http://paste.openstack.org/show/508949/
Fix: https://review.fuel-infra.org/#/c/21844/
I hope this will help.

[magicboiz (magicboiz)]

I have no rights to access https://review.fuel-infra.org/#/c/21844/

[Oleksii Chuprykov (ochuprykov)]

Upstream fix: https://review.openstack.org/#/c/327716/

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/21844
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: 11cc95348dc3cf50147af36363bcf0cfaf4383de
Author: huangtianhua <email address hidden>
Date: Thu Aug 4 10:03:47 2016

Make sure create ceilometer alarm successful

If enable re-auth, we will use the stored context
instead of request context, then we can't create
ceilometer alarm resource. There are two problems
when create ceilometer client:
1. the stored context has no domain info, an error
raised from keystone: BadRequest: Expecting to find
domain in project. So this patch will retrive the
user/project domain ids from the auth_ref.
2. after fix the first problem, then another error
raised from keystone: Forbidden: You are not authorized
to perform the requested action. Due keystone doesn't
allow to create a token by a trust-scoped token when
get aodh endpoint. So this patch will pass 'aodh_endpoint'
to ceilometer client to avoid this.

Closes-Bug: #1583208
(cherry picked from commit 20214477c80759836b10d1ae45d16d404b077463)

Conflicts:
 heat/engine/clients/os/ceilometer.py

Change-Id: I44ed5c10b6dec6f39714f4f74cf51a10ef6104a6

[Ekaterina Shutova (eshutova)]

Verified on MOS 8.0 + mu3 updates.

Before updates bug was reproduced:

| stack_status_reason | Resource CREATE failed: Forbidden: |
| | resources.cpu_alarm_high: You are not authorized to |
| | perform the requested action. (HTTP 403) (Request-ID: |
| | req-5e5b49d0-18a5-4b1d-b84f-6a2d66789475)

With fix no autentification error appeared:
root@node-11:~# heat stack-list
+--------------------------------------+------------+-----------------+---------------------+--------------+
| id | stack_name | stack_status | creation_time | updated_time |
+--------------------------------------+------------+-----------------+---------------------+--------------+
| 7ff01ee8-b9c6-4397-ab8b-bb601105deb9 | TESTA | CREATE_COMPLETE | 2016-08-31T15:35:02 | None |
+--------------------------------------+------------+-----------------+---------------------+--------------+