Alarms are only evaluated if the alarm creator is the same as the service credentials used on the server

[Sam Morrison]

When I create an alarm it never gets evaluated by ceilometer-alarm-evaluator. The query this service does always returns 0 alarms.

The only way I can get it to work if I create the alarm with the same credentials as the service credentials set on the ceilometer server.

Is the alarm stuff in ceilometer ready for use? I can't find any documentation and it seems it doesn't work?

The user I have defined in service credentials has a role which relates to context_is_admin in the policy.json but it seems it can only list alarms that it owns.

[ZhiQiang Fan (aji-zqfan)]

# keystone user-role-list --user ceilometer --tenant service

[Sam Morrison (sorrison)]

Why has this been marked as incomplete and what does your comment mean?

[Sam Morrison (sorrison)]

Setting back to new, as unclear why it is incomplete

[Nejc Saje (nejc-saje)]

Hey Sam, incomplete just means that more feedback is needed. ZhiQiang Fan provided a command for you to run in the problematic environment and post the output here.

[Sam Morrison (sorrison)]

Hi ZhiQiang,

The ceilometer user has the admin role in the service tenant. That isn't my problem though.

My account only has the member role. Let me know what other info you need.

Sam

[Nejc Saje (nejc-saje)]

Hey, I'm not the expert on ceilometer<->keystone interaction, but let's try to solve this. I did some steps here http://paste.openstack.org/show/86657/ on a fresh devstack installation and it seems to work fine. Can you try to do similar steps and report the differences? Please note that `. openrc <user> <tenant>` switches environment credentials to the given user and tenant.

[Sam Morrison (sorrison)]

That paste you sent indicates that it doesn't work. The State is in "insufficient data" due to this bug. (although I can't exactly confirm that, just my suspicion...)

Thanks for confirming the bug :-)

My guess is that if you create the alarm as the same user as what the ceilometer service is using it will work.

[Nejc Saje (nejc-saje)]

Nope, the insufficient data is due to there not being any cpu_util samples in the database yet, because I didn't run any instances.
The point is that the ceilometer user in the tenant `service` does in fact see the alarm.

[Sam Morrison (sorrison)]

Ah, thanks for clearing that up. When I do it I don't get any alarms returned by the ceilometer user. This is in our lab running Icehouse. It also doesn't work in our production environment running Havana

Example of command below is done against icehouse:

See http://paste.openstack.org/show/86660/

[Nejc Saje (nejc-saje)]

There must be something wrong with your keystone configuration then, but I can't help you much about that :-) If you tweak your keystone configuration in a way that the last query from your paste will work, then the alarm evaluation will work as well.

[Sam Morrison (sorrison)]

I'm not sure how keystone could affect this?

It has the Admin role which is the only thing I can see in the policy.json of ceilometer that would affect any permissions.

I've looked through the code but I'm not that familiar with ceilometer and it is quite different to nova/cinder etc. in how it does it's API.

Will dive deeper and get some pdb sessions going

[Nejc Saje (nejc-saje)]

Can you run `keystone user-role-list --user ceilometer --tenant service` and confirm that the role is actually `admin`, not `Admin` or `Administrator`?

Looking at the code, there are two possible paths when GET v2/alarms happens.

1. by regular user: db lookup is limited to the user's project_id
2. by admin: db lookup isn't limited

So please do some pdb sessions, possibly break at ceilometer/api/controllers/v2.py:307.

[Sam Morrison (sorrison)]

Our admin role is called "Admin" roles are case insensitive so this shouldn't matter?
I renamed it to admin to check and doesn't make a difference. (had to hack the DB in keystone)

When I debug this at the below line I always get an auth_project = <tenant id of the service tenant> (on_behalf_of is always None too)

auth_project = _get_auth_project(on_behalf_of)

I can't see the bit in the code where if the request has admin role it doesn't care what the project_id is and just returns all the alarms.
For instance if I do any other command as an ceilometer I don't get any results as the ceilometer user doesn't have any resources.

I also came across the following comment
    # when an alarm is created by an admin on behalf of another tenant
    # we must ensure for:

Does this mean the alarm is actually meant to be owned by service tenant and not the users tenant? Maybe the issue is that the alarms aren't being created correctly?

The alarm as stored in mongo is: (the project and user ID mentioned are my user ID)

> db.alarm.findOne()
{
 "_id" : ObjectId("53c62215499af2dbd3cf2b4d"),
 "alarm_actions" : [ ],
 "alarm_id" : "75a66cbe-6817-422a-80b8-d826fc53c0f9",
 "description" : "Alarm when cpu_util is eq a avg of 60.0 over 60 seconds",
 "enabled" : true,
 "insufficient_data_actions" : [ ],
 "name" : "test",
 "ok_actions" : [ ],
 "project_id" : "0bdf024c921848c4b74d9e69af9edf08",
 "repeat_actions" : false,
 "rule" : {
  "meter_name" : "cpu_util",
  "evaluation_periods" : 1,
  "period" : 60,
  "statistic" : "avg",
  "threshold" : 60,
  "query" : [
   {
    "field" : "project_id",
    "value" : "0bdf024c921848c4b74d9e69af9edf08",
    "op" : "eq"
   }
  ],
  "comparison_operator" : "eq",
  "exclude_outliers" : false
 },
 "state" : "insufficient data",
 "state_timestamp" : ISODate("2014-07-16T06:56:21.332Z"),
 "time_constraints" : [ ],
 "timestamp" : ISODate("2014-07-16T06:56:21.332Z"),
 "type" : "threshold",
 "user_id" : "d9646718471b46aeb5fd94c702336ca9"
}

[Nejc Saje (nejc-saje)]

If you get auth_project = <tenant id of the service tenant>, then the common openstack code (oslo) isn't marking ceilometer as admin. I don't know why that is, but it's not ceilometer's fault.

[Sam Morrison (sorrison)]

OK I have finally figured it out. It was to do with our policy.json file. It seems that the syntax accepted is different to the syntax of nova and other openstack projects for some reason? We had some other stuff in there that was copied from how nova does it to allow other roles to be admin.

[Sam Morrison (sorrison)]

Was an issue with wrong syntax in policy.json