[AODH] AODH cannot retrieve statistics for alarms

[Vitaly Gusev]

Information about ISO is here http://paste.openstack.org/show/489303/

Steps to reproduce:
1. Deploy an environment with 3 controllers+mongo, 1 compute, Ceilometer and AODH enabled, Cinder LVM
2. Create an alarm
aodh alarm create -t threshold --name test_alarm -m test --period 600 --statistic avg --comparison-operator lt --threshold 10
3. Create a sample
ceilometer sample-create -m test -r 1234567890 --meter-type delta --meter-unit test-unit --sample-volume 5
3. Check statistics for this metric
+--------+---------------------+----------------------------+-----+-----+-----+-----+-------+----------+----------------------------+----------------------------+
| Period | Period Start | Period End | Max | Min | Avg | Sum | Count | Duration | Duration Start | Duration End |
+--------+---------------------+----------------------------+-----+-----+-----+-----+-------+----------+----------------------------+----------------------------+
| 0 | 2016-02-26T08:36:04 | 2016-03-04T08:36:30.700000 | 5.0 | 5.0 | 5.0 | 5.0 | 1 | 0.0 | 2016-03-04T08:36:30.700000 | 2016-03-04T08:36:30.700000 |
+--------+---------------------+----------------------------+-----+-----+-----+-----+-------+----------+----------------------------+----------------------------+

Expected result:
Alarm state was changed to 'alarm'

Actual result:
Alarm state is 'insufficient data':
root@node-1:~# aodh alarm list -t threshold
+--------------------------------------+------------+-------------------+----------+---------+----------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| alarm_id | name | state | severity | enabled | repeat_actions | time_constraints | threshold_rule |
+--------------------------------------+------------+-------------------+----------+---------+----------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 9a3543bb-52d2-4ada-8c32-cea3eca24bed | test_alarm | insufficient data | low | True | False | [] | {u'meter_name': u'test', u'evaluation_periods': 1, u'period': 600, u'statistic': u'avg', u'threshold': 10.0, u'query': [], u'comparison_operator': u'lt', u'exclude_outliers': False} |
+--------------------------------------+------------+-------------------+----------+---------+----------------+------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
In /var/log/aodh/aodh-evaluator.log we can see the following error http://paste.openstack.org/show/489304/

[Ivan Berezovskiy (iberezovskiy)]

Please, use http://paste.openstack.org/ instead of internal and update a description then

[Vitaly Gusev (vgusev)]

@Ivan,

done

[Vitaly Gusev (vgusev)]

The root cause is in incorrect auth_url for aodh in /etc/aodh/aodh.conf. By default it's
os_auth_url=http://192.168.0.2:5000/

We changed it to
os_auth_url=http://192.168.0.2:5000/v2.0

and after this aodh works fine

[Ivan Berezovskiy (iberezovskiy)]

@Vitaly, does it work with v3? Also I'd prefer to use versionless urls, may be we need to change another parameter for that.

[Vitaly Gusev (vgusev)]

It doesn't work with v3. In logs I can see also this trace:
2016-03-16 15:15:53.847 7572 INFO aodh.evaluator [-] initiating evaluation cycle on 1 alarms
2016-03-16 15:15:53.847 7572 DEBUG aodh.evaluator [-] evaluating alarm 834b34fb-da67-462a-a05f-1fc2c7eb816b _evaluate_alarm /usr/lib/python2.7/dist-packages/aodh/evaluator/__init__.py:221
2016-03-16 15:15:53.848 7572 DEBUG aodh.evaluator.threshold [-] query stats from 2016-03-16 14:55:53.848041 to 2016-03-16 15:15:53.848041 _bound_duration /usr/lib/python2.7/dist-packages/aodh/evaluator/threshold.py:79
2016-03-16 15:15:53.848 7572 DEBUG aodh.evaluator.threshold [-] stats query [{'field': 'timestamp', 'value': '2016-03-16T15:15:53.848041', 'op': 'le'}, {'field': 'timestamp', 'value': '2016-03-16T14:55:53.848041', 'op': 'ge'}] _statistics /usr/lib/python2.7/dist-packages/aodh/evaluator/threshold.py:114
2016-03-16 15:15:53.851 7572 DEBUG keystoneauth.identity.v2 [-] Making authentication request to http://192.168.0.2:5000/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneauth1/identity/v2.py:63
2016-03-16 15:15:53.860 7572 DEBUG keystoneauth.session [-] Request returned failure status: 404 request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:468
2016-03-16 15:15:53.861 7572 ERROR aodh.evaluator.threshold [-] alarm stats retrieval failed

Looks like aodh uses keystone v2 for authentication.

[Artem Minasyan (aminasyan)]

Verified on ISO: http://paste.openstack.org/show/495014/