notifications.error should be consumed or not be sent

[ZhiQiang Fan]

In a havana environment, I have noticed that there will be a message sent to notifications.error when nova did live-migrate but fail. After tracking the code, I think in stable/icehouse branch, the same issue exists.

In Ceilometer stable/icehouse, we just comsumes messages send to notifications.info topic, see:
https://github.com/openstack/ceilometer/blob/stable/icehouse/ceilometer/compute/notifications/__init__.py#L43

In Nova stable/icehouse, if instance migrate fails, or something other important vm related action fails, a message will be sent to notifications.error, see:
https://github.com/openstack/nova/blob/stable/icehouse/nova/conductor/manager.py#L773
https://github.com/openstack/nova/blob/stable/icehouse/nova/scheduler/utils.py#L101
https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/notify/notifier.py#L256

The problem is that, Ceilometer has connected to many OpenStack services, if Nova has this issue, other services may too.

I don't know why service sends a notification which will not be consumed, this only makes operator confuse and (may be low but true) adds AMQP's burden.

So my opinion would be:
1) modify Ceilometer's code, let's consume the notifications.{warn, error, critical} besides notications.info, and drop them directly
2) notify other project's contributor, let them stop to send such notifications

[wingwj (wingwj)]

I agree with the first one.
But I don't think dropping them is a good idea. If we dealt with the notifications.info messages, no reason to ignore higher level ones. Maybe we can discuss the topic in ML.

Thanks.

[ZhiQiang Fan (aji-zqfan)]

according to:
https://github.com/openstack/ceilometer/blob/master/ceilometer/plugin.py#L43
and
https://github.com/openstack/oslo.messaging/blob/master/oslo/messaging/notify/listener.py#L68

I think master branch has same problem too

[gordon chung (chungg)]

welcome back ZhiQiang Fan!... i'm not sure it's a bug or not... in most environments, ceilometer is not the only consumer of notifications (ie. could be for 3rd party purposes, to let a project/service other than ceilometer know of error)... i'm not sure which case its for... we may need to check with nova but i would think it's a useful event for ceilometer to consume

[gordon chung (chungg)]

hi zqfan, are you planning on working on this item? if not, i can look at it as i'd like to have it in for kilo

[ZhiQiang Fan (aji-zqfan)]

sure, go ahead

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/153362

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/153362
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=f7ed2c2a16e8dded6bd2258ea0a5e5954a13bada
Submitter: Jenkins
Branch: master

commit f7ed2c2a16e8dded6bd2258ea0a5e5954a13bada
Author: gordon chung <email address hidden>
Date: Thu Feb 5 15:27:38 2015 -0500

    start recording error notifications

    projects (specifically nova) send notifications on error topic when
    an error occurs. we should capture this in events.

    Change-Id: Ic42cbce948b8b409f83934146407b2480602921d
    Closes-Bug: #1364708

[Flávio Ramalho (flaviosr)]

Does this really got fixed?

Running Kilo 2015.1.0 and getting:

# rabbitmqctl list_queues | grep notifications
notifications.error 49
notifications.info 0

Also, there is no consumer for the notifications.error queue.

[gordon chung (chungg)]

do you have store_events enabled?

[Flávio Ramalho (flaviosr)]

I do.

Here is my ceilometer.conf

$ cat /etc/ceilometer/ceilometer.conf | egrep -v "^\s*(#|$)"
[DEFAULT]
auth_strategy = keystone
verbose = true
store_events = true
rpc_backend = rabbit
[database]
connection = mongodb://-:-@ceilometer:27017/ceilometer
[keystone_authtoken]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_user = ceilometer
admin_password = -
admin_tenant_name = service
[service_credentials]
os_auth_url = http://controller:5000/v2.0
os_username = ceilometer
os_tenant_name = service
os_password = -
os_endpoint_type = internalURL
os_region_name = lsd-0
[publisher]
telemetry_secret = -
[matchmaker_redis]
[matchmaker_ring]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
amqp_durable_queues = true
rabbit_hosts = 192.168.0.6:5672,192.168.0.7:5672
rabbit_userid = openstack
rabbit_password = -
rabbit_ha_queues = true

[gordon chung (chungg)]

hi Flavio,

store_events should actually be under [notification] section rather than [DEFAULT]. it could also be under [collector] section but this is deprecated.

[Flávio Ramalho (flaviosr)]

Hi Gordon,

After moving store_events to [notifications], ceilometer-agent-notification started throwing:

ceilometer.openstack.common.threadgroup TypeError: coercing to Unicode: need string or buffer, NoneType found

Putting event_pipeline.yaml, from https://github.com/openstack/ceilometer/blob/stable/kilo/etc/ceilometer/event_pipeline.yaml, in /etc/ceilometer (fixed in https://review.openstack.org/#/c/152525/) solved and everything is working perfectly.

Thanks Gordon.

[gordon chung (chungg)]

no problem.

i should mention, the error messages are only visible as events, not meters (as there are no (known) measurements in error notifications

[George (lmihaiescu)]

Hi Gordon,

Is this fix going to be backported to Juno?

Thank you,
George

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/345390

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/345390
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=a3eb0fafadcdcbb8f48f3db13ffc103717bae99b
Submitter: Jenkins
Branch: master

commit a3eb0fafadcdcbb8f48f3db13ffc103717bae99b
Author: Mehdi Abaakouk <email address hidden>
Date: Thu Jul 21 14:33:10 2016 +0200

    consumes error notif. when event are disabled

    When we fixes #1364708, this was worked only when store_event=True.

    This change fixes the issue when store_event=False.

    Change-Id: I6748397718be03e3f93ae2ccaa99642decdd9745
    Related-bug: #1364708

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/349836

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/349838

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/349838
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=09afee9a8fd665097f912defc844bb80e54fc23d
Submitter: Jenkins
Branch: stable/liberty

commit 09afee9a8fd665097f912defc844bb80e54fc23d
Author: Mehdi Abaakouk <email address hidden>
Date: Thu Jul 21 14:33:10 2016 +0200

    consumes error notif. when event are disabled

    When we fixes #1364708, this was worked only when store_event=True.

    This change fixes the issue when store_event=False.

    Change-Id: I6748397718be03e3f93ae2ccaa99642decdd9745
    Related-bug: #1364708
    (cherry picked from commit a3eb0fafadcdcbb8f48f3db13ffc103717bae99b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/349836
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=54a30501a9406742427b731d4f7fb2469529720e
Submitter: Jenkins
Branch: stable/mitaka

commit 54a30501a9406742427b731d4f7fb2469529720e
Author: Mehdi Abaakouk <email address hidden>
Date: Thu Jul 21 14:33:10 2016 +0200

    consumes error notif. when event are disabled

    When we fixes #1364708, this was worked only when store_event=True.

    This change fixes the issue when store_event=False.

    Change-Id: I6748397718be03e3f93ae2ccaa99642decdd9745
    Related-bug: #1364708
    (cherry picked from commit a3eb0fafadcdcbb8f48f3db13ffc103717bae99b)