Keystone overwhelms Ceilometer with Identity Events

[Adam Young]

Description of problem:
When configuring OpenStack from OSP director, keystone is enabled to produce ceilometer events. These events spam Ceilometer, and any CloudForms instance managing the Overcloud with "identity.authenticate" events. These events cause unneeded processing on CloudForms and unneeded data storage in ceilometer as they have no practical use

Version-Release number of selected component (if applicable):
openstack-keystone-8.0.1-1.el7ost.noarch
python-tripleoclient-0.3.4-6.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy Overcloud with ceilometer Events
parameter_defaults:
  CeilometerStoreEvents: true
2. login to controller
3. sudo openstack-config --get /etc/keystone/keystone.conf DEFAULT notification driver
messagin

Actual results:
literally nearly 100,000 identity events get created per day. Here is a sample of about 22 hours from an unused Cloud.

 grep /ManageIQ/System/Event/EmsEvent/OPENSTACK evm.log | awk '{ print $10 }' | sort | uniq -c
...
 86317 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.authenticate]
   473 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.created.role_assignment]
     2 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.domain.created]
    54 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.endpoint.created]
    23 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.OS-TRUST:trust.created]
    21 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.OS-TRUST:trust.deleted]
    29 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.project.created]
    21 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.project.deleted]
     2 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.region.created]
   473 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.role_assignment.created]
     8 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.role.created]
    18 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.service.created]
   467 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.user.created]
   404 [/ManageIQ/System/Event/EmsEvent/OPENSTACK/identity.user.deleted]
...

NOTE: the 86,317 identity.authenticate events produced by the Overcloud

Expected results:

identity events need not be produced by default for CloudForms to do its thing. these are essentially SPAM events that use valuable resources

Additional info:

Suggest setting notification_driver to either log or noop in /etc/keystone/keystone.conf

[Adam Young (ayoung)]

Note that you can reduce the number events produced by Keystone using a configuration option. According to the Keystone documentation,

http://git.openstack.org/cgit/openstack/keystone/tree/etc/keystone.conf.sample?h=9.0.2#n105

[DEFAULT]
notification_opt_out=identity.authenticate.success

This would prevent keystone from sending notifications out on successful authentications.

Successful authentications and validations of user tokens are probably redundant: the creation alone does not mean anything, only the use on the remote system. One or the other should be removed.

[Steve Martinelli (stevemar)]

We could opt-out of successful auths by default, set ``notification_opt_out`` to ``identity.authenticate.success``.

It's a list option, so we can write out the option many times

[Steve Martinelli (stevemar)]

Setting it to low since there's a workaround, and i'm not sure what we can do other than setting the success events to be ignored by default.

[gordon chung (chungg)]

does this need anything from ceilometer pov?

[gordon chung (chungg)]

http://eavesdrop.openstack.org/irclogs/%23openstack-telemetry/%23openstack-telemetry.2016-11-03.log.html