No way to disable notifications from services

[Ben Nemec]

We recently added an environment to disable telemetry services in the overcloud[1], but it turns out this is actually dangerous in a real-world deployment. Without anything consuming notifications, the rabbit queues fill up and eventually start causing failures due to an inability to deal with that many pending messages.

We need to be able to set notification_driver for the other services to "noop", but currently this is not exposed via t-h-t. Ideally these would be wired in as template parameters so they could be set in the disable-telemetry environment file.

I believe this is already exposed via the puppet modules[2], so it should be possible to address mostly within t-h-t.

1: https://github.com/openstack/tripleo-heat-templates/blob/master/environments/disable-telemetry.yaml
2: https://github.com/openstack/puppet-cinder/blob/master/manifests/ceilometer.pp#L28

[Bogdan Dobrelya (bogdando)]

Note, the easier fix would be setting expiration for notification queues globally

[Emilien Macchi (emilienm)]

Are we really sure that Telemetry are the only consumers of RPC notifications? I'm thinking at services like Cinder, Nova, Neutron who also might consume some notifications.

[Ben Nemec (bnemec)]

I'm sure that if you disable the telemetry services then the notification queues fill up. I'm also sure that we disabled notifications in rh1 for all the services and everything is working fine.

I'm also fine with setting an expiration for the notification queues, although I don't know off the top of my head how to do that. Even if we do, I feel like we should allow disabling of notifications completely so we're not wasting CPU for a service that is already frequently a bottleneck in large deployments.

[Emilien Macchi (emilienm)]

There are no currently open reviews on this bug, changing the status back to the previous state and unassigning. If there are active reviews related to this bug, please include links in comments.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/481726

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/481726
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=d5145167cb13170cb2f8b928b3c920648e7ecaf0
Submitter: Jenkins
Branch: master

commit d5145167cb13170cb2f8b928b3c920648e7ecaf0
Author: Emilien Macchi <email address hidden>
Date: Fri Jul 7 11:35:28 2017 -0700

    Allow to set Notification Driver to 'noop'

    This patch does 2 things:
    * Configure messagingv2 as default driver for Oslo Notifications sent on
      RPC.
    * Allow users to choose between messagingv2 (default) and noop when we
      want to disable notifications (for example, when Telemetry is disabled).
    * Deprecate KeystoneNotificationDriver in favor of NotificationDriver.

    Change-Id: Ia547d7f4bfb51e7c45246b097b48fd86da231bd3
    Related-Bug: #1701357

[Ben Nemec (bnemec)]

Marking fixed since https://review.openstack.org/481726 does in fact make it possible to disable notifications.