Detailed bug description:
Basically, a customer requires ceilometer to be disabled. We have done that successfully, however, as you may know the rest of the services will continue to publish messages to the info queue for ceilometer to consume. Without ceilometer that queue will grow and grow. We have since set `notification_driver=` to blank on all services and restart all APIs/neutron-server/and nova-compute. The info queue is a lot smaller now after purging the queue, however, the queue still grows when we delete and create instances. It appears to be neutron, but I cant for the life of me figure out exactly how to stop it.
Here is the message payload:
{"_context_roles": ["Member"], "_context_request_id": "req-95b0ebc6-2a94-4b11-b763-8b790ae077c5", "event_type": "port.delete.start", "timestamp": "2016-08-22 18:39:33.145863", "_context_tenant_id": "2297e1dd886e4e4e973985299bdce5e9", "_unique_id": "afca5549c40a42baa0f3a9b5fb1d96df", "_context_tenant_name": "Win2k8", "_context_user": "3d8a88bd06444dcb87a44907b1ffe604", "_context_user_id": "3d8a88bd06444dcb87a44907b1ffe604", "payload": {"port_id": "1953f662-9ccf-42e7-aec1-9a4f856ef5f7"}, "_context_project_name": "Win2k8", "_context_read_deleted": "no", "_context_tenant": "2297e1dd886e4e4e973985299bdce5e9", "priority": "INFO", "_context_is_admin": false, "_context_project_id": "2297e1dd886e4e4e973985299bdce5e9", "_context_timestamp": "2016-08-22 18:39:33.142453", "_context_user_name": "0171070", "publisher_id": "<email address hidden>(edited for privacy reasons)", "message_id": "ad0329e0-eb56-4b36-b0e1-480a90ce403a"}
[root@CONTROLLER-1 ~]# find /etc/ -name \*.conf -exec grep -HiR '^notification_driver' {} \;
/etc/cinder/cinder-amers1a.conf:notification_driver =
/etc/cinder/cinder.conf:notification_driver =
/etc/cinder/cinder-amers1b.conf:notification_driver =
/etc/nova/nova.conf:notification_driver=
/etc/glance/glance-api.conf:notification_driver =
/etc/heat/heat.conf:notification_driver =
/etc/neutron/neutron.conf:notification_driver =
[root@COMPUTE-1 ~]# find /etc/ -name \*.conf -exec grep -HiR '^notification_driver' {} \;
/etc/nova/nova.conf:notification_driver=
/etc/nova/nova.conf:notification_driver=
/etc/nova/nova.conf:notification_driver=
Steps to reproduce:
1. Stop all ceilometer related services on controllers and compute nodes.
2. Find all conf files with notification_driver variable (find /etc/ -name \*.conf -exec grep -HiR '^notification_driver' {} \;)
3. Set notification_driver variable to blank or noop.
4. Restart all corresponding APIs, neutron-server, nova-compute.
5. Purge notifications.info queue to clear queue.
6. Create and delete instance.
7. notifiactions.info queue grows again.
Expected results:
Queue should no longer be published to.
Actual result:
Neutron appears to be publishing messages from port create and port delete (see above payload)
Reproducibility:
100%
Workaround:
Cron job to purge queue manually.
Impact:
If no cron job is configured the queue will grow to large and rabbit may experience issues.
Description of the environment:
- Operation system: Ubuntu 14.04
- Versions of components: MOS 7.0 MU4, MOS 5.1 ( I assume other MOS versions are affected too)
As I can see here:
https:/
in the heat a notification driver setup as constant string (notice that is custom notification driver - heat.openstack.*) without any dependency on the ceilometer.
So, heat always sends notifications and we need to change this behavior:
if ceilometer is enabled - the notification_driver value must be heat.openstack.
if not - notification_driver must be empty string (means notifications are disabled)