Comment 0 for bug 1616135

Revision history for this message
Javier Diaz Jr (javierdiazcharles) wrote :

Detailed bug description:
Basically, a customer requires ceilometer to be disabled. We have done that successfully, however, as you may know the rest of the services will continue to publish messages to the info queue for ceilometer to consume. Without ceilometer that queue will grow and grow. We have since set `notification_driver=` to blank on all services and restart all APIs/neutron-server/and nova-compute. The info queue is a lot smaller now after purging the queue, however, the queue still grows when we delete and create instances. It appears to be neutron, but I cant for the life of me figure out exactly how to stop it.

Here is the message payload:

{"_context_roles": ["Member"], "_context_request_id": "req-95b0ebc6-2a94-4b11-b763-8b790ae077c5", "event_type": "port.delete.start", "timestamp": "2016-08-22 18:39:33.145863", "_context_tenant_id": "2297e1dd886e4e4e973985299bdce5e9", "_unique_id": "afca5549c40a42baa0f3a9b5fb1d96df", "_context_tenant_name": "S1 Win2k8 Docvert - Development", "_context_user": "3d8a88bd06444dcb87a44907b1ffe604", "_context_user_id": "3d8a88bd06444dcb87a44907b1ffe604", "payload": {"port_id": "1953f662-9ccf-42e7-aec1-9a4f856ef5f7"}, "_context_project_name": "S1 Win2k8 Docvert - Development", "_context_read_deleted": "no", "_context_tenant": "2297e1dd886e4e4e973985299bdce5e9", "priority": "INFO", "_context_is_admin": false, "_context_project_id": "2297e1dd886e4e4e973985299bdce5e9", "_context_timestamp": "2016-08-22 18:39:33.142453", "_context_user_name": "0171070", "publisher_id": "network.node-10.int.thomsonreuters.com", "message_id": "ad0329e0-eb56-4b36-b0e1-480a90ce403a"}

[root@CONTROLLER-1 ~]# find /etc/ -name \*.conf -exec grep -HiR '^notification_driver' {} \;
/etc/cinder/cinder-amers1a.conf:notification_driver =
/etc/cinder/cinder.conf:notification_driver =
/etc/cinder/cinder-amers1b.conf:notification_driver =
/etc/nova/nova.conf:notification_driver=
/etc/glance/glance-api.conf:notification_driver =
/etc/heat/heat.conf:notification_driver =
/etc/neutron/neutron.conf:notification_driver =

[root@COMPUTE-1 ~]# find /etc/ -name \*.conf -exec grep -HiR '^notification_driver' {} \;
/etc/nova/nova.conf:notification_driver=
/etc/nova/nova.conf:notification_driver=
/etc/nova/nova.conf:notification_driver=

Steps to reproduce:
1. Stop all ceilometer related services on controllers and compute nodes.
2. Find all conf files with notification_driver variable (find /etc/ -name \*.conf -exec grep -HiR '^notification_driver' {} \;)
3. Set notification_driver variable to blank or noop.
4. Restart all corresponding APIs, neutron-server, nova-compute.
5. Purge notifications.info queue to clear queue.
6. Create and delete instance.
7. notifiactions.info queue grows again.

Expected results:
Queue should no longer be published to.
Actual result:
Neutron appears to be publishing messages from port create and port delete (see above payload)
Reproducibility:
100%
Workaround:
Cron job to purge queue manually.
Impact:
If no cron job is configured the queue will grow to large and rabbit may experience issues.
Description of the environment:
- Operation system: Ubuntu 14.04
- Versions of components: MOS 7.0 MU4