ignore_projects does not inhibit gnocchi_swift positive feedback completely

[Paulo Matias]

We still cannot measure the severity of the feedback loop because https://bugs.launchpad.net/openstack-ansible/+bug/1624788 is unresolved, but it seems that setting ignore_projects for [filter:ceilometer] in Swift's proxy-server.conf is not enough to completely inhibit the gnocchi_swift positive feedback loop.

By inspecting the notifications.info queue inside my /swift rabbit vhost, I've found several messages related to the gnocchi_swift project:

{"oslo.message": "{\"priority\": \"INFO\", \"_unique_id\": \"7566e4797cfc44a690fa45b2b3de95cf\", \"event_type\": \"objectstore.http.request\", \"timestamp\": \"2016-09-18 00:25:19.718389\", \"publisher_id\": \"ceilometermiddleware\", \"payload\": {\"typeURI\": \"http://schemas.dmtf.org/cloud/audit/1.0/event\", \"initiator\": {\"typeURI\": \"service/security/account/user\", \"project_id\": \"95e0cb483b3548bc83e1526cd54e4c81\", \"id\": \"b9ee60791abf424f9fceb8e66a883220\"}, \"target\": {\"action\": \"head\", \"typeURI\": \"service/storage/object\", \"id\": \"d5b4b33531764a379c1af3fa68fe533a\", \"metadata\": {\"path\": \"v1/AUTH_d5b4b33531764a379c1af3fa68fe533a\", \"version\": \"v1\", \"container\": null, \"object\": null}}, \"observer\": {\"id\": \"target\"}, \"eventType\": \"activity\", \"eventTime\": \"2016-09-18T00:25:19.717517\", \"action\": \"read\", \"outcome\": \"success\", \"id\": \"3c654292-c182-5591-83be-843edb3a4887\"}, \"message_id\": \"53a5e241-a8c0-4993-86ef-7dcae22f3d3b\"}", "oslo.version": "2.0"}

{"oslo.message": "{\"priority\": \"INFO\", \"_unique_id\": \"2a601a4589db4438b6ad44d7565dae62\", \"event_type\": \"objectstore.http.request\", \"timestamp\": \"2016-09-18 00:25:19.798204\", \"publisher_id\": \"ceilometermiddleware\", \"payload\": {\"typeURI\": \"http://schemas.dmtf.org/cloud/audit/1.0/event\", \"eventTime\": \"2016-09-18T00:25:19.797177\", \"target\": {\"action\": \"get\", \"typeURI\": \"service/storage/object\", \"id\": \"d5b4b33531764a379c1af3fa68fe533a\", \"metadata\": {\"path\": \"v1/AUTH_d5b4b33531764a379c1af3fa68fe533a\", \"version\": \"v1\", \"container\": null, \"object\": null}}, \"observer\": {\"id\": \"target\"}, \"eventType\": \"activity\", \"measurements\": [{\"metric\": {\"metricId\": \"9cbc46c8-d62b-5058-b337-faf84b1c381e\", \"name\": \"storage.objects.outgoing.bytes\", \"unit\": \"B\"}, \"result\": 35295}], \"initiator\": {\"typeURI\": \"service/security/account/user\", \"project_id\": \"95e0cb483b3548bc83e1526cd54e4c81\", \"id\": \"b9ee60791abf424f9fceb8e66a883220\"}, \"action\": \"read\", \"outcome\": \"success\", \"id\": \"1f3c1ec5-faac-5ab8-9bf0-add8fcb8b66e\"}, \"message_id\": \"c5bdbd46-d0fc-495a-a970-560bf16fee9e\"}", "oslo.version": "2.0"}

I'm getting an incoming message rate of 1 to 4 messages per second in this queue. Please note that currently I'm only using Swift as the Gnocchi storage backend.

The project IDs in my installation are the following:

d5b4b33531764a379c1af3fa68fe533a - gnocchi_swift
95e0cb483b3548bc83e1526cd54e4c81 - service
97084add947541f7b800d1a2d90e7937 - admin

[Paulo Matias (paulo-matias)]

By patching the gnocchi dispatcher in the ceilometer-collector container with a logging statement (dumping the sample when _is_gnocchi_activity returns True), we confirm that at least the two kind of messages I have posted in the original bug report can be filtered at the dispatcher level.

For Ocata, it would be nice if we could get upstream to provide a complete filter at the swift ceilometermiddleware level.

By now, let's fix by filtering both at the middleware and at the dispatcher, as discussed earlier with stevelle in the IRC channel.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/372863

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/372863
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_ceilometer/commit/?id=e0c1459667da5df651b9b72998788128f04e144c
Submitter: Jenkins
Branch: master

commit e0c1459667da5df651b9b72998788128f04e144c
Author: Paulo Matias <email address hidden>
Date: Tue Sep 20 00:28:24 2016 -0300

    Filter Gnocchi Swift activity in the Ceilometer dispatcher

    Currently, ignoring Gnocchi activity in the Swift ceilometermiddleware
    is not enough to avoid all Swift notifications generated by Gnocchi's
    own usage of Swift. Thefore, we also filter this activity at the
    dispatcher level.

    Change-Id: I3e2804561143617d0345c2a0cd5a462b841e73a3
    Closes-Bug: #1624900