Telemetry of S3 traffic

[Dmitry]

I use Openstack Swift (train) with Ceilometer + Gnocchi as telemetry service for billing.
json messages from swift proxy server to rabbitmq have no "project_id" and gnocchi creates new resource (named as filename instead of project_id) for every new file uploaded by S3. It means it creates directory structure on its storage and 9 new rows in mysql database.
Amount of space used by gnocchi highly increasing and there is no way to use these measures for billing, as they not attached to one project.
Message for swift api traffic:
{"message_id": "d2f54d0d-1ebe-48a6-9cfe-9aae7a01b407", "publisher_id": "ceilometermiddleware", "event_type": "objectstore.http.request", "priority": "INFO", "payload": {"typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event", "eventType": "activity", "id": "ef4a85f0-aed7-5909-8547-165139082c80", "eventTime": "2020-11-05T12:44:00.549474", "action": "update", "outcome": "success", "observer": {"id": "target"}, "initiator": {"id": "edf7a98ce8b54bb9844dbd8c41c3ba6a", "typeURI": "service/security/account/user", "project_id": "490a32b56ce94d71af7e3621b2034141"}, "target": {"id": "490a32b56ce94d71af7e3621b2034141", "typeURI": "service/storage/object", "metadata": {"path": "v1/AUTH_490a32b56ce94d71af7e3621b2034141/cont01/receipts/16816830/5fa3f38e0691a5fa3f38e06958.pdf", "version": "v1", "container": "cont01", "object": "receipts/16816830/5fa3f38e0691a5fa3f38e06958.pdf"}, "action": "put"}, "measurements": [{"result": 932311, "metric": {"metricId": "a7a13df1-bca6-5e2a-8a55-03949534a60c", "unit": "B", "name": "storage.objects.incoming.bytes"}}]}, "timestamp": "2020-11-05 12:44:00.550097", "_unique_id": "77e88300ba504b6ea178f236e2b119b5"}

Message for S3api traffic:
{"message_id": "13bf7e10-4204-4e03-ae85-3ba103276fa8", "publisher_id": "ceilometermiddleware", "event_type": "objectstore.http.request", "priority": "INFO", "payload": {"typeURI": "http://schemas.dmtf.org/cloud/audit/1.0/event", "eventType": "activity", "id": "0a07d96a-0b7c-53d5-b499-8c912b6fdfe2", "eventTime": "2020-11-05T13:07:00.613230", "action": "read", "outcome": "success", "observer": {"id": "target"}, "initiator": {"id": "354ca9fc-446a-5b6e-a5c6-1d278fa4d6c5", "typeURI": "service/security/account/user", "project_id": null}, "target": {"id": "cont2/greatoldones.jpg", "typeURI": "service/storage/object", "metadata": {"path": "cont2/greatoldones.jpg", "version": "cont2", "container": null, "object": null}, "action": "get"}, "measurements": [{"result": 216, "metric": {"metricId": "ec8f9a04-2f38-5522-bba4-654292a86fef", "unit": "B", "name": "storage.objects.outgoing.bytes"}}]}, "timestamp": "2020-11-05 13:07:00.614358", "_unique_id": "c096567ca9924ba89fa27eb818ecb390"}

[Tim Burke (1-tim-z)]

Makes me think of https://review.opendev.org/#/c/718086/ and https://review.opendev.org/#/c/718085/ -- do you think those two are likely to address the issue? How soon do you expect to be able to upgrade to Ussuri (which includes these)? I can backport the Swift change to Train (or earlier), but I don't have much control over Ceilometer...

[Dmitry (kozlovdmtry)]

I have a lab with Openstack Ussuri and it have the same issue.
Versions of soft:
openstack-swift-proxy-2.25.0-1
openstack-ceilometer-*-14.0.0-1
When I upload a file to container via S3Browser gnocchi creates new resource for each file.
Messages, that ceilometer sends to gnocchi api are like following:
{"cont01_puttygen.exe":{"storage.objects.incoming.bytes":{"measures":[{"timestamp":"2020-11-19T08:01:07.587579+00:00","value":407104}],"archive_policy_name":"ceilometer-lo
w","unit":"B"}}}

[Dmitry (kozlovdmtry)]

On every uploaded by S3Browser file Swift proxy logs warning:
STDERR: /usr/lib/python3.6/site-packages/pycadf/identifier.py:72: UserWarning: Invalid uuid: cont01/puttygen.exe. To ensure interoperability, identifiers should be a valid uuid.#012 'identifiers should be a valid uuid.' % (value))) (txn: txe20a974e15104fa2a614e-005fb62643) (client_ip:10.64.65.143)