S3 driver uses up all available buckets

Bug #1671925 reported by Nicolas Vila on 2017-03-10
26
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Gnocchi
Fix Released
High
gordon chung

Bug Description

Using Gnocchi v3.1.1.dev31 deployed in Amazon AWS using S3 backend, the maximum limit (by default) of 100 buckets is immediately reached and, when this happens, both gnocchi-metricd can't process new measures (http://paste.openstack.org/show/602267/) while gnocchi API is also unable to work properly.

There are several strategies regarding the partition of objects in S3 (https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/), perhaps it would be convenient to have a config option that limits the amount of buckets created in S3, so that if an AWS account is already using S3 and has N available buckets, the user would be able to choose a performant value that doesn't exceed this 100-bucket limit.

Nicolas Vila (nvlan) wrote :

Output of 'gnocchi resource list' with all buckets used: http://paste.openstack.org/show/602277/

Julien Danjou (jdanjou) on 2017-03-11
Changed in gnocchi:
status: New → Confirmed
importance: Undecided → High
gordon chung (chungg) on 2017-03-20
Changed in gnocchi:
status: Confirmed → Triaged
Nicolas Vila (nvlan) wrote :

Hello,

Is there any update regarding the S3 driver? It's unusable as it is...

Thanks, regards.
Nicolas

gordon chung (chungg) wrote :

i might have something tomorrow. but that said, i don't think s3 is still a viable solution. ignoring cost/transactions, the latency of external storage is going to suck regardless.

Fix proposed to branch: master
Review: https://review.openstack.org/453302

Changed in gnocchi:
assignee: nobody → gordon chung (chungg)
status: Triaged → In Progress
Nicolas Vila (nvlan) wrote :

Hello Gordon,

After a week of testing, the patch seems to work as expected. Only two buckets were created (-aggregates and -measure) and gnocchi-metricd is reporting an average of about 2.5s to process measures:

2017-04-12 14:17:09.895 3697 INFO gnocchi.storage._carbonara [-] Computed new metric 1e969a9f-c662-4268-ae9a-314721886923 with 1 new measures in 2.52 seconds (14262 points/s, 16 measures/s)
2017-04-12 14:17:11.203 3715 INFO gnocchi.storage._carbonara [-] Computed new metric ae89933c-5c37-4acf-801e-e96a358f8e06 with 1 new measures in 2.72 seconds (13249 points/s, 15 measures/s)
2017-04-12 14:17:23.860 3726 INFO gnocchi.storage._carbonara [-] Computed new metric e8f67d11-1930-4f93-8575-5a6312d8f8ec with 1 new measures in 1.84 seconds (19586 points/s, 22 measures/s)
2017-04-12 14:17:52.454 3728 INFO gnocchi.storage._carbonara [-] Computed new metric d985a031-b2f1-49e1-8b66-da3eb3ee6f27 with 1 new measures in 1.88 seconds (19147 points/s, 22 measures/s)
2017-04-12 14:17:53.592 3747 INFO gnocchi.storage._carbonara [-] Computed new metric 2352549b-f442-421e-b461-83bfdc9038f8 with 1 new measures in 1.96 seconds (18323 points/s, 21 measures/s)
2017-04-12 14:17:53.631 3739 INFO gnocchi.storage._carbonara [-] Computed new metric d2281e85-96a6-4f38-938e-4f9f908dbe14 with 1 new measures in 1.90 seconds (18897 points/s, 22 measures/s)

Thanks, regards.

gordon chung (chungg) wrote :

nice. thanks for confirming!

that still seems very slow but i think it's probably just related to latency of external storage. i would leverage aggregation_workers_number if you're not already.

Reviewed: https://review.openstack.org/453302
Committed: https://git.openstack.org/cgit/openstack/gnocchi/commit/?id=0f210a7ba66b8fcfe1aa09e2db17a2c1528b5e64
Submitter: Jenkins
Branch: master

commit 0f210a7ba66b8fcfe1aa09e2db17a2c1528b5e64
Author: gord chung <email address hidden>
Date: Tue Apr 4 19:18:48 2017 +0000

    s3: use a single bucket for aggregate storage

    s3 has a bucket limit so we can't just create a bucket per metric.
    s3 has some partitioning logic to divide a bucket based on prefix[1]
    so let us use the metric as a prefix of the object key.

    [1] https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/

    Change-Id: If4a1ec3345d85a9937b4961b0ef61751b0a5e90d
    Closes-Bug: #1671925

Changed in gnocchi:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers