sharding: container GETs to root container get slow

Bug #1781291 reported by John Dickinson on 2018-07-11
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)

Bug Description

During and object PUT to a sharded contianer, the proxy does a GET to the container server to get the shard ranges. This container GET starts to get really slow under high concurrency.

To reproduce, make a container with a lot of shards. Then start concurrently putting new objects into it. As concurrency increases, the container GET gets very slow.

There's a few ideas on fixing this (none of these are final)

Cache the shard ranges in the proxy. Since they are unbounded, memcache isn't likely a good place. Maybe some in-process memory can be used and have some sort of LRU on it. But then you've got cache invalidation issues...

Defer the child DB update until later. Only tell the object server to update the root, and let the updater handle the redirects. Also let the root DB enqueue child updates.

Instead of updating the root, send the updates to the root's handoffs (some number of them) so that they get sharded in place and the sharder handles getting the updates to the right place.

Don't do any of the deferring in the server and instead slow down the client (ie backpressure).

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers