container-sync needs to be more parallel
Bug #1068426 reported by
Faidon Liambotis
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
In Progress
|
Wishlist
|
Eran Rom |
Bug Description
The way container sync works right now, it syncs one object at a time, which is hugely inefficient in large containers. Moreover, running multiple sync processes from multiple container servers makes it actually *worse*, since they try to sync the very same files that their sibling just synced (since the state remains in the yet unrsynced sqlites).
It'd be nice if the sync process had some way to parallelize the problem and sync multiple files at the same time for efficiency reasons.
Changed in swift: | |
importance: | Undecided → Wishlist |
Changed in swift: | |
status: | New → Triaged |
Changed in swift: | |
assignee: | nobody → Eran Rom (eranr) |
To post a comment you must log in.
I'm confused about which container-sync processes are doing duplicate work. Are you saying
(a) when running multiple swift-container -sync processes on a single node, they operate on the containers in the same order and therefore do duplicate work,
(b) when running two swift-container -sync processes on two separate nodes, they duplicate each others' work, or
(c) all of the above?