object replicator collect_jobs loops over all replicas before each run
Bug #1395879 reported by
Caleb Tennis
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Object Storage (swift) |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
The object replicator collect_jobs loops over all devices, then does an os.listdir on each one. Imagine a node where replication is slow because there is a lot of IO. You can get in a cycle where there are so many partitions to look through, it takes forever for the replicator to start. I found that to load a objects directory with 50,000 partitions in it takes about 6-7 minutes, and that's just a single device.
I don't know what the right fix here is, maybe something more incremental. In addition, specifying -d or -p to limit the scope of the replicator doesn't bypass this step - it still loads all partitions first and THEN only works on the subset specified.
To post a comment you must log in.
maybe find (or write with ctypes) a python wrapper for getdents?
https:/ /www.olark. com/developers- corner/ you-can- list-a- directory- with-8- million- files-but- not-with- ls?old_ blog_path= %2Fspw% 2F2011% 2F08%2Fyou- can-list- a-directory- with-8- million- files-but- not-with- ls%2F