object replicator collect_jobs loops over all replicas before each run

[Caleb Tennis]

The object replicator collect_jobs loops over all devices, then does an os.listdir on each one. Imagine a node where replication is slow because there is a lot of IO. You can get in a cycle where there are so many partitions to look through, it takes forever for the replicator to start. I found that to load a objects directory with 50,000 partitions in it takes about 6-7 minutes, and that's just a single device.

I don't know what the right fix here is, maybe something more incremental. In addition, specifying -d or -p to limit the scope of the replicator doesn't bypass this step - it still loads all partitions first and THEN only works on the subset specified.

[clayg (clay-gerrard)]

maybe find (or write with ctypes) a python wrapper for getdents?

https://www.olark.com/developers-corner/you-can-list-a-directory-with-8-million-files-but-not-with-ls?old_blog_path=%2Fspw%2F2011%2F08%2Fyou-can-list-a-directory-with-8-million-files-but-not-with-ls%2F

[Caleb Tennis (ctennis)]

I think https://review.openstack.org/#/c/149384/ handles fixing this, so once that's merged this can be closed.

[Samuel Merritt (torgomatic)]

that thing was merged