Container synchronization is limited to 10000

[Fabien Boucher]

Your object sync source code looks like it will quietly skip syncing anything more than the first 10,000 objects in a container: Where your code calls swiftclient’s get_container() (defined in swiftclient/client.py) to list all objects, it will only get the first 10,000 since the optional parameter full_listing defaults to false. Even if full_listing was set true, with millions of objects you could hit memory limitations.

[Fabien Boucher (fabien-boucher)]

A change is proposed here : https://review.openstack.org/#/c/73329/

[Jim Freeman (jim-freeman)]

Thanks Fabien, I looked at your change, the fix of setting 'full_listing = true' should solve this.

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/73329
Committed: https://git.openstack.org/cgit/stackforge/swiftsync/commit/?id=fa25f45e6ccc77c46d5cf1a98ab32cc3bd6a5c93
Submitter: Jenkins
Branch: master

commit fa25f45e6ccc77c46d5cf1a98ab32cc3bd6a5c93
Author: Fabien Boucher <email address hidden>
Date: Thu Feb 13 16:40:12 2014 +0100

    Set full_listing to True to avoid the default limit

    - Swiftclient cuts to 10000 by default the
    amount of objects listed with get_container. This
    patch set the full_listing to True.

    - This patch has no solutions to avoid the
    high memory consumption when container
    contains a huge amount of objects.

    Closes-Bug: #1279804

    Change-Id: I2d991baa14f9d4c6a827629c6b43fcc693e0262a