object delete doesn't work with object count greater than page size

[John Dickinson]

When trying to delete a large number of objects, the client will detect this and use the bulk delete feature if it's enabled in the cluster. However, the bulk delete feature has a page size. If the number of objects to be deleted is larger than the page size, the operation will fail silenty. no error is returned.

This was seen "in the wild" when trying to delete approximately 150000 objects with a bulk delete page size set to the default of 10000.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/454849
Committed: https://git.openstack.org/cgit/openstack/python-swiftclient/commit/?id=0cc4d8af18157000d4d5d1205700d6494c5935ce
Submitter: Jenkins
Branch: master

commit 0cc4d8af18157000d4d5d1205700d6494c5935ce
Author: John Dickinson <email address hidden>
Date: Tue Apr 4 15:20:30 2017 -0700

    respect bulk delete page size and fix logic error

    Previously, using SwiftService to delete "many" objects would use
    bulk delete if available, but it would not respect the bulk delete
    page size. If the number of objects to delete exceeded the bulk delete
    page size, SwiftService would ignore the error and nothing would be
    deleted.

    This patch changes _should_bulk_delete() to be _bulk_delete_page_size();
    instead of returning a simple True/False, it returns the page size for
    the bulk deleter, or 1 if objects should be deleted one at a time.
    Delete SDK calls are then spread across multiple bulk DELETEs if the
    requested number of objects to delete exceeds the returned page size.

    Fixed the logic in _should_bulk_delete() so that if the object list
    is exactly 2x the thread count, it will not bulk delete. This is the
    natural conclusion following the logic that existed previously: if
    the delete request can be satisfied by every worker thread doing one
    or two tasks, don't bulk delete. But if it requires a worker thread
    to do three or more tasks, do a bulk delete instead. Previously, the
    logic would mean that if every worker thread did exactly two tasks, it
    would bulk delete. This patch changes a "<" to a "<=".

    Closes-Bug: 1679851
    Change-Id: I3c18f89bac1170dc62187114ef06dbe721afcc2e

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-swiftclient 3.4.0 release.