Auto-extract feature of bulk.py can bypass quota settings

[Steve Mayer]

When using the archive auto-extract feature of the bulk.py module, it is possible to blow out the account and/or container quotas that might be configured.

Once the initial size of the archive to be uploaded passes the quota checks, nothing is done to ensure that the expanded size of the archive contents also meets quota restrictions. This allows for the opportunity of blowing the quota by quite a bit depending on the contents of the archive.

Using Swift 1.8.0 release.

[David Goetz (david-goetz)]

This is only true if the bulk middleware is after the quota middleware in the proxy pipeline. The bulk middleware should always be to the left of ratelimiting, auth, quota middleware, etc so that every subrequest is handled properly and restrictions are maintained.

[Steve Mayer (smayer69)]

Thanks for the information. I'll test this out. Is there any documentation on the placement within the pipeline that I missed?

Thanks!

Steve Mayer
<email address hidden>

On Jul 16, 2013, at 8:45, David Goetz <email address hidden> wrote:

> This is only true if the bulk middleware is after the quota middleware
> in the proxy pipeline. The bulk middleware should always be to the left
> of ratelimiting, auth, quota middleware, etc so that every subrequest is
> handled properly and restrictions are maintained.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1201844
>
> Title:
> Auto-extract feature of bulk.py can bypass quota settings
>
> Status in OpenStack Object Storage (Swift):
> New
>
> Bug description:
> When using the archive auto-extract feature of the bulk.py module, it
> is possible to blow out the account and/or container quotas that might
> be configured.
>
> Once the initial size of the archive to be uploaded passes the quota
> checks, nothing is done to ensure that the expanded size of the
> archive contents also meets quota restrictions. This allows for the
> opportunity of blowing the quota by quite a bit depending on the
> contents of the archive.
>
> Using Swift 1.8.0 release.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/swift/+bug/1201844/+subscriptions

[David Goetz (david-goetz)]

i think maybe only in etc/proxy-server.conf-sample in the bulk section but it only mentions auth and ratelimiting not quotas

[Steve Mayer (smayer69)]

Okay.

While moving the bulk entry to the left ( as you described) seems to work with account quotas, it doesn't seem to work with container quotas. ( or is the account admin user exempt from the container quota limitations?)

Thanks!

Steve Mayer
<email address hidden>

On Jul 16, 2013, at 10:14, David Goetz <email address hidden> wrote:

> i think maybe only in etc/proxy-server.conf-sample in the bulk section
> but it only mentions auth and ratelimiting not quotas
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1201844
>
> Title:
> Auto-extract feature of bulk.py can bypass quota settings
>
> Status in OpenStack Object Storage (Swift):
> New
>
> Bug description:
> When using the archive auto-extract feature of the bulk.py module, it
> is possible to blow out the account and/or container quotas that might
> be configured.
>
> Once the initial size of the archive to be uploaded passes the quota
> checks, nothing is done to ensure that the expanded size of the
> archive contents also meets quota restrictions. This allows for the
> opportunity of blowing the quota by quite a bit depending on the
> contents of the archive.
>
> Using Swift 1.8.0 release.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/swift/+bug/1201844/+subscriptions

[David Goetz (david-goetz)]

Probably because of the admin user. Don't know too much about container quotas. There should be very little difference between the expand subrequests and making many smaller client requests as long as the bulk appears before the other middleware in the pipeline.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/37472

[Chmouel Boudjnah (chmouel)]

@David I have added it to the proxy-server-sample so it gets as well tested in the functional tests on gating.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/37472
Committed: http://github.com/openstack/swift/commit/89ccd95996194bcfc5fb41c6102131457aa6fb9d
Submitter: Jenkins
Branch: master

commit 89ccd95996194bcfc5fb41c6102131457aa6fb9d
Author: Chmouel Boudjnah <email address hidden>
Date: Wed Jul 17 15:46:03 2013 +0200

    Add bulk middleware to proxy-server.conf-sample

    - Fixes bug 1201844.

    Change-Id: I8eed54d0a17a0c6b746ed616634fc9adb89e5f37