[OSSA 2015-020] Glance storage quota bypass when token is expired (CVE-2015-5286)

[Mike Fedosin]

About a year ago it was a vulnerability called 'Glance user storage quota bypass': https://security.openstack.org/ossa/OSSA-2015-003.html, where any user could overcome the quota and clog up the storage.
The fix was proposed in master and all other stable branches, but it turned out, that it doesn't completely remove the issue and any user still can exceed the quota.

It happens in case if user token is expired during file upload and when glance tries to update image status from 'saving' to 'active'. Then glance gets Unauthenticated exception from registry server and fails with 500 error. On the other side garbage file is left in storage.

Steps to reproduce mostly coincide with the related from the previous bug, but in general it is:
1. Set some value (like 1Gb) to user_storage_quota in glance-api.conf and restart the server.
2. Make sure that your token will expire soon, when you'll be able to create an image instance in DB and begin the upload, but the token will expire during it.
3. Create an image, begin the upload and quickly remove the image with 'glance image-delete'.
4. After the upload check that image is not in the list, i.e. it's deleted, and file is still located in the store.
5. Perform steps 2-4 several times to make sure that user quota is exceeded.

Related script (test_images.py from here https://bugs.launchpad.net/glance/+bug/1398830) works fine, too, but it's better to reduce token life time in keystone config to 1 or 2 minutes, just for not to wait for one hour.

Glance api v2 is affected as well, but only if registry db_api is enabled.

[Nikhil Komawar (nikhil-komawar)]

Thanks Mike. This is a legitimate issue.

The above patch looks good for v1 and v2. Can you please apply the same for import task too?

Thanks!

[Mike Fedosin (mfedosin)]

Thanks for your comment Nikhil. I'll do it tomorrow!

[Tristan Cacqueray (tristan-cacqueray)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Nikhil Komawar (nikhil-komawar)]

This looks tentative for RC1 but keeping here for tracking.

[Tristan Cacqueray (tristan-cacqueray)]

It seems like this could use an OSSA. However token lifetime is not user-controllable afaik... can the user tries more than 1 attempt per token expiration ?

Also the impact is different from OSSA-2015-003 if the left-over images are still located in the store.

Here is the proposed impact description.

Title: Glance storage overrun
Reporter: Mike Fedosin (Mirantis)
Products: Glance
Affects: versions through 2014.2.3 and 2015.1 versions through 2015.1.1

Description:
Mike Fedosin from Mirantis reported a vulnerability in Glance. By deleting images that are being uploaded using a token that is about to expire, a malicious user can overcome the storage quota and accumulate untracked image data in the backend resulting in potential resource exhaustion and denial of service. All Glance setups using the V1 API are affected and all setups using the V2 API with the registry db_api enabled are affected.

[Mike Fedosin (mfedosin)]

Hi again Tristan :)

Token lifetime is not user-controllable, indeed. But user can avoid it by waiting a decent amount of time.
Example:
Let's assume that token lifetime is set for one hour, which is default for Keystone, and user quota is 5,5Gb. To exceed the quota user gets the token (with 'keystone token-get') and waits for 55 minutes, after that he consecutively creates and deletes 5Gb images each 3 seconds. After 5 minutes his token expires, but uploading of all images continues. And when server wants to update theirs status from 'saving' to 'active' it fails with 500 error and clogs the storage.

Also please add Alexei Galkin as reporter, too - because it's he, who originally found this bug. He works in Mirantis as well.

[Jeremy Stanley (fungi)]

It seems like this is a much, much slower potential denial of service than that described in bug 1398830 (for OSSA 2015-003).

[Tristan Cacqueray (tristan-cacqueray)]

Well the user can also generate a lot of tokens in advance too. But I agree the impact is rather moderate here, and considering the release schedule constrain, I would suggest to threat this bug without embargo.

[Mike Fedosin (mfedosin)]

Technically the impact is the same, because user can do exactly the same things, but a little bit trickier.
But yeah, I can't deny that here security risk is smaller. I consider this patch as an amendment to the previous one, which fully eliminates the possibility of quota bypass.

By the way, this patch also fixes the consequences of the old famous bug with token expiration.

[Mike Fedosin (mfedosin)]

I made a fix for tasks as Nikhil asked.

[Nikhil Komawar (nikhil-komawar)]

Thanks Mike, the patch looks good to me.

[Nikhil Komawar (nikhil-komawar)]

Need to add glance-core-sec team to this bug as I can't find them on the right panel.

[Tristan Cacqueray (tristan-cacqueray)]

Jeremy, I'd like to keep the proposed impact description, the title seems correct to me since the left-over image remain in the backend. This is actually worst than bug 1398830 (for OSSA 2015-003) where left-over image are actually deleted after the upload finished iiuc.

Here is an update impact description to include the additional reporter and new affected version line:

Title: Glance storage overrun
Reporter: Mike Fedosin and Alexei Galkin (Mirantis)
Products: Glance
Affects: <=2014.2.3, >=2015.1.0, <=2015.1.1

Description:
Mike Fedosin and Alexei Galkin from Mirantis reported a vulnerability in Glance. By deleting images that are being uploaded using a token that is about to expire, a malicious user can overcome the storage quota and accumulate untracked image data in the backend resulting in potential resource exhaustion and denial of service. All Glance setups using the V1 API are affected and all setups using the V2 API with the registry db_api enabled are affected.

[Nikhil Komawar (nikhil-komawar)]

I'm ok with either.

[Hemanth Makkapati (hemanth-makkapati)]

The patch looks good to me.
A couple of minor comments though.
1. Logging a warn message in v2 is probably not required. A debug log message just like the one you have for v1 should suffice in my opinion
2. The log message says it's an 'Authorization problem'. It's authentication really.

Anyway, if there isn't enough time to address the above minor issues, that's fine.

[Flavio Percoco (flaper87)]

The code looks good to me. I took the liberty of addressing Hermanth comments. Here are the three patches backported from master all the way back to juno

[Stuart McLaren (stuart-mclaren)]

Thanks Mike/Flavio.

Will the fix work for the multi-tenant swift store? In that case can we delete image data when we don't have a token?

[Flavio Percoco (flaper87)]

Stuart,

I honestly don't have a way to test that but, by reading the code, I think it does. If you have a chance to test this w/ a multi-tenant swift store, please, help! :)

[Mike Fedosin (mfedosin)]

Hi, sorry for late response...

About Hemanth's second comment: but anyway we have to raise Unauthorized exception, as it was done in the other cases, for example here https://github.com/openstack/glance/blob/master/glance/api/v2/images.py#L90
Also it seems that it's more authZ problem rather than AuthN, because authenticated user can't be authorized with expired token.

@Stuart, unfortunately no... To delete files form Multitenant store valid user's token is required. It can't be fixed without trusts. But it's turned off by default and considered as experimental realization, am I right?

[Flavio Percoco (flaper87)]

Can we schedule this bug to be disclosed next week?

[Jeremy Stanley (fungi)]

Tristan's impact description in comment #16 looks great.

[Tristan Cacqueray (tristan-cacqueray)]

Proposed public disclosure date/time:
2015-10-01, 1500UTC

[Nikhil Komawar (nikhil-komawar)]

Adding some context from IRC conversations I had recently on this bug,

the latest proposed fixes do not include the tasks related fix. So, the in flight review atm -- https://review.openstack.org/#/q/I17a66eca55bfb83107046910e69c4da01415deec,n,z , do not include the fix from https://bugs.launchpad.net/glance/+bug/1498163/comments/13 for getting rid if this vulnerability via tasks.

This is definitely out of some confusion so, I think we need to send another patch for tasks to avoid more of the same.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/229971

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/229972

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/229973

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/229975

[Nikhil Komawar (nikhil-komawar)]

https://review.openstack.org/#/q/I17a66eca55bfb83107046910e69c4da01415deec,n,z and https://review.openstack.org/#/q/Ia6e1fe0d27b13b920ee7e728feb5305dec77e066,n,z together look good. Please do the needful.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/230056

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229943
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=301bf5108a99ee5299c50089927edfd71024eb58
Submitter: Jenkins
Branch: master

commit 301bf5108a99ee5299c50089927edfd71024eb58
Author: Mike Fedosin <email address hidden>
Date: Sun Sep 20 17:01:22 2015 +0300

    Cleanup chunks for deleted image if token expired

    In patch I47229b366c25367ec1bd48aec684e0880f3dfe60 it was
    introduced the logic that if image was deleted during file
    upload when we want to update image status from 'saving'
    to 'active' it's expected to get Duplicate error and delete
    stale chunks after that. But if user's token is expired
    there will be Unathorized exception and chunks will stay
    in store and clog it.
    And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image
    which was deleted from saving status, by correctly handle
    auth exceptions from registry server.

    Partial-bug: #1498163

    Change-Id: I17a66eca55bfb83107046910e69c4da01415deec

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229971
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=5be26e13ade3f33b98f551940ca39b25a5790c80
Submitter: Jenkins
Branch: master

commit 5be26e13ade3f33b98f551940ca39b25a5790c80
Author: Mike Fedosin <email address hidden>
Date: Thu Oct 1 18:28:48 2015 +0300

    Catch NotAuthenticated exception in import task

    If glance uses registry as data_api then it's possible
    that token may expire during image import task and Glance
    will have NotUauthenticated exception.

    This code adds a correct handling of this exception and
    allows Glance to remove stale chunks from store.

    Partial-Bug: #1498163

    Change-Id: Ia6e1fe0d27b13b920ee7e728feb5305dec77e066

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229972
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=ebdf076cc9bd5d9239cdc96c6e7cecc72f852bbb
Submitter: Jenkins
Branch: stable/liberty

commit ebdf076cc9bd5d9239cdc96c6e7cecc72f852bbb
Author: Mike Fedosin <email address hidden>
Date: Thu Oct 1 18:28:48 2015 +0300

    Catch NotAuthenticated exception in import task

    If glance uses registry as data_api then it's possible
    that token may expire during image import task and Glance
    will have NotUauthenticated exception.

    This code adds a correct handling of this exception and
    allows Glance to remove stale chunks from store.

    Partial-Bug: #1498163

    Change-Id: Ia6e1fe0d27b13b920ee7e728feb5305dec77e066

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/230056
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=fc32f0554de0ba7773d98e6828da157ca7c66002
Submitter: Jenkins
Branch: stable/liberty

commit fc32f0554de0ba7773d98e6828da157ca7c66002
Author: Mike Fedosin <email address hidden>
Date: Sun Sep 20 17:01:22 2015 +0300

    Cleanup chunks for deleted image if token expired

    In patch I47229b366c25367ec1bd48aec684e0880f3dfe60 it was
    introduced the logic that if image was deleted during file
    upload when we want to update image status from 'saving'
    to 'active' it's expected to get Duplicate error and delete
    stale chunks after that. But if user's token is expired
    there will be Unathorized exception and chunks will stay
    in store and clog it.
    And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image
    which was deleted from saving status, by correctly handle
    auth exceptions from registry server.

    Partial-bug: #1498163

    Change-Id: I17a66eca55bfb83107046910e69c4da01415deec

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229946
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=5447e8419d92f0cbb14de53b207e772ce9067933
Submitter: Jenkins
Branch: stable/juno

commit 5447e8419d92f0cbb14de53b207e772ce9067933
Author: Mike Fedosin <email address hidden>
Date: Sun Sep 20 17:01:22 2015 +0300

    Cleanup chunks for deleted image if token expired

    In patch I47229b366c25367ec1bd48aec684e0880f3dfe60 it was
    introduced the logic that if image was deleted during file
    upload when we want to update image status from 'saving'
    to 'active' it's expected to get Duplicate error and delete
    stale chunks after that. But if user's token is expired
    there will be Unathorized exception and chunks will stay
    in store and clog it.
    And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image
    which was deleted from saving status, by correcly handle
    auth exceptions from registry server.

    This patch fixes the issue of left over chunks for an image
    which was deleted from saving status, by correctly handle
    auth exceptions from registry server.

    Partial-bug: #1498163

    Conflicts:
     glance/api/v1/upload_utils.py
            (Kilo catches NotFound instead of ImagenotFound)

    Change-Id: I17a66eca55bfb83107046910e69c4da01415deec
    (cherry picked from commit 50e3a7c58a9862206d92fef577540c5b144ecbf0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229945
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=5bebd513fa71edcdb84f7dec7b16f3523c0c1092
Submitter: Jenkins
Branch: stable/kilo

commit 5bebd513fa71edcdb84f7dec7b16f3523c0c1092
Author: Mike Fedosin <email address hidden>
Date: Sun Sep 20 17:01:22 2015 +0300

    Cleanup chunks for deleted image if token expired

    In patch I47229b366c25367ec1bd48aec684e0880f3dfe60 it was
    introduced the logic that if image was deleted during file
    upload when we want to update image status from 'saving'
    to 'active' it's expected to get Duplicate error and delete
    stale chunks after that. But if user's token is expired
    there will be Unathorized exception and chunks will stay
    in store and clog it.
    And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image
    which was deleted from saving status, by correctly handle
    auth exceptions from registry server.

    Partial-bug: #1498163

    Conflicts:
     glance/api/v1/upload_utils.py
            (Kilo catches NotFound instead of ImagenotFound)

    Change-Id: I17a66eca55bfb83107046910e69c4da01415deec
    (cherry picked from commit 98a8832777a0639a4031e52c69f0d565b3f500c5)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229975
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=868d9161c60f839cbf53393b804d7c0c41338e5b
Submitter: Jenkins
Branch: stable/juno

commit 868d9161c60f839cbf53393b804d7c0c41338e5b
Author: Mike Fedosin <email address hidden>
Date: Thu Oct 1 18:28:48 2015 +0300

    Catch NotAuthenticated exception in import task

    If glance uses registry as data_api then it's possible
    that token may expire during image import task and Glance
    will have NotUauthenticated exception.

    This code adds a correct handling of this exception and
    allows Glance to remove stale chunks from store.

    Partial-Bug: #1498163

    Change-Id: Ia6e1fe0d27b13b920ee7e728feb5305dec77e066
    (cherry picked from ebdf076cc9bd5d9239cdc96c6e7cecc72f852bbb)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/229973
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=15e34b54a5ad59d9aeb0284cc31453eb43b60cf1
Submitter: Jenkins
Branch: stable/kilo

commit 15e34b54a5ad59d9aeb0284cc31453eb43b60cf1
Author: Mike Fedosin <email address hidden>
Date: Thu Oct 1 18:28:48 2015 +0300

    Catch NotAuthenticated exception in import task

    If glance uses registry as data_api then it's possible
    that token may expire during image import task and Glance
    will have NotUauthenticated exception.

    This code adds a correct handling of this exception and
    allows Glance to remove stale chunks from store.

    Partial-Bug: #1498163

    Change-Id: Ia6e1fe0d27b13b920ee7e728feb5305dec77e066
    (cherry picked from ebdf076cc9bd5d9239cdc96c6e7cecc72f852bbb)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/235346

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/235346
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=431fa95ecf92a9ebb7082f829c0a99f958363cc3
Submitter: Jenkins
Branch: master

commit 69516fad5f651a085a047a337a05c58b39023c1b
Author: Mike Fedosin <email address hidden>
Date: Mon Oct 12 15:34:54 2015 +0300

    Add 'deactivated' status to image schema.

    New 'deactivated' status was introduced in Kilo release,
    but it doesn't listed in available image statuses in the schema.

    It leads to issues on the client side, when it can't validate
    the image with this status against the schema and returns the error.

    Change-Id: I5ec264614ae7ecf54b846ad0600442a18c61d24c
    Closes-bug: #1505218
    Related-bug: #1505134

commit c5b6901527b8b4a1250bdc179405c8af66fbae7e
Author: Mike Fedosin <email address hidden>
Date: Tue Oct 13 00:33:27 2015 +0300

    Add testresources and testscenarios used by oslo.db fixture

    If we use oslo.db fixtures, we'll need these 2 packages or
    the next version of oslo.db release will break us.

    Change-Id: I7c0d2f6dabc20bd4ff0d29d3b47b948aa24ea56b
    Closes-Bug: #1503501

commit fc32f0554de0ba7773d98e6828da157ca7c66002
Author: Mike Fedosin <email address hidden>
Date: Sun Sep 20 17:01:22 2015 +0300

    Cleanup chunks for deleted image if token expired

    In patch I47229b366c25367ec1bd48aec684e0880f3dfe60 it was
    introduced the logic that if image was deleted during file
    upload when we want to update image status from 'saving'
    to 'active' it's expected to get Duplicate error and delete
    stale chunks after that. But if user's token is expired
    there will be Unathorized exception and chunks will stay
    in store and clog it.
    And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image
    which was deleted from saving status, by correctly handle
    auth exceptions from registry server.

    Partial-bug: #1498163

    Change-Id: I17a66eca55bfb83107046910e69c4da01415deec

commit ca8d909a61ba335805d8d17070230ce9478a000d
Author: Stuart McLaren <email address hidden>
Date: Wed Sep 30 16:54:12 2015 +0000

    Download forbidden when get_image_location is set.

    When using v2 an attempt to download an image would return a 403 if the
    get_image_location policy was set.

    Note: We had been returning both 404 and 204 when no data was
    available. There was no way to detect the 404 case without trying to
    access the image locations so I've standardized on 204.

    Change-Id: I658b08a35d3a8cb8a7096baf716ccb3d6e7d9abf
    Closes-bug: 1501672
    (cherry picked from commit b47f625443c3b46483506926f31fee42478705d4)

commit ebdf076cc9bd5d9239cdc96c6e7cecc72f852bbb
Author: Mike Fedosin <email address hidden>
Date: Thu Oct 1 18:28:48 2015 +0300

    Catch NotAuthenticated exception in import task

    If glance uses registry as data_api then it's possible
    that token may expire during image import task and Glance
    will have NotUauthenticated exception.

    This code adds...