Cause conflicts within glance public metadefs

[Niall Bunting]

Overview:

Through creation of a new public namespace by any user of the system, you can create a clash of namespaces, that breaks all accessibility to that namespace. This therefore can be used to cause a denial of service attack or you have to disable the service completely.

How to produce:

As a regular user run the command:
curl -v -X POST http://16.49.138.140:9292/v2/metadefs/namespaces -H "Content-Type: application/json" -H "X-Auth-Token: 1a499605071a46a8b9b2a938fac5fac7" -d '{"namespace": "OS::Computer::WebServers", "visibility": "public"}'

This will create a new namespace with the same name as the existing namespace. This has now rendered the original namespace inaccessible. If a GET request is done to the namespaces name by any other user via (or viewing in horizon):
curl -v -X GET http://16.49.138.140:9292/v2/metadefs/namespaces/OS::Computer::WebServers -H "Content-Type: application/json" -H "X-Auth-Token: 1a499605071a46a8b9b2a938fac5fac7"

It will cause the following output in the api console:
2015-07-28 23:41:42.175 ERROR glance.api.v2.metadef_properties [req-e3a80995-6f37-4e5c-b7dd-a1ce978478c7 f76c222365fb490792300f9e49ec9bd0 9db14ac3320b4396b58222f99dd04e4e] Multiple rows were found for one()

Returning a 500 to the user and therefore the namespace inaccessible meaning a successful denial of service to most of the metadefs api as most require it.

Attempted preventative measures:
In the policy.json files there are only the following values:
"get_metadef_namespace": "",
 "get_metadef_namespaces":"",
 "modify_metadef_namespace":"",
"add_metadef_namespace":"",
meaning that creating namespaces has to be disabled completely(not default ) as there in no publicize option.

[Grant Murphy (gmurphy)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Niall Bunting (niall-bunting)]

The two fixes that need to be in place are:
1. The not allowing to create two namespaces with the same name.
2. Adding a policy to disallow non-admins publicizing namespaces.

[Nikhil Komawar (nikhil-komawar)]

Niall, are you working on solution for this?

OpenStack Vulnerability team: I would like to add Travis Tripp to subscribers as he is most familiar with metadef use cases and would like a solution proposal from him.

[Niall Bunting (niall-bunting)]

Nikhil, I am not currently working on this, however I would be happy to.

[Lakshmi N Sampath (lakshmi-sampath)]

Verified the issue. Seems like duplicate namespaces are getting created which was not the case before. Will have to verify the db code.

[Wayne (wayne-okuma)]

This bug has a fix that was submitted back in June: https://review.openstack.org/#/c/195820/

[Nikhil Komawar (nikhil-komawar)]

Thanks Wayne. Did not click then!

On 8/7/15 2:14 PM, Wayne wrote:
> This bug has a fix that was submitted back in June:
> https://review.openstack.org/#/c/195820/
>

--

Thanks,
Nikhil

[Stuart McLaren (stuart-mclaren)]

There is a previous bug/patch for this:

https://review.openstack.org/#/c/195820

but I'm not sure if the security aspect of it was covered.

From my -- fairly limited -- understanding of metadefs it does seem like there is a vulnerability here: a DOS of the metadefs API.

[Tristan Cacqueray (tristan-cacqueray)]

Is there others side effects beside the metadata API not being available ? For example, in a Kilo devstack setup, having the metadef API unavailable (returning 500) does not prevents instances to start or the image list to be displayed in Horizon.

I'm not familiar with that feature, is there a configuration option to enable its use by the others API ?

Assuming this was introduced in Juno, here is the impact description draft #1:

Title: Glance metadata v2 API DoS through duplicate namespaces
Reporter: Niall Bunting (HP)
Products: Glance
Affects: 2014.2 versions through 2014.2.3 and 2015.1.0 versions through 2015.1.1

Description:
Niall Buntin from HP reported a vulnerability in the Glance metadata v2 API. By creating duplicated namespaces, an authenticated user may crash the Glance metadata v2 API, resulting in a denial of services. Only setups using the Glance V2 API are affected by this flaw.

[Stuart McLaren (stuart-mclaren)]

Hi Tristan,

Thanks for the draft.

> Title: Glance metadata v2 API DoS through duplicate namespaces

I'd suggest changing "metadata" here to "metadefs". I think that makes it clearer that we're talking about the /v2/metadefs resource

Description: typo: Niall Buntin -> Niall Bunting

@Niall

Is the original metadef description recoverable after a user has written over it? Or has it been lost?

[Niall Bunting (niall-bunting)]

@Stuart

It's not lost its still in the database. However its just accessible through the API, with control over the database you could delete the other interfering entry.

[Tristan Cacqueray (tristan-cacqueray)]

Thanks for the review, here is the updated impact description draft #2:

Title: Glance metadefs v2 API DoS through duplicate namespaces
Reporter: Niall Bunting (HP)
Products: Glance
Affects: 2014.2 versions through 2014.2.3 and 2015.1.0 versions through 2015.1.1

Description:
Niall Bunting from HP reported a vulnerability in the Glance metadata v2 API. By creating duplicated namespaces, an authenticated user may crash the Glance metadata v2 API, resulting in a denial of services. Only setups using the Glance V2 API are affected by this flaw.

[Tristan Cacqueray (tristan-cacqueray)]

The fix https://review.openstack.org/#/c/195820/ does not applies properly to stable/juno at least.

Since it's already public, backports could be pushed to gerrit (referencing bug 1468946) while we keep this bug private to finalize the advisory side.

[Jeremy Stanley (fungi)]

I've subscribed the OSSG core security reviewers too.

I'm dubious about keeping this bug private. At this point bug 1468946 is public and mostly explains the risk (without explicitly calling it a denial of service). The fix is public in master and seems very close to being approved. Making this bug public will also make it a lot easier to explain to the stable branch reviewers why the backports are urgent and can hopefully speed up resolution there.

Unless there are serious objections, I want to switch this bug to public security this Wednesday, August 19.

[Robert Clark (robert-clark)]

Given the circumstances, I agree with Jeremy's proposed path of action (OSSG CoreSec)

[Travis McPeak (travis-mcpeak)]

+1 - enough of the issue is already public. It's a simple jump to go from what's listed in 1468946 to DoS.

[Thierry Carrez (ttx)]

The fix for master included a database schema migration, which is usually out of bounds for a stable backport (stable branches are a source of safe and automated fixes -- we need people to be able to upgrade without manual risky or lengthy steps like a database migration).

Is there any other way to fix the issue that does not involve database schema changes ?

[Wayne (wayne-okuma)]

I don't think there is a better way than to have the database system reject duplicates automatically. Otherwise, will have to introduce code to check before inserting rows that each record's keys don't already exist. This would be redundant to what the unique constraints do.

[Wayne (wayne-okuma)]

Also, not running the migration won't get the user out of the DOS against a duplicate namespace.

[Tristan Cacqueray (tristan-cacqueray)]

If this can't be backported, then we'll have to make this a class B1 ( https://security.openstack.org/vmt-process.html#incident-report-taxonomy ).

@ossg-coresec, thought about OSSA vs OSSN here ?

[Travis McPeak (travis-mcpeak)]

How about back-porting the input validation solutions to the branches where we don't want to do database migration?

My concern with a note is that I'm not sure what we would advise deployers to do. If we're fairly comfortable recommending the policy change mentioned by Niall in comment #2, a note around that advice seems appropriate.

[Niall Bunting (niall-bunting)]

@Travis We can't really advice what I mentioned due to the fact that there is no policy that will disallow publicizing images. The only other thing we could do is advise them to implement the fix.

[Tristan Cacqueray (tristan-cacqueray)]

Until this can be safely backported, the OSSA task is switched to Won't fix.