[OSSN-0088] Glance leaks namespace existence to unauthorized users

[Lance Bragstad]

╭─ubuntu@glance-devstack ~/devstack ‹master*›
╰─➤ $ source openrc demo demo
╭─ubuntu@glance-devstack ~/devstack ‹master*›
╰─➤ $ openstack token issue
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires | 2021-02-25T14:11:38+0000 |
| id | gAAAAABgN6IKDUKTn9RNudtZD605vA9l9eErCcXDrdxZwfhePYVlAHXzzCdQs6FK6XDwFvuexzfymc0uX7NY5RisEnQmUBl6eLccgBMYE6vSpVWCDTkFuKIuPfLh3xSkJGjZcpG7nfJ_ImU_wCJJFgcclf1zHTHWQ9Y15k-mAE7l3xceqUkOx2Y |
| project_id | ed4fade2e2cd4be0932ef30357f6d7a1 |
| user_id | e83b2f50463c4959bcc00a96b52b2f86 |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
╭─ubuntu@glance-devstack ~/devstack ‹master*›
╰─➤ $ glance md-namespace-show foo
+----------------------------+----------------------------------+
| Property | Value |
+----------------------------+----------------------------------+
| created_at | 2021-02-25T04:54:10Z |
| namespace | foo |
| owner | ed4fade2e2cd4be0932ef30357f6d7a1 |
| protected | False |
| resource_type_associations | ["bar"] |
| schema | /v2/schemas/metadefs/namespace |
| updated_at | 2021-02-25T04:54:10Z |
| visibility | private |
+----------------------------+----------------------------------+
╭─ubuntu@glance-devstack ~/devstack ‹master*›
╰─➤ $ source alicerc
╭─ubuntu@glance-devstack ~/devstack ‹master*›
╰─➤ $ glance md-resource-type-associate --name test foo
HTTP 403 Forbidden: Forbidding request, metadata definition namespace=foo is not visible.

This might not be a security issue since the user needs to know the namespace name, but opening this in private based on a recommendation from jokke.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Jeremy Stanley (fungi)]

It has come to my attention that Dan Smith was also aware of this report, so I have subscribed him after discussing with Lance.

[Dan Smith (danms)]

I think it is a security issue. We consider being able to test existence of another tenant's image by guessing its UUID to be an information leak, so being able to test for the existence of any other resource by name is worse, IMHO. Customers could easily be encoding sensitive information into resource names, which would make this more than just a leak of infrastructure details. However, since these issues seem to have been baked into the APIs since the beginning, I think we probably shouldn't hold up notification on a heavyweight retooling of the code to fix the underlying issues.

From what I've seen in some brief initial research on these issues, I think cooking up a real fix for this is likely to be non-trivial, especially given where we are in the cycle. I think that the best thing to do here might be to mitigate this problem by changing the policy rules to disable these metadef APIs by default, along with docs and renos about why, how to re-enable them, and what the potential risk is if you do. People using these APIs may decide to continue doing so, but perhaps with different naming conventions to limit exposure. As noted in another related bugs (#1916922) there are other exposures related to this, as well as resource exhaustion issues (#1545702) which would all be things we would want a refactor to address.

After we get those defaults changed and people notified, we can work on either a better long-term fix, or maybe decide that more invasive surgery is needed. AFAICT, these APIs have always been broken in this way, as the policy enforcement points don't pass any target object to the enforcer step(s). Thus it seems like the informational part of this (i.e. an advisory and mitigation steps) are the important part here, as well as warning any new users against building on this functionality without due care.

[Abhishek Kekane (abhishek-kekane)]

Agree with Dan here, at this point it makes sense to disable these API's using policy rules with proper documentation and notes and work for better fix afterwards.

[Jeremy Stanley (fungi)]

Okay, so to clarify, the current suggestion is to write a patch which disables the problem API method(s) with backports to all supported stable branches and attach them to this bug, decide on a disclosure date, privately notify downstream stakeholders with a copy of the patches and our timeline, then on the determined day switch this and perhaps related report(s) to public, push the changes to Gerrit and publish an advisory linking to them.

Thereafter, work will happen in public to recreate the intended functionality in a safer way.

Does that sum it up?

[Dan Smith (danms)]

I guess I'm not sure if/why the patch to disable this by default really needs to happen in total unison with the advisory. Since this is already disable-able via policy (and we'd just be flipping policy defaults in said patch), I would expect the least-effort and lowest-latency approach would be to issue the advisory, basically suggesting that people tweak their policy files to disable this. They don't really need new packages from their vendors in order to do that.

I'm also not sure on the backports, as that may surprise users who 'apt upgrade' (as opposed to dist-upgrade) their way to an API that is now disabled.

But, if Abhi wants to do all the stuff you suggest (and/or you think it's critical do it that way) then ... yes your list is correct :)

[Jeremy Stanley (fungi)]

OpenStack Security Advisories are a channel to notify consumers of the software of vulnerabilities which we have corrected, and to urge them to upgrade to protect their deployments. If the plan instead is to merely provide guidance to operators that they should make configuration changes to secure their deployments, we have a different mechanism for that which we refer to as OpenStack Security Notes (these are considered addenda of the Security Manual essentially). Rule of thumb, any solution which is "default secure" gets an OSSA, solutions requiring setting non-default configuration to mitigate are OSSN.

[Dan Smith (danms)]

Ack, I wasn't aware there was an actual distinction in name, but was sure we had a mechanism for the "notification of recommended config" as you note. So yeah, I guess I'd tend to lean towards the "note" option given that this has been broken since its introduction, but am fine with whatever Abhi decides.

[Abhishek Kekane (abhishek-kekane)]

As this is broken since introduction and can be disabled by policies, we can go ahead with OpenStack Security Notes and then work towards fixing it in coming cycle.

[Jeremy Stanley (fungi)]

While we don't normally draft security notes privately under embargo (unlike advisories), we have on occasion done so and notified downstream stakeholders with an advance copy prior to publication. This might be one of those times where private advance notification is prudent.

The process for writing an OSSN is documented here:

    https://wiki.openstack.org/wiki/Security/Security_Note_Process

Would anyone like to have a go at writing up some guidance for this and the related bugs? I gather it involves policy configuration changes to disable some API method(s) but the specifics are where I cease to be useful on this matter. I'm happy to help with coordinating a publication date and getting it sent to the stakeholders.

[Jeremy Stanley (fungi)]

I probably should also make it clear, please put the proposed text of an OSSN in a comment on this private bug report, not in the public wiki. We would copy it to the wiki on some future publication date which can be scheduled once we're all satisfied with the wording.

[Abhishek Kekane (abhishek-kekane)]

I will try to come up with proposed text by early next week.

[Abhishek Kekane (abhishek-kekane)]

Some of the Glance metadef APIs likely to leak resources
--------------------------------------------------------

### Summary ###
Metadef APIs are vulnerable and potentially leaking information to
unauthorized users and also there is currently no limit on creation
of metadef namespaces, objects, properties, resources and tags. This
can be abused by malicious users to fill the Glance database resulting
in a Denial of Service (DoS) condition.

### Affected Services / Software ###
Glance

### Discussion ###
There is no restriction on creation of metadef namespaces, objects,
properties, resources and tags as well as it could also leak the
information to unauthorized users or to the users outside of project. By
taking advantage of this lack of restrictions around metadef APIs, a
single user could cause the fill the Glance database by creating
unlimited resource, resulting in a Denial Of Service (DoS) style
attack.

Glance does allow metadef APIs to be controlled by policy. However, the
default policy setting for metadef APIs allows all users to create or
read the metadef information.

Because metadef resources are not properly isolated to the
owner, any use of them with potentially sensitive names (such as internal
infrastructure details, customer names, etc) could unintentionally
expose that information to a malicious user.

### Recommended Actions ###
Since these fundamental issues have been present since the API was
introduced, the Glance project is recommending operators disable all
metadef APIs by default in their deployments.

Here is an example of disabling the metadef APIs in the deployments for current
stable OpenStack releases either in policy.json or policy.yaml.

---- begin example policy.json/policy.yaml snippet ----
"metadef_default": "!",

"get_metadef_namespace": "rule:metadef_default",
"get_metadef_namespaces": "rule:metadef_default",
"modify_metadef_namespace": "rule:metadef_default",
"add_metadef_namespace": "rule:metadef_default",

"get_metadef_object": "rule:metadef_default",
"get_metadef_objects": "rule:metadef_default",
"modify_metadef_object": "rule:metadef_default",
"add_metadef_object": "rule:metadef_default",

"list_metadef_resource_types": "rule:metadef_default",
"get_metadef_resource_type": "rule:metadef_default",
"add_metadef_resource_type_association": "rule:metadef_default",

"get_metadef_property": "rule:metadef_default",
"get_metadef_properties": "rule:metadef_default",
"modify_metadef_property": "rule:metadef_default",
"add_metadef_property": "rule:metadef_default",

"get_metadef_tag": "rule:metadef_default",
"get_metadef_tags": "rule:metadef_default",
"modify_metadef_tag": "rule:metadef_default",
"add_metadef_tag": "rule:metadef_default",
"add_metadef_tags": "rule:metadef_default"
---- end example policy.json/policy.yaml snippet ----

To re-enable metadef policies to be allowed to be admin only, operator(s)
can make a change in respective policy.json or policy.yaml as shown below;
(assuming all metadef policies are configured to use rule:metadeta_default
as shown in above example)

---- begin example policy.json/policy.yaml snippet ----
"metadef_default": "rule:admin",
---- begin example policy.json/policy.yaml...

[Gage Hugo (gagehugo)]

Draft looks good, we may want to clarify the versions affects similar to past OSSN:

OpenStack Image Service (Glance) versions through Stein (<=18.0.1) as well as Train (<=19.0.4), Ussuri (<=20.0.1), and Victoria (<=21.0.0)

Not sure if we want to specify Wallaby as well?

[Jeremy Stanley (fungi)]

It's my understanding that all supported versions are affected, but that the Glance team may merge a default policy change in time for Wallaby (no changes are expected to be backported to older releases though). If the default policy is changing in Wallaby we could say it impacts all Glance versions prior to 18.0.0 and just leave it at that.

[Gage Hugo (gagehugo)]

Works for me

[Dan Smith (danms)]

The draft also looks good to me. I haven't actually deployed with that policy snippet to check it for typos, but it looks right to me.

We _are_ currently planning to make the policy default changes in Wallaby, but that hasn't happened yet (and the gate is currently busy for all and wedged for some projects).

My recommendation is that we go ahead and make this public and send the note. For all released versions, the note and suggested policy changes are the important and actionable thing. We'll follow up and try to get the policy defaults change agreed-to and merged for Wallaby, which will be much easier when this is public, as it will require either tempest or devstack changes to land before any policy change.

Sound okay?

[Abhishek Kekane (abhishek-kekane)]

Sounds correct approach to me at this moment.

[Gage Hugo (gagehugo)]

I am alright with this approach as well.

[Jeremy Stanley (fungi)]

Based on the facts that:

1. the risk is only present when users keep sensitive data in metadefs and namespace names

2. it's not a core component of Glance's functionality (at least checking https://refstack.openstack.org/#/guidelines I don't see anything having to do with metadefs in either the required or advisory images-.* entries)

3. the guidance to operators is to disallow access in order to protect users from otherwise relying on it

I'm fine going ahead with direct publication and not providing advance private notification to downstream stakeholders (public service providers and distributors). This is a reasonable compromise, in order to avoid ending up in a situation where we've got a known problem feature exposed by default in the upcoming Wallaby release.

While the announcement may take some operators of existing deployments by surprise, and cause a bit of a scramble to turn off access to a feature some of their users could be relying on, it's still better that we move forward with changing the default in Glance as soon as possible so we can reduce future pain.

[Jeremy Stanley (fungi)]

So I think the following set of tasks need to happen all at roughly the same time:

0. Pick the next OSSN number in sequence... looks like that should be OSSN-0088.

1. I will update this bug to Public Security, setting the security task Won't Fix and adding a security note bugtask assigned to Abhishek; I'll also remove the embargo preamble from the bug description and add [OSSN-0088] as a prefix on the title.

2. Abhishek (preferably) sends a copy of the security note from comment #13 to the openstack-announce mailing list with openstack-discuss in Cc (remember to prefix the subject with [OSSN-0088] when you do).

3. Someone paste a copy of the same into https://wiki.openstack.org/w/index.php?title=OSSN/OSSN-0088&action=edit and save it.

4. I'll approve Abhishek's message through the openstack-announce moderation queue.

5. Someone pushes the default change for the master branch of openstack/glance to Gerrit for review.

Does this sound like a reasonable plan? If so, I'm ready to do #1 when Abhishek is ready to send the message for #2.

[Abhishek Kekane (abhishek-kekane)]

It almost midnight at my end, is it ok if we do it tomorrow (or Dan can do it)?
Also for comment 13 I know the 'This OSSN' will be OSSN-0088 but what will be the 'CVE number' or it should be left out?

[Jeremy Stanley (fungi)]

Yes, that's fine. Tomorrow is actually better anyway, since people are generally more likely to notice and be able to immediately act on Tuesday announcements (we actually try to avoid making announcements on Mondays, Fridays, weekends, or major international holidays).

Would 15:00 UTC work for you, or is that too late? I can be available to coordinate it with you in IRC earlier than that too, just let me know.

[Jeremy Stanley (fungi)]

The recommendations are now published here: https://wiki.openstack.org/wiki/OSSN/OSSN-0088