[OSSA-2018-002] GET /v3/OS-FEDERATION/projects leaks project information (CVE-2018-14432)

[Lance Bragstad]

The /v3/OS-FEDERATION/projects API was developed to let federated users discover what projects they have access to. This mirrored a similar API in keystone, v3/auth/projects. Both were intended to behave the same way, by only returning what projects a user has a role assignment on.

Eventually the /v3/OS-FEDERATION/projects API was deprecated after the /v3/auth/projects API was able to support federated tokens.

The /v3/OS-FEDERATION/projects API appears to be broken because it returns all projects in the deployment, not just the ones a user has access to. The following recreates the issue:

lbragstad|devstack|~ >>> cat /etc/openstack/clouds.yaml
clouds:
  devstack:
    auth:
      auth_url: http://192.168.1.5/identity
      password: nomoresecret
      project_domain_id: default
      project_name: demo
      user_domain_id: default
      username: demo
    identity_api_version: '3'
    region_name: RegionOne
    volume_api_version: '2'
  devstack-admin:
    auth:
      auth_url: http://192.168.1.5/identity
      password: nomoresecret
      project_domain_id: default
      project_name: admin
      user_domain_id: default
      username: admin
    identity_api_version: '3'
    region_name: RegionOne
    volume_api_version: '2'
  devstack-alt:
    auth:
      auth_url: http://192.168.1.5/identity
      password: nomoresecret
      project_domain_id: default
      project_name: alt_demo
      user_domain_id: default
      username: alt_demo
    identity_api_version: '3'
    region_name: RegionOne
    volume_api_version: '2'
lbragstad|devstack|~ >>> openstack role assignment list --names --os-cloud devstack-admin
+-------------+------------------+-------------------+----------------------------+---------+-----------+
| Role | User | Group | Project | Domain | Inherited |
+-------------+------------------+-------------------+----------------------------+---------+-----------+
| member | | nonadmins@Default | demo@Default | | False |
| anotherrole | | nonadmins@Default | demo@Default | | False |
| member | | nonadmins@Default | alt_demo@Default | | False |
| anotherrole | | nonadmins@Default | alt_demo@Default | | False |
| admin | | admins@Default | admin@Default | | False |
| admin | admin@Default | | demo@Default | | False |
| admin | admin@Default | | admin@Default | | False |
| admin | admin@Default | | alt_demo@Default | | False |
| admin | admin@Default | | | Default | False |
| member | demo@Default | | demo@Default | | False |
| anotherrole | demo@Default | | demo@Default | | False |
| member | demo@Default | | invisible_to_admin@Default | | False |
| member | alt_demo@Default | | alt_demo@Default | | False |
| anotherrole | alt_demo@Default | | alt_demo@Default | | False |
| admin | admin@Default | | | | False |
+-------------+------------------+-------------------+----------------------------+---------+-----------+
lbragstad|devstack|~ >>> openstack token issue --os-cloud devstack
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires | 2018-06-28T21:03:47+0000 |
| id | gAAAAABbNT8jpqwHC_ZeIQvo3YBIvp0UxletD5d0xZ7BFbuNbNGminfpp0FtG5RYZgIIIW4i8OOMtYnmDtQ1b4FOGLzFexayG5D3gTTrDBvQAFy95gQiaSxxJGsscCQ36pxiFWxqA0KBzvdCMPDpYDtuG1pd0b3KiskApbVcwE-uuESisyzj36w |
| project_id | 44053df0d12f4ba0aa4c28c3364aa1a1 |
| user_id | cef2773684114d55a6399e928ecc78e4 |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
lbragstad|devstack|~ >>> export TOKEN='gAAAAABbNT8jpqwHC_ZeIQvo3YBIvp0UxletD5d0xZ7BFbuNbNGminfpp0FtG5RYZgIIIW4i8OOMtYnmDtQ1b4FOGLzFexayG5D3gTTrDBvQAFy95gQiaSxxJGsscCQ36pxiFWxqA0KBzvdCMPDpYDtuG1pd0b3KiskApbVcwE-
uuESisyzj36w'
lbragstad|devstack|~ >>> curl -H "X-Auth-Token: $TOKEN" http://localhost/identity/v3/auth/projects | python -m json.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 669 100 669 0 0 20476 0 --:--:-- --:--:-- --:--:-- 20906
{
    "links": {
        "next": null,
        "previous": null,
        "self": "http://192.168.1.5/identity/v3/auth/projects"
    },
    "projects": [
        {
            "description": "",
            "domain_id": "default",
            "enabled": true,
            "id": "44053df0d12f4ba0aa4c28c3364aa1a1",
            "is_domain": false,
            "links": {
                "self": "http://192.168.1.5/identity/v3/projects/44053df0d12f4ba0aa4c28c3364aa1a1"
            },
            "name": "demo",
            "parent_id": "default",
            "tags": []
        },
        {
            "description": "",
            "domain_id": "default",
            "enabled": true,
            "id": "8c92de6ab3884f94b508ce2f2dd62c4d",
            "is_domain": false,
            "links": {
                "self": "http://192.168.1.5/identity/v3/projects/8c92de6ab3884f94b508ce2f2dd62c4d"
            },
            "name": "invisible_to_admin",
            "parent_id": "default",
            "tags": []
        }
    ]
}
lbragstad|devstack|~ >>> curl -H "X-Auth-Token: $TOKEN" http://localhost/identity/v3/OS-FEDERATION/projects | python -m json.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 1270 100 1270 0 0 17528 0 --:--:-- --:--:-- --:--:-- 17638
{
    "links": {
        "next": null,
        "previous": null,
        "self": "http://192.168.1.5/identity/v3/OS-FEDERATION/projects"
    },
    "projects": [
        {
            "description": "",
            "domain_id": "default",
            "enabled": true,
            "id": "44053df0d12f4ba0aa4c28c3364aa1a1",
            "is_domain": false,
            "links": {
                "self": "http://192.168.1.5/identity/v3/projects/44053df0d12f4ba0aa4c28c3364aa1a1"
            },
            "name": "demo",
            "parent_id": "default",
            "tags": []
        },
        {
            "description": "Bootstrap project for initializing the cloud.",
            "domain_id": "default",
            "enabled": true,
            "id": "681b94352ed146b5ac37c152653e90d2",
            "is_domain": false,
            "links": {
                "self": "http://192.168.1.5/identity/v3/projects/681b94352ed146b5ac37c152653e90d2"
            },
            "name": "admin",
            "parent_id": "default",
            "tags": []
        },
        {
            "description": "",
            "domain_id": "default",
            "enabled": true,
            "id": "9a742b4684dc4c8a90dc4896f9ab178e",
            "is_domain": false,
            "links": {
                "self": "http://192.168.1.5/identity/v3/projects/9a742b4684dc4c8a90dc4896f9ab178e"
            },
            "name": "alt_demo",
            "parent_id": "default",
            "tags": []
        },
        {
            "description": "",
            "domain_id": "default",
            "enabled": true,
            "id": "8c92de6ab3884f94b508ce2f2dd62c4d",
            "is_domain": false,
            "links": {
                "self": "http://192.168.1.5/identity/v3/projects/8c92de6ab3884f94b508ce2f2dd62c4d"
            },
            "name": "invisible_to_admin",
            "parent_id": "default",
            "tags": []
        }
    ]
}

Notice that I used the devstack cloud config, which specifies the demo user who only has the `member` and `anotherrole` assigned on two projects (demo and invisible_to_admin). In no way should they have access to view all projects in the deployment.

[Lance Bragstad (lbragstad)]

Kristi came to me with this today in a private IRC discussion. We confirmed the bug, but we accidentally put the results in paste.openstack.org [0][1]. Someone could discover this issue by scrubbing paste (including for full disclosure).

[0] http://paste.openstack.org/show/724547/
[1] http://paste.openstack.org/show/manColzRRAOHKGrZhyNQ/

[Kristi Nikolla (knikolla)]

Just tested this on our public cloud running Pike.

Using a federated token on /v3/OS-FEDERATION/projects gave me the full list of projects on the cloud.
Using a local token on /v3/OS-FEDERATION/projects returns 500.

[Morgan Fainberg (mdrnstm)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Morgan Fainberg (mdrnstm)]

<Keystone Developer Hat>This looks like a pretty severe data leak. While usually project info is not really "privileged data", this leaks the entire project structure and all associated attributes. I would develop a fix and propose backports.

<VMT Member Hat>I would likely classify this as a Class A [0] bug due to the severity of data leaking. This should be backportable to all active branches. While this is not directly exploitable (there is no escalation/inappropriate actions able to be taken) this exposes knowledge of every project in the cloud where it is clearly is not intended.

Alternatively, this could be a Class D bug since no escalation/inappropriate action can be taken (as of now) even with the entire list of projects.

I'll look for further weighing in from the Keystone-Coresec team.

Finally, if the fixes cannot be generated shortly/quickly the fact that this was accidentally released on a public paste site; it would make sense to make public so that cloud operators could modify their policy.json (or load-balancers/other device doing layer-7 inspection/routing) to block access to OS-FEDERATION auth URLs.

[0] https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[Gage Hugo (gagehugo)]

Agreed on Class A bug, should definitely fix/backport to all current active branches.

According to the docs, OS-FEDERATION/projects was deprecated in v1.1, would one option for fixing this be to disable it outright (have it return an empty list as to not break anyone)?

[Tristan Cacqueray (tristan-cacqueray)]

I confirmed the OSSA task based on above comments.

When was the v1.1 released?

If users are relying on that endpoints, they might already be relying on the full list... Could this OS-FEDERATION/projects endpoint return a http redirection to /auth/projects ?

[Lance Bragstad (lbragstad)]

I agree in Morgan's assessment in comment #4, this is a pretty severe data leak.

I'm not sure what the rules are on using redirects, but I wouldn't be opposed to using that as a fix. Especially since it reduces duplication in API that should behave the same.

[Lance Bragstad (lbragstad)]

There is another API that is very similar for getting domains (GET /v3/OS-FEDERATION/domains). This might be affected as well. I can attempt to recreate at some point today.

[Morgan Fainberg (mdrnstm)]

I like the idea of a HTTP 301 for this. It may not be handled well by all clients, but it ensures the code is the same and that folks are using the new path.

We could also just wire up the behind the scenes code to be the same code path, which should also solve the problem.

Being as severe a data leak with an alternative path, I would also support making it a 410 (GONE) with a message saying why it was removed.

[Morgan Fainberg (mdrnstm)]

(or simply 404ing it)

[Morgan Fainberg (mdrnstm)]

The 410/404 method might be the recommended "emergency" path forward if we make this public. It should be trivial to wire this up behind the scenes (at a cursory glance of the code) to de-dupe divergent (and now insecure) code paths.

[Lance Bragstad (lbragstad)]

I was able to recreate this with a test and hacked together a fix that replaces the duplicated logic with a call to the same code that executes GET /v3/auth/projects (the correct API).

I'd like to get opinions from a few others in this thread on the test coverage and solution.

[Lance Bragstad (lbragstad)]

FWIW - I wasn't able to recreate this with domains.

[Kristi Nikolla (knikolla)]

OS-FEDERATION/domains seems to return a random domain(?)

knikolla@smile:~$ curl -H "x-auth-token: <REDACTED_TOKEN>" -H "x-subject-token: <REDACTED_TOKEN>" https://kaizen.massopen.cloud/identity/v3/auth/tokens | python -m json.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 394 100 394 0 0 2359 0 --:--:-- --:--:-- --:--:-- 2359
{
    "token": {
        "audit_ids": [
            "C8ec_RXRTWGVRbOc48MZdg"
        ],
        "expires_at": "2018-06-29T21:03:04.000000Z",
        "issued_at": "2018-06-29T20:03:04.000000Z",
        "methods": [
            "openid"
        ],
        "user": {
            "OS-FEDERATION": {
                "groups": [],
                "identity_provider": {
                    "id": "moc"
                },
                "protocol": {
                    "id": "openid"
                }
            },
            "domain": {
                "id": "Federated",
                "name": "Federated"
            },
            "id": "<REDACTED_ID>",
            "name": "<REDACTED_EMAIL>"
        }
    }
}
knikolla@smile:~$ curl -H "x-auth-token: <REDACTED_TOKEN>" https://kaizen.massopen.cloud/identity/v3/OS-FEDERATION/domains | python -m json.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 375 100 375 0 0 1194 0 --:--:-- --:--:-- --:--:-- 1194
{
    "domains": [
        {
            "description": "Contains users and projects created by heat",
            "enabled": true,
            "id": "<REDACTED_ID>",
            "links": {
                "self": "https://kaizen.massopen.cloud/identity/v3/domains/<REDACTED_ID>"
            },
            "name": "heat"
        }
    ],
    "links": {
        "next": null,
        "previous": null,
        "self": "https://kaizen.massopen.cloud/identity/v3/OS-FEDERATION/domains"
    }
}
knikolla@smile:~$ curl -H "x-auth-token: <REDACTED_TOKEN>" https://kaizen.massopen.cloud/identity/v3/auth/domains | python -m json.tool
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 124 100 124 0 0 1215 0 --:--:-- --:--:-- --:--:-- 1215
{
    "domains": [],
    "links": {
        "next": null,
        "previous": null,
        "self": "https://kaizen.massopen.cloud/identity/v3/auth/domains"
    }
}

[Lance Bragstad (lbragstad)]

Strange... we should expand the patch in comment #12 to include the domain path using the same approach (including a test, too).

[Morgan Fainberg (mdrnstm)]

"A Random Domain" is concerning. Lets update (as Lance said) patch in #12 to cover domain cases as well.

[Kristi Nikolla (knikolla)]

Updated Lances patch to cover domains in a similar way.

[Lance Bragstad (lbragstad)]

The patch in comment #17 looks good. Just double checking, but it doesn't look like we have a test for the random domain bit that Kristi mentioned in comment #14.

Also, in the event we disclose this issue, do we formally agree on the work around? Which is to deactivate the affected APIs via policy.

[Matthew Thode (prometheanfire)]

Hi, the VMT are taking on padawans, myself being one of them. Please review this VMT description for accuracy and clarity.

NOTE: https://security.openstack.org/vmt-process.html#draft-impact-description states that the mitigation method should be specified, but the template (used below) does not have it. Also, I do not know the first release that introduced this bug so I included all YYYY.C releases as affected.

Title: GET /v3/OS-FEDERATION/projects leaks project information
Reporter: Kristi Nikolla
Products: Keystone
Affects: >=2011.3 <=2015.1.4, >=13.0.0 <13.0.1, >=12.0.0 <12.0.1, <11.0.4

Description:
Kristi Nikolla reported a vulnerability in Keystone federation.
By doing GET /v3/OS-FEDERATION/domains an actor may read project
access control data resulting in a leaks a projects full structure
along with all associated attributes.
Only Keystone with federation enabled is affected.

[Lance Bragstad (lbragstad)]

The API for accessing domains isn't actually affected. Based on what Kristi and I have been able to recreate, only GET /v3/OS-FEDERATION/projects is affected.

Kristi did have a case where he was seeing a "domain mismatch" that we should try and include as a testcase in the patch.

[Kristi Nikolla (knikolla)]

Also this does affect Keystone where federation is not enabled also.

[Lance Bragstad (lbragstad)]

The mismatching domain piece actually won't need a test case because the code that exposed the issue is being removed in the patch.

[Matthew Thode (prometheanfire)]

Please review this VMT description for accuracy and clarity. I need the following two points validated specifically.

1. When was this introduced? I'm currently including all keystone releases, but I don't think federation was in 2011.3
2. Is this bug only able to be hit when federation is enabled?

NOTE: https://security.openstack.org/vmt-process.html#draft-impact-description states that the mitigation method should be specified, but the template (used below) does not have it. Also, I do not know the first release that introduced this bug so I included all YYYY.A.B releases as affected.

Title: GET /v3/OS-FEDERATION/projects leaks project information
Reporter: Kristi Nikolla
Products: Keystone
Affects: >=2011.3 <=2015.1.4, >=13.0.0 <13.0.1, >=12.0.0 <12.0.1, <11.0.4

Description:
Kristi Nikolla reported a vulnerability in Keystone federation.
By doing GET /v3/OS-FEDERATION/projects an actor may read project
access control data resulting in a leaks a projects full structure
along with all associated attributes.
Only Keystone with federation enabled is affected.

[Lance Bragstad (lbragstad)]

Answers are as follows:

1.) I believe this affects all supported releases. Kristi discovered and confirmed the bug on a deployment running Pike. It looks like this was the original commit that introduced the functionality [0], which doesn't necessarily mean that all those releases are affected. I'm working on standing up an ocata environment to see if that specific release is affected (I assume it is since Pike is affected and not much changed between those two releases).

2.) Correct, the /v3/OS-FEDERATION/ path must be enabled and accessible to users. Although a user doesn't need a federated token in order to expose the vulnerability. A user only needs an unscoped token proving their membership within the deployment. The noted workaround would be to disable those APIs via policy configuration and force users to use the /v3/auth/projects path, which isn't affected.

I have a minor nit on the grammatical wording in the Description above:

"By doing GET /v3/OS-FEDERATION/projects an actor may discover
projects they have no authority to access, leaking all projects
in the deployment and their attributes."

[0] https://review.openstack.org/#/c/71353/

[Gage Hugo (gagehugo)]

1. Looking at the history, this is a bit vague. The federation controller was moved sometime in Newton. Will need to dig into this more, but it's definitely all current supported versions afaik.
2. No, (see Kristi's comment above #21)

[Gage Hugo (gagehugo)]

Addon 2. Lance's comment "the /v3/OS-FEDERATION/ path must be enabled and accessible to users" is a good thing to include in the description.

[Matthew Thode (prometheanfire)]

Please review this VMT description for accuracy and clarity.

NOTE: https://security.openstack.org/vmt-process.html#draft-impact-description states that the mitigation method should be specified, but the template (used below) does not have it.

Title: GET /v3/OS-FEDERATION/projects leaks project information
Reporter: Kristi Nikolla
Products: Keystone
Affects: >=2014.1 <=2015.1.4, >=13.0.0 <13.0.1, >=12.0.0 <12.0.1, <11.0.4

Description:
Kristi Nikolla reported a vulnerability in Keystone federation.
By doing GET /v3/OS-FEDERATION/projects an actor may discover
projects they have no authority to access, leaking all projects
in the deployment and their attributes.
Only Keystone with the OS-FEDERATION endpoint enabled is affected.

[Morgan Fainberg (mdrnstm)]

As a note, OS-FEDERATION endpoint cannot be disabled short of changes to policy.json even as far back as Ocata. It is not practical to highlight "only keystone with the OS-FEDERATION endpoint enabled is affected".

We should add Kristi's organization (if he wants it) to the report.

With exception of the NOTEs I've provided this is def. headed the right direction for an impact statement.

[Morgan Fainberg (mdrnstm)]

Note: My above assertion is incorrect. as this requires a federated token. OS-FEDERATION as a URL is not something that can be disabled. However, if the user is unable to get a federated token (no federation endpoint, aka shibboleth via apache) is setup, the OS-FEDERATION code raises a 500 error (as described by kristi).

We should be more specific in that this only impacts federated auth tokens to avoid confusion (like I just had) regarding what is meant by OS-FEDERATION

[Matthew Thode (prometheanfire)]

Please review this VMT description for accuracy and clarity.

NOTE: https://security.openstack.org/vmt-process.html#draft-impact-description states that the mitigation method should be specified, but the template (used below) does not have it.

Title: GET /v3/OS-FEDERATION/projects leaks project information
Reporter: Kristi Nikolla
Products: Keystone
Affects: <11.0.4, ==13.0.0, ==12.0.0

Description:
Kristi Nikolla reported a vulnerability in Keystone federation.
By doing GET /v3/OS-FEDERATION/projects with a federated token
an actor may discover projects they have no authority to access,
leaking all projects in the deployment and their attributes.
Only Keystone with the /v3/OS-FEDERATION endpoint enabled via
policy.json is affected.

[Kristi Nikolla (knikolla)]

Morgan, the 500 is what I reported when trying /OS-FEDERATION/projects on our public cloud. There it only worked with a federated token.

On a Devstack (master branch), it actually worked with any token regardless if it is federated or not.

ubuntu@devstack-ldap:~$ source devstack/openrc demo demo
WARNING: setting legacy OS_TENANT_NAME to support cli tools.
ubuntu@devstack-ldap:~$ export TOKEN=`openstack token issue -f value -c id`
ubuntu@devstack-ldap:~$ curl -H "x-auth-token: $TOKEN" http://localhost/identity/v3/OS-FEDERATION/projects
{"links": {"self": "http://192.168.0.8/identity/v3/OS-FEDERATION/projects", "previous": null, "next": null}, "projects": [{"is_domain": false, "description": "", "links": {"self": "http://192.168.0.8/identity/v3/projects/0b93617b390e41de94594a99e3e154e6"}, "tags": [], "enabled": true, "domain_id": "default", "parent_id": "default", "id": "0b93617b390e41de94594a99e3e154e6", "name": "invisible_to_admin"}, {"is_domain": false, "description": "", "links": {"self": "http://192.168.0.8/identity/v3/projects/e1a071e7b2f74f6cac392505e086878d"}, "tags": [], "enabled": true, "id": "e1a071e7b2f74f6cac392505e086878d", "parent_id": "default", "domain_id": "default", "name": "alt_demo"}, {"is_domain": false, "description": "", "links": {"self": "http://192.168.0.8/identity/v3/projects/6c9b45545ca44817a2bf2b634ca01fa1"}, "tags": [], "enabled": true, "domain_id": "default", "parent_id": "default", "id": "6c9b45545ca44817a2bf2b634ca01fa1", "name": "demo"}, {"is_domain": false, "description": "Bootstrap project for initializing the cloud.", "links": {"self": "http://192.168.0.8/identity/v3/projects/9f99b93e45874c41a89551252b3b9d4c"}, "tags": [], "enabled": true, "id": "9f99b93e45874c41a89551252b3b9d4c", "parent_id": "default", "domain_id": "default", "name": "admin"}]}

[Kristi Nikolla (knikolla)]

Forgot to mention in the previous comment that our public cloud is running Pike

[Morgan Fainberg (mdrnstm)]

ack. we will need to be specific. we should try pike without federation setup as well, I wonder when we switched from failing wih a local token to succeeding.

If someone else can't confirm before this weekend, I'll start working on replicating without federation from pike forward. So we can be as accurate as possible.

[Lance Bragstad (lbragstad)]

I just got an Ocata deployment up and running. I'm unable to get a list of projects from the v3/OS-FEDERATION/projects API with an unfederated token (unscoped, project-scoped, or otherwise).

I believe the KeyError is due to the API assuming that group_ids must be present (e.g. mapping assertions from SAML into group ids and populating them in auth context).

As far as I know, this is inconvenient from a UX perspective. It also doesn't "leak" information like it does on master when handed an unfederated token.

2018-07-12 21:16:19.525 3451 DEBUG keystone.common.controller [req-9038014f-da7e-48d2-9445-6dcc2cbd1a89 3c1b6b7382c246838b99e8943b797a67 e32ef264357b494da96cb437c24533cd - default default] RBAC: Authorizing identity:list_projects_for_user() _build_policy_check_credentials /opt/stack/keystone/keystone/common/controller.py:78
2018-07-12 21:16:19.526 3451 DEBUG keystone.policy.backends.rules [req-9038014f-da7e-48d2-9445-6dcc2cbd1a89 3c1b6b7382c246838b99e8943b797a67 e32ef264357b494da96cb437c24533cd - default default] enforce identity:list_projects_for_user: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': u'3c1b6b7382c246838b99e8943b797a67', 'roles': [u'admin'], 'user_domain_id': u'default', 'consumer_id': None, 'trustee_id': None, 'is_domain': False, 'is_admin_project': True, 'trustor_id': None, 'token': <KeystoneToken (audit_id=cUS2MG6ISrOuFHP2l9jVxA, audit_chain_id=cUS2MG6ISrOuFHP2l9jVxA) at 0x7ffb8766f178>, 'project_id': u'e32ef264357b494da96cb437c24533cd', 'trust_id': None, 'project_domain_id': u'default'} enforce /opt/stack/keystone/keystone/policy/backends/rules.py:77
2018-07-12 21:16:19.530 3451 DEBUG keystone.common.controller [req-9038014f-da7e-48d2-9445-6dcc2cbd1a89 3c1b6b7382c246838b99e8943b797a67 e32ef264357b494da96cb437c24533cd - default default] RBAC: Authorization granted inner /opt/stack/keystone/keystone/common/controller.py:163
2018-07-12 21:16:19.530 3451 ERROR keystone.common.wsgi [req-9038014f-da7e-48d2-9445-6dcc2cbd1a89 3c1b6b7382c246838b99e8943b797a67 e32ef264357b494da96cb437c24533cd - default default] 'group_ids': KeyError: 'group_ids'
2018-07-12 21:16:19.530 3451 ERROR keystone.common.wsgi Traceback (most recent call last):
2018-07-12 21:16:19.530 3451 ERROR keystone.common.wsgi File "/opt/stack/keystone/keystone/common/wsgi.py", line 228, in __call__
2018-07-12 21:16:19.530 3451 ERROR keystone.common.wsgi result = method(req, **params)
2018-07-12 21:16:19.530 3451 ERROR keystone.common.wsgi File "/opt/stack/keystone/keystone/common/controller.py", line 164, in inner
2018-07-12 21:16:19.530 3451 ERROR keystone.common.wsgi return f(self, request, *args, **kwargs)
2018-07-12 21:16:19.530 3451 ERROR keystone.common.wsgi File "/opt/stack/keystone/keystone/federation/controllers.py", line 477, in list_projects_for_user
2018-07-12 21:16:19.530 3451 ERROR keystone.common.wsgi request.auth_context['group_ids'])
2018-07-12 21:16:19.530 3451 ERROR keystone.common.wsgi KeyError: 'group_ids'
2018-07-12 21:16:19.530 3451 ERROR keystone.common.wsgi

[Lance Bragstad (lbragstad)]

I would adjust the description in comment #30 by removing the term "federated":

"By doing GET /v3/OS-FEDERATION/projects with a federated token
an actor may discover projects they have no authority to access,
leaking all projects in the deployment and their attributes."

to

"Calling GET /v3/OS-FEDERATION/projects an actor may discover
projects they have no authority to access, leaking all projects
in the deployment and their attributes."

[Matthew Thode (prometheanfire)]

Please review this VMT description for accuracy and clarity.

NOTE: https://security.openstack.org/vmt-process.html#draft-impact-description states that the mitigation method should be specified, but the template (used below) does not have it.

Title: GET /v3/OS-FEDERATION/projects leaks project information
Reporter: Kristi Nikolla
Products: Keystone
Affects: <11.0.4, ==13.0.0, ==12.0.0

Description:
Kristi Nikolla reported a vulnerability in Keystone federation.
By doing GET /v3/OS-FEDERATION/projects with a token an actor
may discover projects they have no authority to access, leaking
all projects in the deployment and their attributes.
Only Keystone with the /v3/OS-FEDERATION endpoint enabled via
policy.json is affected.

[Tristan Cacqueray (tristan-cacqueray)]

Kristi, is there an affiliation/employer we can credit for your report?

Matthew, maybe we could replace "with a token an actor may" by "an authenticated user may" ?

The description looks good to me, thanks.

[Matthew Thode (prometheanfire)]

Sure, I'll wait before editing again though (for the company). Don't want too many in a day.

[Kristi Nikolla (knikolla)]

Hi Tristan,

Please use "Boston University" for affiliation.

[Matthew Thode (prometheanfire)]

Please review this VMT description for accuracy and clarity.

NOTE: https://security.openstack.org/vmt-process.html#draft-impact-description states that the mitigation method should be specified, but the template (used below) does not have it.

Title: GET /v3/OS-FEDERATION/projects leaks project information
Reporter: Kristi Nikolla with Boston University
Products: Keystone
Affects: <11.0.4, ==13.0.0, ==12.0.0

Description:
Kristi Nikolla with Boston University reported a vulnerability
in Keystone federation. By doing GET /v3/OS-FEDERATION/projects
an authenticated user may discover projects they have no
authority to access, leaking all projects in the deployment and
their attributes.
Only Keystone with the /v3/OS-FEDERATION endpoint enabled via
policy.json is affected.

[Jeremy Stanley (fungi)]

Thanks Matthew! Other than making sure the Affects header is sorted in a chronologically increasing order, that impact description looks great. We should be able to proceed with privately requesting a CVE assignment from MITRE, and assuming Kristi/Lance/someone is prepared to attach backports of the patch from comment #17 for each of our supported stable branches we can probably start discussing a disclosure schedule for this one.

[Lance Bragstad (lbragstad)]

The description in comment #41 looks good. The patch is pretty trivial and it won't take much to get it proposed to the affected releases. Just for clarity though, here is the summary of proposing the patch from comment #17 to all supported branches:

master: Developed for Rocky, no conflicts.
stable/queens: Patch cleanly cherry-picked from master.
stable/pike: Patch didn't apply cleanly due to internal changes to use a different dependency registry. The fix should still be trivial to apply since it's not using the old or new dependency registry directly (but it won't be exactly like the patch applied to master).
stable/ocata: Same conflict as noted with stable/pike.

Thoughts on the cases for stable/pike and stable/ocata?

Since we have at least 4 keystone folks on this thread, I think we can schedule a time to fast-track the fix and backport it through the supported branches as soon as the VMT gives us the nod. I'd also like to get at least two keystone cores to +2/+A the patch in comment #17 prior to proposals to gerrit (in order to keep review time to a minimum after public disclosure).

[Jeremy Stanley (fungi)]

If there is to be progressive downstream disclosure under embargo, we'd need the clean backports attached to this bug in advance so that we can put them in the pre-OSSA notification to other registered stakeholders. Then at the scheduled public disclosure time the same patches would be proposed to Gerrit (immediately after this bug report is made public) and an official advisory would be issued linking to them. https://security.openstack.org/vmt-process.html#embargoed-disclosure

[Lance Bragstad (lbragstad)]

Attached is the resolved cherry-pick for stable/pike.

[Lance Bragstad (lbragstad)]

Attached is the resolved cherry-pick for stable/pike.

[Lance Bragstad (lbragstad)]

Disregard the patch in comment #45. The patch attached here is the correct patch for stable/pike.

[Lance Bragstad (lbragstad)]

Apparently I had a second patch locally I kept uploading... sorry for the spam :(

This should be the correct patch for stable/pike (third time is a charm)

[Lance Bragstad (lbragstad)]

Attached is the patch with conflicts resolved for stable/ocata.

[Gage Hugo (gagehugo)]

The patches look alright, looks like things got moved around between ocata/pike/queens-on but the change is the same.

[Kristi Nikolla (knikolla)]

Looks good to me.

[Lance Bragstad (lbragstad)]

Thanks Kristi and Gage for the reviews. We'll wait for the nod from the VMT before proposing anything to gerrit. Once we get a time and date for the disclosure, we'll make sure we have at least a couple cores around to fast-approve the reviews.

[Morgan Fainberg (mdrnstm)]

I just wanted to swoop in and as a Keystone core offer a +2 for the current patches (as a VMT member everything is looking good).

When we are ready to publish the changes:

Lance: you, I, Kristi, or Gage can push the patches in and approve them in one swoop (as keystone-cores) referencing this bug for approvals.

[Matthew Thode (prometheanfire)]

We have a potential disclosure date of Wednesday, 25-07-2018, 1500UTC. Are Keystone cores available at that time/date?

[Gage Hugo (gagehugo)]

I should be available at that time.

[Lance Bragstad (lbragstad)]

Matthews, I'll be available at that time. I'll wait for confirmation from Kristi and Morgan to make sure we have appropriate core coverage, including stable branches. Thanks for being available, Gage.

[Kristi Nikolla (knikolla)]

I will be available at that time also.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/585782

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/585788

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/585792

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/585802

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585788
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=7b1b6a6fc8bdbd2218501282be760d87d096de61
Submitter: Zuul
Branch: stable/queens

commit 7b1b6a6fc8bdbd2218501282be760d87d096de61
Author: Lance Bragstad <email address hidden>
Date: Wed Jul 25 15:07:16 2018 +0000

    Reduce duplication in federated auth APIs

    The GET /v3/OS-FEDERATION/projects and GET /v3/OS-FEDERATION/domains
    APIs were introduced to handle tokens from federated users, but now
    that GET /v3/auth/projects and GET /v3/auth/domains know how to handle
    federated tokens, they're just duplicate APIs.

    In the past we deprecated these federated auth APIs, but they still
    used separate code paths from GET /v3/auth/projects and GET
    /v3/auth/domains. The two code paths are true duplication in that they
    don't expect to differ over time and should provide the same user
    experience.

    Instead of running the risk that comes with two code paths that do the
    same thing, we should consolidate them.

    Co-Authored-By: Kristi Nikolla <email address hidden>

    Closes-Bug: 1779205
    Change-Id: Ib906c42e1dd2c2408ccd2e256ffd876af02af3fe
    (cherry picked from commit df5d75571ed274b2964ed52048768c6d9f24d138)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585802
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=ade177ad357d28746ab4203e56a74e6e1e89c074
Submitter: Zuul
Branch: stable/ocata

commit ade177ad357d28746ab4203e56a74e6e1e89c074
Author: Lance Bragstad <email address hidden>
Date: Wed Jul 25 15:07:16 2018 +0000

    Reduce duplication in federated auth APIs

    The GET /v3/OS-FEDERATION/projects and GET /v3/OS-FEDERATION/domains
    APIs were introduced to handle tokens from federated users, but now
    that GET /v3/auth/projects and GET /v3/auth/domains know how to handle
    federated tokens, they're just duplicate APIs.

    In the past we deprecated these federated auth APIs, but they still
    used separate code paths from GET /v3/auth/projects and GET
    /v3/auth/domains. The two code paths are true duplication in that they
    don't expect to differ over time and should provide the same user
    experience.

    Instead of running the risk that comes with two code paths that do the
    same thing, we should consolidate them.

    Conflicts:
      keystone/federation/controllers.py due to the fact that pre-Queens
      code used a different dependency framework. This was reworked in
      the Queens release, causing a conflict with this patch since it
      touches the same code.
      keystone/tests/unit/test_v3_auth.py due to a couple test with
      similar naming and placement within the same test module.

    Co-Authored-By: Kristi Nikolla <email address hidden>

    Closes-Bug: 1779205
    Change-Id: Ib906c42e1dd2c2408ccd2e256ffd876af02af3fe
    (cherry picked from commit df5d75571ed274b2964ed52048768c6d9f24d138)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585782
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=df5d75571ed274b2964ed52048768c6d9f24d138
Submitter: Zuul
Branch: master

commit df5d75571ed274b2964ed52048768c6d9f24d138
Author: Lance Bragstad <email address hidden>
Date: Wed Jul 25 15:07:16 2018 +0000

    Reduce duplication in federated auth APIs

    The GET /v3/OS-FEDERATION/projects and GET /v3/OS-FEDERATION/domains
    APIs were introduced to handle tokens from federated users, but now
    that GET /v3/auth/projects and GET /v3/auth/domains know how to handle
    federated tokens, they're just duplicate APIs.

    In the past we deprecated these federated auth APIs, but they still
    used separate code paths from GET /v3/auth/projects and GET
    /v3/auth/domains. The two code paths are true duplication in that they
    don't expect to differ over time and should provide the same user
    experience.

    Instead of running the risk that comes with two code paths that do the
    same thing, we should consolidate them.

    Co-Authored-By: Kristi Nikolla <email address hidden>

    Closes-Bug: 1779205
    Change-Id: Ib906c42e1dd2c2408ccd2e256ffd876af02af3fe

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585792
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=ce46cc25dc4d967c062587ab21b2b38cab045e00
Submitter: Zuul
Branch: stable/pike

commit ce46cc25dc4d967c062587ab21b2b38cab045e00
Author: Lance Bragstad <email address hidden>
Date: Wed Jul 25 15:07:16 2018 +0000

    Reduce duplication in federated auth APIs

    The GET /v3/OS-FEDERATION/projects and GET /v3/OS-FEDERATION/domains
    APIs were introduced to handle tokens from federated users, but now
    that GET /v3/auth/projects and GET /v3/auth/domains know how to handle
    federated tokens, they're just duplicate APIs.

    In the past we deprecated these federated auth APIs, but they still
    used separate code paths from GET /v3/auth/projects and GET
    /v3/auth/domains. The two code paths are true duplication in that they
    don't expect to differ over time and should provide the same user
    experience.

    Instead of running the risk that comes with two code paths that do the
    same thing, we should consolidate them.

    Conflicts:
      keystone/federation/controllers.py due to the fact that pre-Queens
      code used a different dependency framework. This was reworked in
      the Queens release, causing a conflict with this patch since it
      touches the same code.

    Co-Authored-By: Kristi Nikolla <email address hidden>

    Closes-Bug: 1779205
    Change-Id: Ib906c42e1dd2c2408ccd2e256ffd876af02af3fe
    (cherry picked from commit df5d75571ed274b2964ed52048768c6d9f24d138)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 13.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 11.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 12.0.1 release.

[Abhishek Sharma M (abhi.sharma)]

Description part in http://seclists.org/oss-sec/2018/q3/59 says that - "Only Keystone with the /v3/OS-FEDERATION endpoint enabled via policy.json is affected". We are not using federation in our product & neither is there any mention of this api in our keystone's policy.json, but still we see that any user with a token is able to run GET /v3/OS-FEDERATION/projects & gets all projects in response.

[Guang Yee (guang-yee)]

As far as I can tell, this problem seem to impact stable/queens and master (Rocky) only. It does NOT seem to impact the older branches. My assessment was based on devstack with federation enabled. i.e.

  # only enable the services we care about
  disable_all_services
  enable_plugin keystone git://git.openstack.org/openstack/keystone.git stable/pike
  enable_service mysql rabbit keystone keystone-saml2-federation

  KEYSTONE_ENABLE_MOD_WSGI=True
  KEYSTONE_BRANCH=stable/pike

I setup K2K federtation with the same instance acting as both IdP and SP. btw,
keystone-saml2-federation does not support this configuration so I had do some manual work. Devstack was deployed in a vagrant so each branch was deployed in a
fresh vagrant. So far I tested the following branches:

newton-eol
stable/ocata
stable/pike
stable/queens
master

Here's what I found.

1. The problem does NOT impact the federated tokens. Using a federated token, /v3/OS-FEDERATION/projects API correctly return the 'federated_project' in the 'federated_domain' only. This is true for ALL the branches.

2. Using a local user token (i.e. demo/demo), /v3/OS-FEDERATION/projects API will return ALL the projects for all the domains in stable/queens and master only, regardless of user's group membership. stable/pike and older branches are NOT impacted. With those branches, user will get an HTTP 500 Internal Server Error from the API and something similar to this in the keystone logs.

  Aug 1 11:18:27 ubuntu <email address hidden>[19553]: ERROR keystone.common.wsgi #033[01;35m#033[00m File "/opt/stack/keystone/keystone/federation/controllers.py", line 472, in list_projects_for_user
  Aug 1 11:18:27 ubuntu <email address hidden>[19553]: ERROR keystone.common.wsgi #033[01;35m#033[00m request.auth_context['group_ids'])
  Aug 1 11:18:27 ubuntu <email address hidden>[19553]: ERROR keystone.common.wsgi #033[01;35m#033[00mKeyError: 'group_ids'

[Guang Yee (guang-yee)]

The original problem may've caused by a couple of things.

1. This patch basically automatically added the 'group_ids' attribute in the auth_context regardless whether there are any groups or not.

https://review.openstack.org/#/c/523650/

It landed in Queens. This patch basically enabled the /v3/OS-FEDERATION/projects API to NOT failed with an HTTP 500 Internal Server Error with the presence of the 'group_ids' key in auth_context.

2. This could be a another bug. When listing assignments for groups, we should be using the groups filter, not 'source_from_group_ids'.

https://github.com/openstack/keystone/blob/master/keystone/assignment/core.py#L242

'source_from_group_ids' work differently. See

https://github.com/openstack/keystone/blob/master/keystone/assignment/core.py#L961

So it appears that if 'source_from_group_ids' is empty, it will return the kitchen-and-sink, so to speak. In other words, this API was secured purely by dumb luck. :-)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/588331

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 14.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lance Bragstad (<email address hidden>) on branch: master
Review: https://review.openstack.org/588331