Potential information disclosure in EC2 "credentials"

[Charles Neill]

When creating a "credential" in Keystone, instead of using uuid.uuid4() like in most places to generate a unique identifier, the id is created from the SHA256 hash value of whatever is passed in as the "access" key in the POST request (Code here: https://github.com/openstack/keystone/blob/master/keystone/credential/controllers.py#L36-L60)

===== EXAMPLE REQUEST =====

    POST /v3/credentials HTTP/1.1
    Host: [ENDPOINT]
    X-Auth-Token: [ADMIN TOKEN]
    Content-Length: 231
    Content-Type: application/json

    {
        "credential": {
            "blob": "{\"access\":\"<script>alert(2)</script>\",\"secret\":\"secretKey\"}",
            "project_id": "12345",
            "type": "ec2",
            "user_id": "12345"
        }
    }

    HTTP/1.1 201 Created
    Date: Tue, 30 Aug 2016 19:14:54 GMT
    Server: Apache/2.4.7 (Ubuntu)
    Vary: X-Auth-Token
    Content-Length: 383
    Content-Type: application/json

    {"credential": {"user_id": "12345", "links": {"self": "[ENDPOINT]/v3/credentials/141ce7a938b5973dd71c90bcdd7e4097317ee7374259cf6d8774fdfd86c1f8ea"}, "blob": "{\"access\":\"<script>alert(2)</script>\",\"secret\":\"secretKey\"}", "project_id": "12345", "type": "ec2", "id": "141ce7a938b5973dd71c90bcdd7e4097317ee7374259cf6d8774fdfd86c1f8ea"}}

===== /EXAMPLE =====

The id from the example above is "141ce7a938b5973dd71c90bcdd7e4097317ee7374259cf6d8774fdfd86c1f8ea", which is the same as the SHA256 value of "<script>alert(2)</script>" (you can test this with `echo -n "<script>alert(2)</script>" | openssl dgst -sha256` on *nix)

The documentation here seems to show MD5s and possibly tenant IDs used as "access" values: http://developer.openstack.org/api-ref/identity/v3/?expanded=assign-role-to-user-on-projects-owned-by-domain-detail,create-policy-detail,show-credential-details-detail,list-credentials-detail,create-credential-detail#list-credentials

Bruteforcing an actual MD5 isn't a huge security risk (i.e. trying to predict all 32 characters from thin air), but if the MD5 is a hash of a known value (i.e. the string "admin"), it would be trivial to test for common values:

    md5(admin) = 21232f297a57a5a743894a0e4a801fc3
    sha256(21232f297a57a5a743894a0e4a801fc3) = 465c194afb65670f38322df087f0a9bb225cc257e43eb4ac5a0c98ef5b3173ac

If tenant IDs are used, this task becomes even easier: just generate SHA256 hashes for 0 - 999999

A non-admin user can determine whether there are credentials using a given access key by attempting to access the resource from its sha256 url identifier:

===== EXAMPLE REQUESTS =====

Existing credential

    GET /v3/credentials/141ce7a938b5973dd71c90bcdd7e4097317ee7374259cf6d8774fdfd86c1f8ea HTTP/1.1
    Host: [ENDPOINT]
    X-Auth-Token: [NON-ADMIN TOKEN]
    Content-Type: application/json
    Connection: close

    HTTP/1.1 403 Forbidden
    Date: Tue, 30 Aug 2016 19:55:24 GMT
    Server: Apache/2.4.7 (Ubuntu)
    Vary: X-Auth-Token
    Content-Length: 140
    Content-Type: application/json

    {"error": {"message": "You are not authorized to perform the requested action: identity:get_credential", "code": 403, "title": "Forbidden"}}

Non-existent credential

    GET /v3/credentials/deadbeef HTTP/1.1
    Host: [ENDPOINT]
    X-Auth-Token: [NON-ADMIN TOKEN]
    Content-Type: application/json

    HTTP/1.1 404 Not Found
    Date: Tue, 30 Aug 2016 20:03:38 GMT
    Server: Apache/2.4.7 (Ubuntu)
    Vary: X-Auth-Token
    Content-Length: 96
    Content-Type: application/json

    {"error": {"message": "Could not find credential: deadbeef", "code": 404, "title": "Not Found"}}

===== /EXAMPLE =====

It is also possible to get a 500 error by creating a credential with an invalid character in the "access" key:

===== EXAMPLE REQUEST =====

    POST /v3/credentials HTTP/1.1
    Host: [ENDPOINT]
    X-Auth-Token: [ADMIN TOKEN]
    Content-Length: 212
    Content-Type: application/json

    {
        "credential": {
            "blob": "{\"access\":\"\uffff\",\"secret\":\"secretKey\"}",
            "project_id": "12345",
            "type": "ec2",
            "user_id": "12345"
        }
    }

    HTTP/1.1 500 Internal Server Error
    Date: Tue, 30 Aug 2016 20:06:16 GMT
    Server: Apache/2.4.7 (Ubuntu)
    Vary: X-Auth-Token
    Content-Length: 143
    Content-Type: application/json

    {"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

===== /EXAMPLE =====

I'm unsure what the security impact would be here, mainly because of the ambiguous examples provided in the Keystone API documentation (linked above). If either of the 2 scenarios I outlined is a reasonable use case (i.e. MD5 of a guessable value, or tenant IDs), there may be a risk of information leakage by brute-force. It would also be possible to prevent others from creating credentials with a given access key by simply creating lots of credentials in Keystone with predictable access keys. This would cause a collision whenever attempting to create a credential set with an access key that has already been used.

If, on the other hand, the credentials are always in the format described by AWS here ( link: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html ), it would require a huge number of requests to bruteforce the access key (though it would not be impossible). However, it would be possible, using the approach described above with a regular user token, to determine whether a known EC2 access key was in place as a credential in a given Keystone database.

I'm unclear on the utility of using SHA256 for the identifier at all here, since random UUIDs would make this potential issue moot.

[Morgan Fainberg (mdrnstm)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Morgan Fainberg (mdrnstm)]

Ok, A couple of comments here:

Tenant IDs are UUID4 (previous incarnations back in the original days not withstanding) and should not be purely bruteforceable as are user_ids. The example you referenced is poor in that it is very contrived values; this however does not imply a lack of security issue here.

This only affects ec2 credentials, normal credentials get the standard uuid generation:
https://github.com/openstack/keystone/blob/f7f1ee74350f62cb3e3fe9e7eb3adc0eaa99d85e/keystone/credential/controllers.py#L62

It looks like in the code creating EC2 credentials, the access token is in-fact a uuid.uuid4():

https://github.com/openstack/keystone/blob/f7f1ee74350f62cb3e3fe9e7eb3adc0eaa99d85e/keystone/contrib/ec2/controllers.py#L181-L191

Overall this looks to be generally an impractical security risk (guessing uuids). The blocking of "credential" would likewise be impractical since the DB should fill up long before a uuid4 or sha256 collides when it comes from a uuid4 source material.

There is a minor concern with the lack of entropy when converting a uuid4 to a SHA256 sum.

Finally, the "500" error should be a separate bug and is not part of the security related issues.

With all the above information, I am guessing this is a "Class D" bug https://security.openstack.org/vmt-process.html#incident-report-taxonomy

I'll wait for confirmation from the rest of the keystone coresec team before making any changes/updates here.

[Morgan Fainberg (mdrnstm)]

Alternative would be a C1 since it involved guessing UUIDs:

https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[Charles Neill (charles-neill)]

Apologies for neglecting to mention that this is indeed only relevant for EC2 credentials.

I didn't investigate the EC2 controller previously, but it's certainly a good thing that it uses UUIDs by default. The likelihood of bruteforcing this space is very low in the real world, and I would agree that C1 seems reasonable for that potential scenario.

For this to be a more practical attack, it seems an admin user would have to manually add credentials as "ec2" type, and choose something sensitive and/or predictable as the access key, instead of typical values. I think this is at least a possibility for some users, given that the API documentation currently discusses mostly EC2 credentials, and uses values that might be misleading to users who are unaware of what "EC2 credentials" are in the first place.

Clarifying the documentation on this matter and mentioning it in a short OSSN might be helpful, so that anyone who might've accidentally used this functionality in a non-standard way can take mitigating actions.

Re: "500" errors, our team is currently testing Keystone for security defects with the tool we've been building (https://github.com/openstack/syntribos) as part of OSIC, and I anticipate we will probably find a few more of these types of bugs. In the future, should we post these 500 errors individually, or aggregate them together? Thanks!

[Charles Neill (charles-neill)]

One other point I forgot to mention in my last comment - the ability for a non-admin user to detect the use of a KNOWN access key is still possible regardless of what that access key is

[Morgan Fainberg (mdrnstm)]

Regarding the 500 error, if it's not a common fix, separate bugs for each error would be good. It comes down to being better at validating the input. If the bugs are solved in the same place (definitively) you can group them.

[Morgan Fainberg (mdrnstm)]

Re Comment #5, I think that is still clearly a Class C1 since the typical case of EC2 credentials is uuid4 - in most cases it's guessing a UUID or someone is explicitly going around the API that ensures the use of UUID4.

[Charles Neill (charles-neill)]

Re: #5 It is not true that the user has to "guess" here - if they already have the access key, they can confirm/disconfirm that it is in use by Keystone without any guesswork (the SHA256 of the access key is deterministic - the SHA256 of the access key is either present, indicating use of that access key, or it isn't). It is of course up to the team to decide whether this is a risk worth mitigating.

Re: #7 - I would disagree that they are "going around the API." The API allows you to put whatever you want in the "access" key, and nowhere in the documentation does it even make reference to UUIDs. It is true that they would have to be accessing the API _manually_, but they're not doing anything special to force the API to accept non-UUIDs.

If UUIDs are supposed to be the only thing the API accepts, they should be enforced on the credentials creation endpoint. That is different from having one potential path by which users would automatically generate UUIDs, but free rein to insert whatever else they want manually. Perhaps at least a minimum length on the access key would be a good mitigating control.

[Tristan Cacqueray (tristan-cacqueray)]

I agree with #7, this seems like a C1 type of bug. Since it's already mentioned publicly, is it ok to switch this to public ?

[Charles Neill (charles-neill)]

Any new thoughts on this? Should I reach out to MITRE about getting a CVE assigned?

[Jeremy Stanley (fungi)]

Anyone's welcome to request CVE assignment for a bug, though it looks like the OpenStack VMT considers this impractical to exploit and so doesn't plan to issue an advisory for it.

Before possibly switching to a public bug report, I'm subscribing the security team reviewers for a second opinion.

[Robert Clark (robert-clark)]

I'm not sure how to progress on this, Ccneil maybe we can discuss on IM.

[Tristan Cacqueray (tristan-cacqueray)]

Robert, any progress on this issue?

[Robert Clark (robert-clark)]

I'm happy enough to class it as C1 at this point.

@charles-neill - we can create an OSSN for this instead, would that work for you?

[Charles Neill (charles-neill)]

+1 Sounds good. Whatever everyone feels is the best for the community. Sorry for the radio silence; I have been moved off the OSIC project and onto other things, so haven't been keeping up with my Launchpad issues.

[Tristan Cacqueray (tristan-cacqueray)]

Switched to public security, closed the OSSA task and added an OSSN task based on above comments.

[Luke Hinds (lhinds)]

Just read the thread as I have had access issues before preventing me from viewing this OSSN.

So from what I can see, there are no mitigating issues that an operator can take to de-risk any given threat scenario, coupled with the VMT seeing this as impractical to exploit.

I am not sure we can really do much with this as an OSSN. If an OSSN goes out without a concrete 'Recommended Actions' section, it can result in a lot of concerned and confused ops replying as they are not sure what to do with the information.

Unless someone strongly sees the opposite, I would also recommend a 'Won't Fix' for for an OSSN.

[Luke Hinds (lhinds)]

mitigating issues/mitigating actions

[Charles Neill (charles-neill)]

The main thing that I think would help is updating the documentation to include more reasonable examples. As stated above, I think the examples provided in the docs increase the risk of someone storing sensitive/easily guessed information where it could be exposed by this vector.

[Robert Clark (robert-clark)]

Replying by email because my 2FA has broken and I can't log into LP.

I agree that documentation enhancements would be useful. Luke we could
consider rolling this enhanced guidance into an OSSN but I share your
concerns about posting something without a clear fix/workaround.

-Rob

On Wed, May 24, 2017 at 3:58 PM, Charles Neill <email address hidden>
wrote:

> The main thing that I think would help is updating the documentation to
> include more reasonable examples. As stated above, I think the examples
> provided in the docs increase the risk of someone storing
> sensitive/easily guessed information where it could be exposed by this
> vector.
>
> --
> You received this bug notification because you are a member of OSSG
> CoreSec, which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1618615
>
> Title:
> Potential information disclosure in EC2 "credentials"
>
> Status in OpenStack Identity (keystone):
> New
> Status in OpenStack Security Advisory:
> Won't Fix
> Status in OpenStack Security Notes:
> New
>
> Bug description:
> When creating a "credential" in Keystone, instead of using
> uuid.uuid4() like in most places to generate a unique identifier, the
> id is created from the SHA256 hash value of whatever is passed in as
> the "access" key in the POST request (Code here:
> https://github.com/openstack/keystone/blob/master/keystone/
> credential/controllers.py#L36-L60)
>
> ===== EXAMPLE REQUEST =====
>
> POST /v3/credentials HTTP/1.1
> Host: [ENDPOINT]
> X-Auth-Token: [ADMIN TOKEN]
> Content-Length: 231
> Content-Type: application/json
>
> {
> "credential": {
> "blob": "{\"access\":\"<script>alert(
> 2)</script>\",\"secret\":\"secretKey\"}",
> "project_id": "12345",
> "type": "ec2",
> "user_id": "12345"
> }
> }
>
> HTTP/1.1 201 Created
> Date: Tue, 30 Aug 2016 19:14:54 GMT
> Server: Apache/2.4.7 (Ubuntu)
> Vary: X-Auth-Token
> Content-Length: 383
> Content-Type: application/json
>
> {"credential": {"user_id": "12345", "links": {"self":
> "[ENDPOINT]/v3/credentials/141ce7a938b5973dd71c90bcdd7e40
> 97317ee7374259cf6d8774fdfd86c1f8ea"},
> "blob":
> "{\"access\":\"<script>alert(2)</script>\",\"secret\":\"secretKey\"}",
> "project_id": "12345", "type": "ec2", "id":
> "141ce7a938b5973dd71c90bcdd7e4097317ee7374259cf6d8774fdfd86c1f8ea"}}
>
> ===== /EXAMPLE =====
>
> The id from the example above is
> "141ce7a938b5973dd71c90bcdd7e4097317ee7374259cf6d8774fdfd86c1f8ea",
> which is the same as the SHA256 value of "<script>alert(2)</script>"
> (you can test this with `echo -n "<script>alert(2)</script>" | openssl
> dgst -sha256` on *nix)
>
> The documentation here seems to show MD5s and possibly tenant IDs used
> as "access" values: http://developer.openstack.org/api-
> ref/identity/v3/?expanded=assign-role-to-user-on-projects-owned-by-
> domain-detail,create-policy-detail,show-credential-details-detail
> ,list-credentials-detail,create-credential-detail#list-credentials
>
> Bruteforcing an actual MD5 isn't a huge secur...

[Luke Hinds (lhinds)]

will add a docs bug for this issue.

[Morgan Fainberg (mdrnstm)]

This is a doc bug as per comments above.