[OSSA-2020-003] Keystone doesn't check signature TTL of the EC2 credential auth method (CVE-2020-12692)

Bug #1872737 reported by kay
262
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Medium
Colleen Murphy
OpenStack Security Advisory
Fix Released
Undecided
Gage Hugo

Bug Description

AWS Signature V4 has a limited TTL for a token signature, used to perform an authenticated request, usually it is 5 minutes. If there is a MITM possible, then an attacker can use a sniffed header only within 5 minutes.

Keystone doesn't have a signature TTL check, and if an attacker can sniff an auth header, this header can be used an unlimited number of times to reissue an openstack token.

I have an https://github.com/kayrus/ec2auth tool to auth against keystone using ec2 credentials. If you set a timestamp (https://godoc.org/github.com/gophercloud/gophercloud/openstack/identity/v3/extensions/ec2tokens#AuthOptions) to "time.Time{}" here: https://github.com/kayrus/ec2auth/blob/master/pkg/main.go#L40, keystone will identify this token as a valid one and return a valid openstack token.

CVE References

kay (kay-diam)
description: updated
Revision history for this message
Jeremy Stanley (fungi) wrote :

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

description: updated
Changed in ossa:
status: New → Incomplete
description: updated
Revision history for this message
Colleen Murphy (krinkle) wrote :

Confirmed that keystone does not verify the Timestamp attribute of a /v3/ec2tokens request. Can you add a link to the AWS documentation explaining the TTL? https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html seems to indicate it's 15 minutes but I am not sure if that is consistent or configurable.

Revision history for this message
kay (kay-diam) wrote :

Hi Colleen. Thanks for looking into this issue. The link you provided is correct. The "requests are valid within 15 minutes of the timestamp in the request" statement is also correct for Amazon API. The purpose of this bugreport is to introduce a configuration option in keystone with a default value, e.g. 15 minutes. Keystone should use this configuration option to validate EC2 signature TTL.

Revision history for this message
Colleen Murphy (krinkle) wrote :

Attaching patch to add an auth_ttl config option to validate against.

Revision history for this message
kay (kay-diam) wrote :

Hi Colleen, thanks for a patch.

I noticed that you're referencing to "timestamp = credentials['params'].get('Timestamp')", which is not correct.

Currently timestamp is fetched from the headers' "X-Amz-Date": https://github.com/openstack/python-keystoneclient/blob/a9d01eef93eef576bcecabf54e86c3a7a49651c8/keystoneclient/contrib/ec2/utils.py#L197
If there is no headers "X-Amz-Date" element, logic should fallback to params["X-Amz-Date"].

Revision history for this message
Colleen Murphy (krinkle) wrote :

Thanks for the review. It looks like our unit tests and API examples were designed for AWS Signature v2:

https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
https://opendev.org/openstack/keystone/src/branch/master/api-ref/source/v2-ext/samples/OS-KSEC2/authenticate-request.json

I've updated the patch to check for both v2 and v4 versions.

Changed in keystone:
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Colleen Murphy (krinkle)
Colleen Murphy (krinkle)
Changed in keystone:
milestone: none → ussuri-rc1
Revision history for this message
Gage Hugo (gagehugo) wrote :

The revised ps looks fine, thanks cmurphy.

Revision history for this message
Lance Bragstad (lbragstad) wrote :

+2 to the patch set proposed in comment #6.

Revision history for this message
Colleen Murphy (krinkle) wrote :

Please re-review the newly attached patch. Running the unit tests locally revealed an error in the handling of the timestamp parameter in the request body. Will attach a diff to show what I changed. This now passes the unit tests for me.

Revision history for this message
Colleen Murphy (krinkle) wrote :

Added diff

Revision history for this message
Colleen Murphy (krinkle) wrote :

Is a CVE being requested for this? How can we move this forward?

Revision history for this message
Gage Hugo (gagehugo) wrote :

Thanks for the diff, looks good to me.

With the attack vector here being sniffing the auth header/no TTL, then using the header to reissue openstack tokens, this appears to be a Class A.[0] If this is agreed to be the case, I can work on drafting up an impact description here.

[0] https://security.openstack.org/vmt-process.html#incident-report-taxonomy

Revision history for this message
Jeremy Stanley (fungi) wrote :

I agree that the failure to check the TTL qualifies this as a class AS report for a vulnerability and we should go ahead with an impact description and requesting a CVE assignment (basically Keystone's not following through on one of the security measures it claims to implement). The requirement that the attacker be able to obtain a copy of the victim's token, however, seems like it presents a significant mitigating factor which reduces the risk of dealing with this report in public. How do folks feel about switching this report to a public workflow? Are tokens generally "easy" to steal in a typical deployment/environment?

Revision history for this message
Jeremy Stanley (fungi) wrote :

Sorry, that should read "...as a class A report for a vulnerability..." (we don't have any "AS" classification).

Revision history for this message
Colleen Murphy (krinkle) wrote :

> The requirement that the attacker be able to obtain a copy of the victim's token, however, seems like it presents a significant mitigating factor which reduces the risk of dealing with this report in public. How do folks feel about switching this report to a public workflow? Are tokens generally "easy" to steal in a typical deployment/environment?

In a typical production environment, all traffic should use HTTPS between all involved services and I would not expect there to be any reason to write the AWS token to disk, so I would tend to agree that the bar for stealing this token is high. kay may be able to give more insight about the typical workflow for this feature.

Revision history for this message
kay (kay-diam) wrote :

Agree. Signature can be reused only within a MITM attack, or when someone shared a screen with a debug log.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Thanks for the feedback, I've switched this to our public workflow effectively ending the embargo. The fixes/backports and advisory for it can be handled publicly in code review.

description: updated
information type: Private Security → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.opendev.org/724124

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.opendev.org/724124
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=ab89ea749013e7f2c46260f68504f5687763e019
Submitter: Zuul
Branch: master

commit ab89ea749013e7f2c46260f68504f5687763e019
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737

Changed in keystone:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/724746

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/ussuri)

Reviewed: https://review.opendev.org/724746
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=8d5becbe4b463f6a5a24a1929dd0f48dab6ae027
Submitter: Zuul
Branch: stable/ussuri

commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/724954

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/725069

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/train)

Reviewed: https://review.opendev.org/724954
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=e3f65d6fbcd18032a8ad3dfa3aaded264a282158
Submitter: Zuul
Branch: stable/train

commit e3f65d6fbcd18032a8ad3dfa3aaded264a282158
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Conflicts due to six removal in e2d83ae9:
            keystone/api/_shared/EC2_S3_Resource.py
            keystone/tests/unit/test_contrib_ec2_core.py

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)
    (cherry picked from commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/stein)

Reviewed: https://review.opendev.org/725069
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=1ef3828516c1b87a8ca84acca73ec593b0b8591d
Submitter: Zuul
Branch: stable/stein

commit 1ef3828516c1b87a8ca84acca73ec593b0b8591d
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Conflicts due to six removal in e2d83ae9:
            keystone/api/_shared/EC2_S3_Resource.py
            keystone/tests/unit/test_contrib_ec2_core.py

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)
    (cherry picked from commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027)
    (cherry picked from commit e3f65d6fbcd18032a8ad3dfa3aaded264a282158)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/725385

Gage Hugo (gagehugo)
Changed in ossa:
assignee: nobody → Gage Hugo (gagehugo)
Gage Hugo (gagehugo)
summary: - Keystone doesn't check signature TTL of the EC2 credential auth method
+ [OSSA-2020-003] Keystone doesn't check signature TTL of the EC2
+ credential auth method
summary: [OSSA-2020-003] Keystone doesn't check signature TTL of the EC2
- credential auth method
+ credential auth method (CVE PENDING)
Jeremy Stanley (fungi)
Changed in ossa:
status: Incomplete → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to ossa (master)

Reviewed: https://review.opendev.org/725879
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=e084cbd224faadb97bc9e1e5e131466416a295aa
Submitter: Zuul
Branch: master

commit e084cbd224faadb97bc9e1e5e131466416a295aa
Author: Gage Hugo <email address hidden>
Date: Wed May 6 09:46:08 2020 -0500

    Add OSSA-2020-003 (CVE Pending)

    Related-Bug: #1872737
    Change-Id: I51b2199d573d0abf088befa8eb504486708f8e70

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/726025

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/726400

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/rocky)

Reviewed: https://review.opendev.org/725385
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=35f09e2b7c00e03cd1d52a2337b51be38dd79480
Submitter: Zuul
Branch: stable/rocky

commit 35f09e2b7c00e03cd1d52a2337b51be38dd79480
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Conflicts due to six removal in e2d83ae9:
            keystone/api/_shared/EC2_S3_Resource.py
            keystone/tests/unit/test_contrib_ec2_core.py

    Conflicts due to flask reorg:
            keystone/api/_shared/EC2_S3_Resource.py

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)
    (cherry picked from commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027)
    (cherry picked from commit e3f65d6fbcd18032a8ad3dfa3aaded264a282158)
    (cherry picked from commit 1ef3828516c1b87a8ca84acca73ec593b0b8591d)

tags: added: in-stable-rocky
Gage Hugo (gagehugo)
summary: [OSSA-2020-003] Keystone doesn't check signature TTL of the EC2
- credential auth method (CVE PENDING)
+ credential auth method (CVE-2020-12692)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/queens)

Reviewed: https://review.opendev.org/726400
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=b25739fa9605cc54bc98325c2a92360ba702e8d8
Submitter: Zuul
Branch: stable/queens

commit b25739fa9605cc54bc98325c2a92360ba702e8d8
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Conflicts due to six removal in e2d83ae9:
            keystone/api/_shared/EC2_S3_Resource.py
            keystone/tests/unit/test_contrib_ec2_core.py

    Conflicts due to v2.0 API testing in stable/queens. The v2.0 tests were
    removed in Rocky but in earlier releases we tested similar functionality
    between v3 and v2.0. This conflict was resolved by porting the timestamp
    to the v2.0 API test:
            keystone/tests/unit/test_contrib_ec2_core.py

    Conflicts due to flask reorg:
            keystone/api/_shared/EC2_S3_Resource.py

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)
    (cherry picked from commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027)
    (cherry picked from commit e3f65d6fbcd18032a8ad3dfa3aaded264a282158)
    (cherry picked from commit 1ef3828516c1b87a8ca84acca73ec593b0b8591d)
    (cherry picked from commit 35f09e2b7c00e03cd1d52a2337b51be38dd79480)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/pike)

Reviewed: https://review.opendev.org/726025
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=1c1cf556f81058a63ea0bd5138540b0e6795f7a0
Submitter: Zuul
Branch: stable/pike

commit 1c1cf556f81058a63ea0bd5138540b0e6795f7a0
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Conflicts due to six removal in e2d83ae9:
            keystone/api/_shared/EC2_S3_Resource.py
            keystone/tests/unit/test_contrib_ec2_core.py

    Conflicts due to v2.0 API testing in stable/queens. The v2.0 tests were
    removed in Rocky but in earlier releases we tested similar functionality
    between v3 and v2.0. This conflict was resolved by porting the timestamp
    to the v2.0 API test:
            keystone/tests/unit/test_contrib_ec2_core.py

    Conflicts due to flask reorg:
            keystone/api/_shared/EC2_S3_Resource.py

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)
    (cherry picked from commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027)
    (cherry picked from commit e3f65d6fbcd18032a8ad3dfa3aaded264a282158)
    (cherry picked from commit 1ef3828516c1b87a8ca84acca73ec593b0b8591d)
    (cherry picked from commit 35f09e2b7c00e03cd1d52a2337b51be38dd79480)
    (cherry picked from commit d6f1006dd0ca8f7999f32133d348fcec8f2299d3)

tags: added: in-stable-pike
Jeremy Stanley (fungi)
Changed in ossa:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone pike-eol

This issue was fixed in the openstack/keystone pike-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone queens-eol

This issue was fixed in the openstack/keystone queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone rocky-eol

This issue was fixed in the openstack/keystone rocky-eol release.

To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.