[OSSA-2020-003] Keystone doesn't check signature TTL of the EC2 credential auth method (CVE-2020-12692)

[kay]

AWS Signature V4 has a limited TTL for a token signature, used to perform an authenticated request, usually it is 5 minutes. If there is a MITM possible, then an attacker can use a sniffed header only within 5 minutes.

Keystone doesn't have a signature TTL check, and if an attacker can sniff an auth header, this header can be used an unlimited number of times to reissue an openstack token.

I have an https://github.com/kayrus/ec2auth tool to auth against keystone using ec2 credentials. If you set a timestamp (https://godoc.org/github.com/gophercloud/gophercloud/openstack/identity/v3/extensions/ec2tokens#AuthOptions) to "time.Time{}" here: https://github.com/kayrus/ec2auth/blob/master/pkg/main.go#L40, keystone will identify this token as a valid one and return a valid openstack token.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

[Colleen Murphy (krinkle)]

Confirmed that keystone does not verify the Timestamp attribute of a /v3/ec2tokens request. Can you add a link to the AWS documentation explaining the TTL? https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html seems to indicate it's 15 minutes but I am not sure if that is consistent or configurable.

[kay (kay-diam)]

Hi Colleen. Thanks for looking into this issue. The link you provided is correct. The "requests are valid within 15 minutes of the timestamp in the request" statement is also correct for Amazon API. The purpose of this bugreport is to introduce a configuration option in keystone with a default value, e.g. 15 minutes. Keystone should use this configuration option to validate EC2 signature TTL.

[Colleen Murphy (krinkle)]

Attaching patch to add an auth_ttl config option to validate against.

[kay (kay-diam)]

Hi Colleen, thanks for a patch.

I noticed that you're referencing to "timestamp = credentials['params'].get('Timestamp')", which is not correct.

Currently timestamp is fetched from the headers' "X-Amz-Date": https://github.com/openstack/python-keystoneclient/blob/a9d01eef93eef576bcecabf54e86c3a7a49651c8/keystoneclient/contrib/ec2/utils.py#L197
If there is no headers "X-Amz-Date" element, logic should fallback to params["X-Amz-Date"].

[Colleen Murphy (krinkle)]

Thanks for the review. It looks like our unit tests and API examples were designed for AWS Signature v2:

https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
https://opendev.org/openstack/keystone/src/branch/master/api-ref/source/v2-ext/samples/OS-KSEC2/authenticate-request.json

I've updated the patch to check for both v2 and v4 versions.

[Gage Hugo (gagehugo)]

The revised ps looks fine, thanks cmurphy.

[Lance Bragstad (lbragstad)]

+2 to the patch set proposed in comment #6.

[Colleen Murphy (krinkle)]

Please re-review the newly attached patch. Running the unit tests locally revealed an error in the handling of the timestamp parameter in the request body. Will attach a diff to show what I changed. This now passes the unit tests for me.

[Colleen Murphy (krinkle)]

Added diff

[Colleen Murphy (krinkle)]

Is a CVE being requested for this? How can we move this forward?

[Gage Hugo (gagehugo)]

Thanks for the diff, looks good to me.

With the attack vector here being sniffing the auth header/no TTL, then using the header to reissue openstack tokens, this appears to be a Class A.[0] If this is agreed to be the case, I can work on drafting up an impact description here.

[0] https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[Jeremy Stanley (fungi)]

I agree that the failure to check the TTL qualifies this as a class AS report for a vulnerability and we should go ahead with an impact description and requesting a CVE assignment (basically Keystone's not following through on one of the security measures it claims to implement). The requirement that the attacker be able to obtain a copy of the victim's token, however, seems like it presents a significant mitigating factor which reduces the risk of dealing with this report in public. How do folks feel about switching this report to a public workflow? Are tokens generally "easy" to steal in a typical deployment/environment?

[Jeremy Stanley (fungi)]

Sorry, that should read "...as a class A report for a vulnerability..." (we don't have any "AS" classification).

[Colleen Murphy (krinkle)]

> The requirement that the attacker be able to obtain a copy of the victim's token, however, seems like it presents a significant mitigating factor which reduces the risk of dealing with this report in public. How do folks feel about switching this report to a public workflow? Are tokens generally "easy" to steal in a typical deployment/environment?

In a typical production environment, all traffic should use HTTPS between all involved services and I would not expect there to be any reason to write the AWS token to disk, so I would tend to agree that the bar for stealing this token is high. kay may be able to give more insight about the typical workflow for this feature.

[kay (kay-diam)]

Agree. Signature can be reused only within a MITM attack, or when someone shared a screen with a debug log.

[Jeremy Stanley (fungi)]

Thanks for the feedback, I've switched this to our public workflow effectively ending the embargo. The fixes/backports and advisory for it can be handled publicly in code review.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/724124

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724124
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=ab89ea749013e7f2c46260f68504f5687763e019
Submitter: Zuul
Branch: master

commit ab89ea749013e7f2c46260f68504f5687763e019
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/724746

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724746
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=8d5becbe4b463f6a5a24a1929dd0f48dab6ae027
Submitter: Zuul
Branch: stable/ussuri

commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/724954

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/725069

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/724954
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=e3f65d6fbcd18032a8ad3dfa3aaded264a282158
Submitter: Zuul
Branch: stable/train

commit e3f65d6fbcd18032a8ad3dfa3aaded264a282158
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Conflicts due to six removal in e2d83ae9:
            keystone/api/_shared/EC2_S3_Resource.py
            keystone/tests/unit/test_contrib_ec2_core.py

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)
    (cherry picked from commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/725069
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=1ef3828516c1b87a8ca84acca73ec593b0b8591d
Submitter: Zuul
Branch: stable/stein

commit 1ef3828516c1b87a8ca84acca73ec593b0b8591d
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Conflicts due to six removal in e2d83ae9:
            keystone/api/_shared/EC2_S3_Resource.py
            keystone/tests/unit/test_contrib_ec2_core.py

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)
    (cherry picked from commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027)
    (cherry picked from commit e3f65d6fbcd18032a8ad3dfa3aaded264a282158)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/725385

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/725879
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=e084cbd224faadb97bc9e1e5e131466416a295aa
Submitter: Zuul
Branch: master

commit e084cbd224faadb97bc9e1e5e131466416a295aa
Author: Gage Hugo <email address hidden>
Date: Wed May 6 09:46:08 2020 -0500

    Add OSSA-2020-003 (CVE Pending)

    Related-Bug: #1872737
    Change-Id: I51b2199d573d0abf088befa8eb504486708f8e70

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/726025

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/726400

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/725385
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=35f09e2b7c00e03cd1d52a2337b51be38dd79480
Submitter: Zuul
Branch: stable/rocky

commit 35f09e2b7c00e03cd1d52a2337b51be38dd79480
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Conflicts due to six removal in e2d83ae9:
            keystone/api/_shared/EC2_S3_Resource.py
            keystone/tests/unit/test_contrib_ec2_core.py

    Conflicts due to flask reorg:
            keystone/api/_shared/EC2_S3_Resource.py

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)
    (cherry picked from commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027)
    (cherry picked from commit e3f65d6fbcd18032a8ad3dfa3aaded264a282158)
    (cherry picked from commit 1ef3828516c1b87a8ca84acca73ec593b0b8591d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/726400
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=b25739fa9605cc54bc98325c2a92360ba702e8d8
Submitter: Zuul
Branch: stable/queens

commit b25739fa9605cc54bc98325c2a92360ba702e8d8
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Conflicts due to six removal in e2d83ae9:
            keystone/api/_shared/EC2_S3_Resource.py
            keystone/tests/unit/test_contrib_ec2_core.py

    Conflicts due to v2.0 API testing in stable/queens. The v2.0 tests were
    removed in Rocky but in earlier releases we tested similar functionality
    between v3 and v2.0. This conflict was resolved by porting the timestamp
    to the v2.0 API test:
            keystone/tests/unit/test_contrib_ec2_core.py

    Conflicts due to flask reorg:
            keystone/api/_shared/EC2_S3_Resource.py

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)
    (cherry picked from commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027)
    (cherry picked from commit e3f65d6fbcd18032a8ad3dfa3aaded264a282158)
    (cherry picked from commit 1ef3828516c1b87a8ca84acca73ec593b0b8591d)
    (cherry picked from commit 35f09e2b7c00e03cd1d52a2337b51be38dd79480)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/726025
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=1c1cf556f81058a63ea0bd5138540b0e6795f7a0
Submitter: Zuul
Branch: stable/pike

commit 1c1cf556f81058a63ea0bd5138540b0e6795f7a0
Author: Colleen Murphy <email address hidden>
Date: Thu Apr 16 17:05:43 2020 -0700

    Check timestamp of signed EC2 token request

    EC2 token requests contain a signature that signs the entire request,
    including the access timestamp. While the signature is checked, the
    timestamp is not, and so these signed requests remain valid
    indefinitely, leaving the token API vulnerable to replay attacks. This
    change introduces a configurable TTL for signed token requests and
    ensures that the timestamp is actually validated against it.

    The check will work for either an AWS Signature v1/v2 'Timestamp'
    parameter[1] or the AWS Signature v4 'X-Aws-Date' header or
    parameter[2].

    Although this technically adds a new feature and the default value of
    the feature changes behavior, this change is required to protect
    credential holders and therefore must be backported to all supported
    branches.

    [1] https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html
    [2] https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html

    Conflicts due to six removal in e2d83ae9:
            keystone/api/_shared/EC2_S3_Resource.py
            keystone/tests/unit/test_contrib_ec2_core.py

    Conflicts due to v2.0 API testing in stable/queens. The v2.0 tests were
    removed in Rocky but in earlier releases we tested similar functionality
    between v3 and v2.0. This conflict was resolved by porting the timestamp
    to the v2.0 API test:
            keystone/tests/unit/test_contrib_ec2_core.py

    Conflicts due to flask reorg:
            keystone/api/_shared/EC2_S3_Resource.py

    Change-Id: Idb10267338b4204b435df233c636046a1ce5711f
    Closes-bug: #1872737
    (cherry picked from commit ab89ea749013e7f2c46260f68504f5687763e019)
    (cherry picked from commit 8d5becbe4b463f6a5a24a1929dd0f48dab6ae027)
    (cherry picked from commit e3f65d6fbcd18032a8ad3dfa3aaded264a282158)
    (cherry picked from commit 1ef3828516c1b87a8ca84acca73ec593b0b8591d)
    (cherry picked from commit 35f09e2b7c00e03cd1d52a2337b51be38dd79480)
    (cherry picked from commit d6f1006dd0ca8f7999f32133d348fcec8f2299d3)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone pike-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone rocky-eol release.