PCI-DSS account lock out DoS and account UUID lookup oracle

Bug #1688137 reported by Samuel de Medeiros Queiroz
266
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Medium
Unassigned
OpenStack Security Advisory
Undecided
Unassigned

Bug Description

This relates to PCI DSS features added in the Newton release.
The involved PCI DSS requirements are 8.1.6 and 8.1.7, as described below:

8.1.6 Limit repeated access attempts by locking out the user ID after not more than six attempts.
8.1.7 Set the lockout duration to a minimum of 30 minutes or until an administrator enables the user ID.

The options lockout_failure_attempts and lockout_duration implement those behaviors, respectively.

If those options are enabled in the keystone configuration file, for example:

[security_compliance]
# Setting the account lockout threshold
lockout_failure_attempts = 2
lockout_duration = 10

All users in the cloud get exposed and can be subject of an attack.

The attacker could lock out an user account by:

1) Try to auth on a user's behalf:

POST /v3/auth/tokens

{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "sam",
          "domain": { "id": "default" },
          "password": "fake_password"
        }
      }
    }
  }
}

And after lockout_failure_attempts attempts, as the password is wrong (as the attacker do not know it), the server returns:

{
  "error": {
    "code": 401,
    "title": "Unauthorized",
    "message": "The account is locked for user: 94ab353983174b04955fc9842779b085."
  }
}

And now the attacker even know the user's ID.

OR

2) Try to change a user's password on their behalf (would need to know the user's ID):

POST /v3/users/<user_id>/password
{
 "user": {
  "original_password": "fake_password",
  "password": "new_password"
 }
}

As the original password is wrong (as the attacker do not know it), after lockout_failure_attempts attempts that user account get locked out by lockout_duration.

For both 1) and 2), before lockout_failure_attempts attempts, you get:

{
  "error": {
    "code": 401,
    "title": "Unauthorized",
    "message": "The request you have made requires authentication."
  }
}

After lockout_failure_attempts attempts, you get:

{
  "error": {
    "code": 401,
    "title": "Unauthorized",
    "message": "The account is locked for user: 94ab353983174b04955fc9842779b085."
  }
}

These approaches can be used by an attacker to lock out users indefinitely by locking out users again and again after lockout_duration has passed.

Revision history for this message
Samuel de Medeiros Queiroz (samueldmq) wrote :

This can get even worse. An attacker can simply use a user name and user's domain name to call /v3/auth/tokens on their behalf and get their account locked.

For example:

POST /v3/auth/tokens

{ "auth": {
    "identity": {
      "methods": ["password"],
      "password": {
        "user": {
          "name": "sam",
          "domain": { "id": "default" },
          "password": "fake_password"
        }
      }
    }
  }
}

And after lockout_failure_attempts attempts, the server returns:

{
  "error": {
    "code": 401,
    "title": "Unauthorized",
    "message": "The account is locked for user: 94ab353983174b04955fc9842779b085."
  }
}

And now the attacker even know the user's ID.

Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

description: updated
Changed in ossa:
status: New → Incomplete
summary: - Attacker may use self-service password reset to lock out users
- indefinitely
+ Attacker may use PCI-DSS 8.1.6 and 8.1.7 to lock out users indefinitely
Revision history for this message
Morgan Fainberg (mdrnstm) wrote : Re: Attacker may use PCI-DSS 8.1.6 and 8.1.7 to lock out users indefinitely

This bug is definitely a leak of information and if the PCI DSS features are enabled, this could lead to user discovery. This bug is a possible DOS for individual users, but does not allow an attacker to perform actions that otherwise could not be done.

Likely the most correct fix is to ensure the "lockout" error is only shown if the password is in-fact valid, eliminating the vector of user discovery. As with any system that has lockouts, it is impossible to determine the difference between a malicious user and a user with a poor memory/forgotten password making many attempts. DOS for the individual user is a known/expected aspeect when lockouts are enabled.

This could be a Class A [0], if the user discovery leak is considered a security vulnerability or a Class D [0] (a bug with security implications and a hardening opportunity). Input from Keystone-Coresec will help to identify the precise class of bug.

[0] https://security.openstack.org/vmt-process.html#incident-report-taxonomy

Revision history for this message
Lance Bragstad (lbragstad) wrote :

These issues are valid, but when I think about them from the perspective of other software there are a couple additional things in play. For example, when using other services if an attacker is trying to log in as me I usually get some sort for notification saying someone is attempting to login from somewhere or that my account has been locked. Most of the time this information is delivered through some sort of recovery contact or address.

In keystone, I believe we emit notifications for things for failed authentication and locked accounts (might need to double check that though). In that case, one possible solution would be to write a consumer that listens for those notifications and implements the recovery notice/steps.

Revision history for this message
Samuel de Medeiros Queiroz (samueldmq) wrote :

I have updated the bug title and description to include what I stated on comment #1, which goes beyond than just using the self-service password API for the attack.

description: updated
Revision history for this message
Samuel de Medeiros Queiroz (samueldmq) wrote :

@Lance, the issue with your proposed possible solution is that we don't have such a consumer service in OpenStack, so we can't expect people have it in their deployments.

Revision history for this message
Lance Bragstad (lbragstad) wrote :

@Sam, correct. Expecting deployments to have that service would be unrealistic. It would be a work around for a deployment susceptible to the issues and want to mitigate out-of-band.

I'd be interested in investigating Morgan's proposal further.

Revision history for this message
Samuel de Medeiros Queiroz (samueldmq) wrote :

I like Morgan's proposal too.

If the attacker do not know it's working (since there is no message saying the user has been locked out), they are unlikely to continue.

Even if they do (they might know for sure that a given username exists), it will be up to the deployer to detect that a username has been under constant attack (we need to make sure we communicate well in the logs) and: 1) change the username or 2) ignore the lockout for that specific username until the attacker gets bored.

This can be combined with blacklisting the IPs from attackers, however I do not know how complex it can be to add such an automatic blacklisting mechanism into deployments.

Revision history for this message
Lance Bragstad (lbragstad) wrote :

I think our first course action is to implement Morgan's suggestion, where user information is only emitted if the password is correct.

Changed in keystone:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Samuel de Medeiros Queiroz (samueldmq) wrote :

I am working on a patch for this bug and I have a question: is it okay to keep emitting a CADF notification with reason "The account is locked for user: <user_id>" ?

I assume it is, since that is not a message for final users (unless there is a system somewhere consuming it and giving it to users, but that is a different conversation and workflow).

Revision history for this message
Lance Bragstad (lbragstad) wrote :

I would agree - I think emitting notifications in this case is fine.

Revision history for this message
Samuel de Medeiros Queiroz (samueldmq) wrote :

This patch fixes the issue with the solution proposed by Morgan, which had agreement from Lance and myself.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

It seems like this warrants an advisory (class A according to VMT's taxonomy: https://security.openstack.org/vmt-process.html#incident-report-taxonomy).

@keystone-coresec, please review proposed patch in comment #12.

Is there a documented manual procedure to unlock accounts?

Revision history for this message
Jeremy Stanley (fungi) wrote :

Some interesting alternatives were floated in a NIST SP 800-63-3 update thread on the crypto ML this week: http://www.metzdowd.com/pipermail/cryptography/2017-August/032640.html (worth a read for anyone with their heads in this space currently).

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Samuel, would you mind formatting the patch you proposed in #12 according to https://security.openstack.org/#how-to-propose-and-review-a-security-patch

Revision history for this message
Gage Hugo (gagehugo) wrote :

Formatted the change from comment #12. I had to make an adjustment however, as a "reason" field was added to the audit notifications since this was reported I believe, which caused the notification to send a failure with no reason (Unauthorized) rather than AccountLocked. Please take a look to make sure this is correct.

The evasive-mode lockout from http://www.metzdowd.com/pipermail/cryptography/2017-August/032640.html was an interesting idea, perhaps that could be investigated in the future.

Revision history for this message
Jeremy Stanley (fungi) wrote :

In keeping with recent OpenStack vulnerability management policy changes, no report should remain under private embargo for more than 90 days. Because this report predates the change in policy, the deadline for public disclosure is being set to 90 days from today. If the report is not resolved within the next 90 days, it will revert to our public workflow as of 2020-05-27. Please see http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012721.html for further details.

description: updated
Revision history for this message
Colleen Murphy (krinkle) wrote :

This seems to still be valid but the proposed patch doesn't apply any more, can someone update the patch and can we move forward on this before this is disclosed?

Revision history for this message
Gage Hugo (gagehugo) wrote :

Update patch against master.

Revision history for this message
Colleen Murphy (krinkle) wrote :

Patch in comment 19 lgtm

Jeremy Stanley (fungi)
description: updated
Revision history for this message
Jeremy Stanley (fungi) wrote :

The embargo for this report has expired and is now lifted, so it's acceptable to discuss further in public.

description: updated
information type: Private Security → Public Security
Jeremy Stanley (fungi)
summary: - Attacker may use PCI-DSS 8.1.6 and 8.1.7 to lock out users indefinitely
+ PCI-DSS account lock out DoS and account UUID lookup oracle
Revision history for this message
Jeremy Stanley (fungi) wrote :

So just to summarize, this report covers three possible vulnerabilities related to the PCI-DSS account lock out feature:

1. If someone can guess a username they can prevent that user from authenticating by repeatedly attempting to log in with an incorrect credential.

2. Someone can identify valid usernames by trying to log in with candidate strings with invalid passwords until the lock out is reached, at which point the change in API response confirms the existence of that user.

3. The lock out response can be used as an oracle to determine the UUID matching any known or guessed username.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to keystone (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/759940

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to keystone (master)

Reviewed: https://review.opendev.org/c/openstack/keystone/+/759940
Committed: https://opendev.org/openstack/keystone/commit/ac2631ae33445877094cdae796fbcdce8833a626
Submitter: "Zuul (22348)"
Branch: master

commit ac2631ae33445877094cdae796fbcdce8833a626
Author: Gage Hugo <email address hidden>
Date: Tue Oct 27 15:22:04 2020 -0500

    Hide AccountLocked exception from end users

    This change hides the AccountLocked exception from being returned
    to the end user to hide sensitive information that a potential
    malicious person could gain insight from.

    The notification handler catches the AccountLocked exception as
    before, but after sending the audit notification, it instead
    bubbles up Unauthorized rather than AccountLocked.

    Co-Authored-By: Samuel de Medeiros Queiroz <email address hidden>

    Change-Id: Id51241989b22c52810391f3e8e1cadbf8613d873
    Related-Bug: #1688137

Revision history for this message
Jeremy Stanley (fungi) wrote :

It looks like the change which merged to master last week addresses potential vulnerabilities #2 and #3 from comment #22. Is there any chance for that to be backported to supported stable branches?

As for potential vulnerability #1, I don't really see a viable way to address that, it's the intent of the feature that too many failed logins lock the account. If a deployment considers that feature problematic, they should disable it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to keystone (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/keystone/+/790440

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to keystone (stable/victoria)

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/keystone/+/790442

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to keystone (stable/ussuri)

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/keystone/+/790443

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to keystone (stable/train)

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/keystone/+/790444

Revision history for this message
Gage Hugo (gagehugo) wrote :
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to keystone (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/keystone/+/790440
Committed: https://opendev.org/openstack/keystone/commit/f510c806de3e20cdedd55291cd58dafa59398bec
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f510c806de3e20cdedd55291cd58dafa59398bec
Author: Gage Hugo <email address hidden>
Date: Tue Oct 27 15:22:04 2020 -0500

    Hide AccountLocked exception from end users

    This change hides the AccountLocked exception from being returned
    to the end user to hide sensitive information that a potential
    malicious person could gain insight from.

    The notification handler catches the AccountLocked exception as
    before, but after sending the audit notification, it instead
    bubbles up Unauthorized rather than AccountLocked.

    Co-Authored-By: Samuel de Medeiros Queiroz <email address hidden>

    Change-Id: Id51241989b22c52810391f3e8e1cadbf8613d873
    Related-Bug: #1688137
    (cherry picked from commit ac2631ae33445877094cdae796fbcdce8833a626)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to keystone (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/keystone/+/790442
Committed: https://opendev.org/openstack/keystone/commit/4649fe6bfc749ab48ec1905ca4dc2fc667914021
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 4649fe6bfc749ab48ec1905ca4dc2fc667914021
Author: Gage Hugo <email address hidden>
Date: Tue Oct 27 15:22:04 2020 -0500

    Hide AccountLocked exception from end users

    This change hides the AccountLocked exception from being returned
    to the end user to hide sensitive information that a potential
    malicious person could gain insight from.

    The notification handler catches the AccountLocked exception as
    before, but after sending the audit notification, it instead
    bubbles up Unauthorized rather than AccountLocked.

    Co-Authored-By: Samuel de Medeiros Queiroz <email address hidden>

    Change-Id: Id51241989b22c52810391f3e8e1cadbf8613d873
    Related-Bug: #1688137
    (cherry picked from commit ac2631ae33445877094cdae796fbcdce8833a626)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to keystone (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/keystone/+/790443
Committed: https://opendev.org/openstack/keystone/commit/8ab4eb27be4c13c9bab2b3ea700f00a190521bf8
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 8ab4eb27be4c13c9bab2b3ea700f00a190521bf8
Author: Gage Hugo <email address hidden>
Date: Tue Oct 27 15:22:04 2020 -0500

    Hide AccountLocked exception from end users

    This change hides the AccountLocked exception from being returned
    to the end user to hide sensitive information that a potential
    malicious person could gain insight from.

    The notification handler catches the AccountLocked exception as
    before, but after sending the audit notification, it instead
    bubbles up Unauthorized rather than AccountLocked.

    Co-Authored-By: Samuel de Medeiros Queiroz <email address hidden>

    Change-Id: Id51241989b22c52810391f3e8e1cadbf8613d873
    Related-Bug: #1688137
    (cherry picked from commit ac2631ae33445877094cdae796fbcdce8833a626)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to keystone (stable/train)

Reviewed: https://review.opendev.org/c/openstack/keystone/+/790444
Committed: https://opendev.org/openstack/keystone/commit/1b573ae7d1c20e0ebfbde79bbe7538a09589c75d
Submitter: "Zuul (22348)"
Branch: stable/train

commit 1b573ae7d1c20e0ebfbde79bbe7538a09589c75d
Author: Gage Hugo <email address hidden>
Date: Tue Oct 27 15:22:04 2020 -0500

    Hide AccountLocked exception from end users

    This change hides the AccountLocked exception from being returned
    to the end user to hide sensitive information that a potential
    malicious person could gain insight from.

    The notification handler catches the AccountLocked exception as
    before, but after sending the audit notification, it instead
    bubbles up Unauthorized rather than AccountLocked.

    Co-Authored-By: Samuel de Medeiros Queiroz <email address hidden>

    Change-Id: Id51241989b22c52810391f3e8e1cadbf8613d873
    Related-Bug: #1688137
    (cherry picked from commit ac2631ae33445877094cdae796fbcdce8833a626)

tags: added: in-stable-train
Revision history for this message
Jeremy Stanley (fungi) wrote :

A fix for the account name and UUID oracles has merged with backports applied as far back as stable/train (so definitely covering all officially maintained branches at this point). We should probably issue a security advisory covering these points.

However, the other concern raised in this report is essentially with the intent of PCI-DSS controls 8.1.6 and 8.1.7, which I think should not be treated as a bug (if you don't want someone to be able to lock our another user's account by repeatedly failing to log into it, don't enable that feature in Keystone). I think the bug report should be retitled to focus on the oracles, which were certainly unintended behaviors detrimental to account security.

To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Duplicates of this bug

Other bug subscribers