Upgrades from Zed to Antelope may fail due to the password truncation

[Damian Dąbrowski]

Since 2023.1, keystone truncates bcrypt passwords to 54 characters[1] while OSA generates passwords for openstack services with length between 16 and 64 characters[2].

It may cause issues with keystone authentication after upgrade because we recently disabled password updates by default.[3]

Example scenario:
1. User1 was created during Zed release with password containing 64 characters.
2. Password was hashed using all 64 characters(In [4] it is only mentioned that bytes 55-72 are not fully mixed into the resulting hash, but it means they are still used to some extent).
3. Openstack is upgraded to 2023.1(where keystone truncates passwords to 54 chars when hashing).
4. User1 cannot authenticate to keystone because hash was originally created using 64 characters but now only 54 characters are used.

As a solution I recommend:
- Enable ``service_update_password`` during upgrade process to rehash service passwords again(and keep information about it in major upgrades guide until 2024.1).
  Please note that it will only fix passwords managed by openstack-ansible. User passwords containing more than 54 characters will stop working.
  Enabling ``service_update_password`` may also prolong API downtime due to the bug #2023370[5].
- Edit pw-token-gen.py script to generate passwords with length up to 54 characters.

I do not suggest switching to scrypt because:
- We cannot rehash bcrypt passwords anyway
- Bcrypt is still default password_hash_algorithm in keystone

[1] https://review.opendev.org/c/openstack/keystone/+/828595
[2] https://opendev.org/openstack/openstack-ansible/src/commit/d1e30257ae0c818780684fe77e1b34ba4dd0dc40/scripts/pw-token-gen.py#L85
[3] https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/888152
[4] https://passlib.readthedocs.io/en/stable/lib/passlib.hash.bcrypt.html#security-issues
[5] https://bugs.launchpad.net/openstack-ansible/+bug/2023370

[Damian Dąbrowski (damiandabrowski)]

FYI, I proposed patches including a suggested fix(I don't know why they are not listed above):

https://review.opendev.org/c/openstack/openstack-ansible/+/889781
https://review.opendev.org/c/openstack/openstack-ansible/+/889801

[Dmitriy Rabotyagov (noonedeadpunk)]

I think, this is actually a keystone bug that happens during upgrade to 2023.1. And while we can workaround it for CI and service users in OpenStack-Ansible, it's worth to be fixed in Keystone, as that most surely also affects users, which will be unable to login after the upgrade.

While I totally do understand background beneath the path, release note to it is completely misleading and at very least should reflect, that the upgrade is breaking and should be performed with special care in case it is impossible to revert to original behaviour

[David Wilde (dave-wilde)]

Hello,

Would it be possible to test changing line https://opendev.org/openstack/keystone/src/branch/master/keystone/common/password_hashing.py#L71 to read

BCRYPT_MAX_LENGTH = 72

and testing again? I'm wondering if we need to account for the partially mixed characters as well.

Thanks,

/Dave

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/keystone/+/890936

[Dmitriy Rabotyagov (noonedeadpunk)]

Yes, you'r right David, setting `BCRYPT_MAX_LENGTH = 72` is an easy fix. Even setting to 71 would result in invalidly trimmed password.

Though, length should be in bytes, not in string length. So in case 1 symbol takes more then one byte in UTF-8 (like Cyrillic or Chinese), it should be trimmed based on that. Simple example of that:

```
>>> s1 = "%HE"
>>> s2 = "%НЕ"
>>> len(s1)
3
>>> len(s2)
3
>>> len(s1.encode())
3
>>> len(s2.encode())
5
>>>
```

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystone/+/890936
Committed: https://opendev.org/openstack/keystone/commit/6730c761d18aa547998f2add833c13f45f257fe7
Submitter: "Zuul (22348)"
Branch: master

commit 6730c761d18aa547998f2add833c13f45f257fe7
Author: Dmitriy Rabotyagov <email address hidden>
Date: Wed Aug 9 20:41:05 2023 +0200

    Properly trimm bcrypt hashed passwords

    bcrypt hashing algorythm has a limitation on length of passwords it
    can hash on 72 bytes. In [1] a password trimm to 54 symbols has been
    implemented, which resulted in password being invalidated after the
    keystone upgrade, since passwords are trimmed differently by bcrypt
    itself, as well as len(str()) is not always equal to
    len(str().encode()) as trimming should be done based on bytes and not
    string itself.

    With the change we return a byte object from
    `verify_length_and_trunc_password`, so it does not need to
    be encoded afterwards, since we need to strip based on bytes
    rather then on length of the string.

    [1] https://review.opendev.org/c/openstack/keystone/+/828595

    Closes-Bug: #2028809
    Related-Bug: #1901891
    Change-Id: Iea95a3c2df041a0046647b3d3dadead1a6d054d1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/keystone/+/891115

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/keystone/+/891116

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystone/+/891115
Committed: https://opendev.org/openstack/keystone/commit/df54af90d03b14ebcd6e662bc8ece1fc52ea7c1d
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit df54af90d03b14ebcd6e662bc8ece1fc52ea7c1d
Author: Dmitriy Rabotyagov <email address hidden>
Date: Wed Aug 9 20:41:05 2023 +0200

    Properly trimm bcrypt hashed passwords

    bcrypt hashing algorythm has a limitation on length of passwords it
    can hash on 72 bytes. In [1] a password trimm to 54 symbols has been
    implemented, which resulted in password being invalidated after the
    keystone upgrade, since passwords are trimmed differently by bcrypt
    itself, as well as len(str()) is not always equal to
    len(str().encode()) as trimming should be done based on bytes and not
    string itself.

    With the change we return a byte object from
    `verify_length_and_trunc_password`, so it does not need to
    be encoded afterwards, since we need to strip based on bytes
    rather then on length of the string.

    [1] https://review.opendev.org/c/openstack/keystone/+/828595

    Closes-Bug: #2028809
    Related-Bug: #1901891
    Change-Id: Iea95a3c2df041a0046647b3d3dadead1a6d054d1
    (cherry picked from commit 6730c761d18aa547998f2add833c13f45f257fe7)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/891633

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/openstack-ansible/+/891633
Committed: https://opendev.org/openstack/openstack-ansible/commit/f22ad8f715138fc6ccd9da9567373f8d46c33624
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit f22ad8f715138fc6ccd9da9567373f8d46c33624
Author: Dmitriy Rabotyagov <email address hidden>
Date: Wed Aug 16 22:11:09 2023 +0200

    Bump keystone version to unblock upgrade jobs

    Closes-Bug: #2028809
    Change-Id: I70d115e3aaa7a2bd6529ecb89e5c252f4c4036a1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystone/+/891116
Committed: https://opendev.org/openstack/keystone/commit/65f1fb6b4a54386f473369b05c8d10d77fb6710c
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 65f1fb6b4a54386f473369b05c8d10d77fb6710c
Author: Dmitriy Rabotyagov <email address hidden>
Date: Wed Aug 9 20:41:05 2023 +0200

    Properly trimm bcrypt hashed passwords

    bcrypt hashing algorythm has a limitation on length of passwords it
    can hash on 72 bytes. In [1] a password trimm to 54 symbols has been
    implemented, which resulted in password being invalidated after the
    keystone upgrade, since passwords are trimmed differently by bcrypt
    itself, as well as len(str()) is not always equal to
    len(str().encode()) as trimming should be done based on bytes and not
    string itself.

    With the change we return a byte object from
    `verify_length_and_trunc_password`, so it does not need to
    be encoded afterwards, since we need to strip based on bytes
    rather then on length of the string.

    [1] https://review.opendev.org/c/openstack/keystone/+/828595

    Closes-Bug: #2028809
    Related-Bug: #1901891
    Change-Id: Iea95a3c2df041a0046647b3d3dadead1a6d054d1
    (cherry picked from commit 6730c761d18aa547998f2add833c13f45f257fe7)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/keystone/+/892333

[OpenStack Infra (hudson-openstack)]

Change abandoned by "David Wilde <email address hidden>" on branch: stable/yoga
Review: https://review.opendev.org/c/openstack/keystone/+/892333
Reason: Looks like this backport is happening elsewhere

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Damian Dąbrowski <email address hidden>" on branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/889801
Reason: fixed on keystone side https://review.opendev.org/c/openstack/keystone/+/890936

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Damian Dąbrowski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/openstack-ansible/+/889781
Reason: fixed on keystone side https://review.opendev.org/c/openstack/keystone/+/890936

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystone/+/890661
Committed: https://opendev.org/openstack/keystone/commit/7852ca24a4eb86cb271ef7ec5e07f8f9c97f926d
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 7852ca24a4eb86cb271ef7ec5e07f8f9c97f926d
Author: Dave Wilde (d34dh0r53) <email address hidden>
Date: Wed Feb 9 11:28:59 2022 -0600

    Force algo specific maximum length & Properly trimm bcrypt hashed passwords

    This is the squash of 2 patches related to bcrypt hashing settings.

    1.
    Force algo specific maximum length

    The bcrypt algorithm that we use for password hashing silently
    length limits the size of the password that is hashed giving the
    user a false sense of security [0]. This patch adds a check
    in the verify_length_and_trunc_password function for the hash in
    use and updates the max_length accordingly, this will override
    the configured value and log a warning if the password is truncated.

    Conflicts:
    * tox.ini

    [0]: https://passlib.readthedocs.io/en/stable/lib/passlib.hash.bcrypt.html#security-issues

    2.
    Properly trimm bcrypt hashed passwords

    bcrypt hashing algorythm has a limitation on length of passwords it
    can hash on 72 bytes. In [1] a password trimm to 54 symbols has been
    implemented, which resulted in password being invalidated after the
    keystone upgrade, since passwords are trimmed differently by bcrypt
    itself, as well as len(str()) is not always equal to
    len(str().encode()) as trimming should be done based on bytes and not
    string itself.

    With the change we return a byte object from
    `verify_length_and_trunc_password`, so it does not need to
    be encoded afterwards, since we need to strip based on bytes
    rather then on length of the string.

    [1] https://review.opendev.org/c/openstack/keystone/+/828595

    Closes-Bug: #2028809
    Related-Bug: #1901891
    original change id: Iea95a3c2df041a0046647b3d3dadead1a6d054d1
    (cherry picked from commit 6730c761d18aa547998f2add833c13f45f257fe7)
    (cherry picked from commit 65f1fb6b4a54386f473369b05c8d10d77fb6710c)

    Closes-bug: #1901891
    Change-Id: I8d0bb2438b23227b5a66b94af6f8e198084fcd8d
    (cherry picked from commit 3288af579de8ee312c36fb78ac9309ce8c554827)
    (cherry picked from commit 1b3536a7a4d72e7f7b95cc1874a450accad3ec8d)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/keystone/+/892865

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystone/+/892865
Committed: https://opendev.org/openstack/keystone/commit/a38ba2a70cdf7e72c0f17bb80d895ccc1e39a010
Submitter: "Zuul (22348)"
Branch: stable/xena

commit a38ba2a70cdf7e72c0f17bb80d895ccc1e39a010
Author: Dave Wilde (d34dh0r53) <email address hidden>
Date: Wed Feb 9 11:28:59 2022 -0600

    Force algo specific maximum length & Properly trimm bcrypt hashed passwords

    This is the squash of 2 patches related to bcrypt hashing settings.

    1.
    Force algo specific maximum length

    The bcrypt algorithm that we use for password hashing silently
    length limits the size of the password that is hashed giving the
    user a false sense of security [0]. This patch adds a check
    in the verify_length_and_trunc_password function for the hash in
    use and updates the max_length accordingly, this will override
    the configured value and log a warning if the password is truncated.

    Conflicts:
    * tox.ini

    [0]: https://passlib.readthedocs.io/en/stable/lib/passlib.hash.bcrypt.html#security-issues

    2.
    Properly trimm bcrypt hashed passwords

    bcrypt hashing algorythm has a limitation on length of passwords it
    can hash on 72 bytes. In [1] a password trimm to 54 symbols has been
    implemented, which resulted in password being invalidated after the
    keystone upgrade, since passwords are trimmed differently by bcrypt
    itself, as well as len(str()) is not always equal to
    len(str().encode()) as trimming should be done based on bytes and not
    string itself.

    With the change we return a byte object from
    `verify_length_and_trunc_password`, so it does not need to
    be encoded afterwards, since we need to strip based on bytes
    rather then on length of the string.

    [1] https://review.opendev.org/c/openstack/keystone/+/828595

    Closes-Bug: #2028809
    Related-Bug: #1901891
    original change id: Iea95a3c2df041a0046647b3d3dadead1a6d054d1
    (cherry picked from commit 6730c761d18aa547998f2add833c13f45f257fe7)
    (cherry picked from commit 65f1fb6b4a54386f473369b05c8d10d77fb6710c)

    Closes-bug: #1901891
    Change-Id: I8d0bb2438b23227b5a66b94af6f8e198084fcd8d
    (cherry picked from commit 3288af579de8ee312c36fb78ac9309ce8c554827)
    (cherry picked from commit 1b3536a7a4d72e7f7b95cc1874a450accad3ec8d)
    (cherry picked from commit 7852ca24a4eb86cb271ef7ec5e07f8f9c97f926d)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/keystone/+/893549

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystone/+/893549
Committed: https://opendev.org/openstack/keystone/commit/11e1258ccda688fd6f1414ac664c50df56ca2989
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 11e1258ccda688fd6f1414ac664c50df56ca2989
Author: Dave Wilde (d34dh0r53) <email address hidden>
Date: Wed Feb 9 11:28:59 2022 -0600

    Force algo specific maximum length & Properly trimm bcrypt hashed passwords

    This is the squash of 2 patches related to bcrypt hashing settings.

    1.
    Force algo specific maximum length

    The bcrypt algorithm that we use for password hashing silently
    length limits the size of the password that is hashed giving the
    user a false sense of security [0]. This patch adds a check
    in the verify_length_and_trunc_password function for the hash in
    use and updates the max_length accordingly, this will override
    the configured value and log a warning if the password is truncated.

    Conflicts:
    * tox.ini

    [0]: https://passlib.readthedocs.io/en/stable/lib/passlib.hash.bcrypt.html#security-issues

    2.
    Properly trimm bcrypt hashed passwords

    bcrypt hashing algorythm has a limitation on length of passwords it
    can hash on 72 bytes. In [1] a password trimm to 54 symbols has been
    implemented, which resulted in password being invalidated after the
    keystone upgrade, since passwords are trimmed differently by bcrypt
    itself, as well as len(str()) is not always equal to
    len(str().encode()) as trimming should be done based on bytes and not
    string itself.

    With the change we return a byte object from
    `verify_length_and_trunc_password`, so it does not need to
    be encoded afterwards, since we need to strip based on bytes
    rather then on length of the string.

    [1] https://review.opendev.org/c/openstack/keystone/+/828595

    Closes-Bug: #2028809
    Related-Bug: #1901891
    original change id: Iea95a3c2df041a0046647b3d3dadead1a6d054d1
    (cherry picked from commit 6730c761d18aa547998f2add833c13f45f257fe7)
    (cherry picked from commit 65f1fb6b4a54386f473369b05c8d10d77fb6710c)

    Closes-bug: #1901891
    Change-Id: I8d0bb2438b23227b5a66b94af6f8e198084fcd8d
    (cherry picked from commit 3288af579de8ee312c36fb78ac9309ce8c554827)
    (cherry picked from commit 1b3536a7a4d72e7f7b95cc1874a450accad3ec8d)
    (cherry picked from commit 7852ca24a4eb86cb271ef7ec5e07f8f9c97f926d)
    (cherry picked from commit a38ba2a70cdf7e72c0f17bb80d895ccc1e39a010)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/keystone/+/893944

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystone/+/893944
Committed: https://opendev.org/openstack/keystone/commit/0d5186cb49f452423240e8c6234ca2b753918fe7
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 0d5186cb49f452423240e8c6234ca2b753918fe7
Author: Dave Wilde (d34dh0r53) <email address hidden>
Date: Wed Feb 9 11:28:59 2022 -0600

    Force algo specific maximum length & Properly trimm bcrypt hashed passwords

    This is the squash of 2 patches related to bcrypt hashing settings.

    1.
    Force algo specific maximum length

    The bcrypt algorithm that we use for password hashing silently
    length limits the size of the password that is hashed giving the
    user a false sense of security [0]. This patch adds a check
    in the verify_length_and_trunc_password function for the hash in
    use and updates the max_length accordingly, this will override
    the configured value and log a warning if the password is truncated.

    Conflicts:
    * tox.ini

    [0]: https://passlib.readthedocs.io/en/stable/lib/passlib.hash.bcrypt.html#security-issues

    2.
    Properly trimm bcrypt hashed passwords

    bcrypt hashing algorythm has a limitation on length of passwords it
    can hash on 72 bytes. In [1] a password trimm to 54 symbols has been
    implemented, which resulted in password being invalidated after the
    keystone upgrade, since passwords are trimmed differently by bcrypt
    itself, as well as len(str()) is not always equal to
    len(str().encode()) as trimming should be done based on bytes and not
    string itself.

    With the change we return a byte object from
    `verify_length_and_trunc_password`, so it does not need to
    be encoded afterwards, since we need to strip based on bytes
    rather then on length of the string.

    [1] https://review.opendev.org/c/openstack/keystone/+/828595

    Closes-Bug: #2028809
    Related-Bug: #1901891
    original change id: Iea95a3c2df041a0046647b3d3dadead1a6d054d1
    (cherry picked from commit 6730c761d18aa547998f2add833c13f45f257fe7)
    (cherry picked from commit 65f1fb6b4a54386f473369b05c8d10d77fb6710c)

    Closes-bug: #1901891
    Change-Id: I8d0bb2438b23227b5a66b94af6f8e198084fcd8d
    (cherry picked from commit 3288af579de8ee312c36fb78ac9309ce8c554827)
    (cherry picked from commit 1b3536a7a4d72e7f7b95cc1874a450accad3ec8d)
    (cherry picked from commit 7852ca24a4eb86cb271ef7ec5e07f8f9c97f926d)
    (cherry picked from commit a38ba2a70cdf7e72c0f17bb80d895ccc1e39a010)
    (cherry picked from commit 11e1258ccda688fd6f1414ac664c50df56ca2989)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible 27.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 24.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 23.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 21.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 22.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone victoria-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone xena-eom release.