[SRU] Error validating X-Service-Token

[Jorge Merlino]

[Impact]
This bug can cause a race condition for long running services that reuse their token (eg. Kubernetes Cinder CSI plugin) when the following occurs:

1 [service] Asks nova to attach a volume to a server
2 ...the user's token expires
3 [service] Asks cinder if the volume has been attached
4 [nova] Asks cinder to attach the volume

In step 3 the token is marked as invalid in the cache and step 4 fails even if the token is accompanied by a valid service token. The key is that step 3 has to happen before step 4 which is not frequent hence the race condition.

Also, the client will ask for a new user token if it is not authorized in the calls in steps 1 or 3 but if the token is marked as invalid in step 3 then step 4 fails and the volume becomes stuck in "detaching" status.

[Test Plan]
It hard to reproduce this bug as it depends on the timing of packets and the token expiration. I was able to reproduce by reducing the token expiration to 60 seconds and running a go script that is constantly attaching and detaching volumes. Even then it may take some time for the bug to occur.

The code used is here: https://paste.ubuntu.com/p/CbGNzGxYt9/
The openstack auth information should be set in lines 99-105 and then the script should be called with 3 parameters: the id of a volume and the ids of two servers. The script attaches and detaches the volume between those two servers.

[Where problems could occur]
The patch removes code that work as an optimization in order to save the time needed for rechecking invalid tokens. So it should not add problems beside the loss of the optimization. The new code will return all tokens from the cache for validation instead of throwing an exception. If the token is actually invalid it will be detected later on.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/860481

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystonemiddleware/+/860481
Committed: https://opendev.org/openstack/keystonemiddleware/commit/e05466c5f439cd05482f109e6eb97a50ba156698
Submitter: "Zuul (22348)"
Branch: master

commit e05466c5f439cd05482f109e6eb97a50ba156698
Author: Jorge Merlino <email address hidden>
Date: Wed Oct 5 14:41:06 2022 -0300

    Remove cache invalidation when using expired token

    This can create a race condition for long running services that reuse
    their token (eg. Kubernetes Cinder CSI plugin) in this case for
    example:

    1 [user] Asks nova to attach a volume to a server
    2 ...the user's token expires
    3 [user] Asks cinder if the volume has been attached
    4 [nova] Asks cinder to attach the volume

    In step 3 the token is marked as invalid in the cache and step 4 fails
    even if allow_expired is true

    Closes-Bug: #1987355
    Change-Id: Ice8e34440a5fe1baa370646ed70b5e085c4af70e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/868284

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystonemiddleware 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystonemiddleware/+/868284
Committed: https://opendev.org/openstack/keystonemiddleware/commit/f602fa718d12e4ab15ab134256fa835db2a1b7ee
Submitter: "Zuul (22348)"
Branch: stable/zed

commit f602fa718d12e4ab15ab134256fa835db2a1b7ee
Author: Jorge Merlino <email address hidden>
Date: Wed Oct 5 14:41:06 2022 -0300

    Remove cache invalidation when using expired token

    This can create a race condition for long running services that reuse
    their token (eg. Kubernetes Cinder CSI plugin) in this case for
    example:

    1 [user] Asks nova to attach a volume to a server
    2 ...the user's token expires
    3 [user] Asks cinder if the volume has been attached
    4 [nova] Asks cinder to attach the volume

    In step 3 the token is marked as invalid in the cache and step 4 fails
    even if allow_expired is true

    Closes-Bug: #1987355
    Change-Id: Ice8e34440a5fe1baa370646ed70b5e085c4af70e
    (cherry picked from commit e05466c5f439cd05482f109e6eb97a50ba156698)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/871993

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystonemiddleware/+/871993
Committed: https://opendev.org/openstack/keystonemiddleware/commit/d1f5085985cbc93d5a9cafbafa96ae7b99c94b57
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit d1f5085985cbc93d5a9cafbafa96ae7b99c94b57
Author: Jorge Merlino <email address hidden>
Date: Wed Oct 5 14:41:06 2022 -0300

    Remove cache invalidation when using expired token

    This can create a race condition for long running services that reuse
    their token (eg. Kubernetes Cinder CSI plugin) in this case for
    example:

    1 [user] Asks nova to attach a volume to a server
    2 ...the user's token expires
    3 [user] Asks cinder if the volume has been attached
    4 [nova] Asks cinder to attach the volume

    In step 3 the token is marked as invalid in the cache and step 4 fails
    even if allow_expired is true

    Closes-Bug: #1987355
    Change-Id: Ice8e34440a5fe1baa370646ed70b5e085c4af70e
    (cherry picked from commit e05466c5f439cd05482f109e6eb97a50ba156698)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/873921

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Jorge Merlino <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/873921
Reason: Wrong branch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/873924

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystonemiddleware/+/873924
Committed: https://opendev.org/openstack/keystonemiddleware/commit/11eef7db7591d12acbc9d6e449a871ce21b76b94
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 11eef7db7591d12acbc9d6e449a871ce21b76b94
Author: Jorge Merlino <email address hidden>
Date: Wed Oct 5 14:41:06 2022 -0300

    Remove cache invalidation when using expired token

    This can create a race condition for long running services that reuse
    their token (eg. Kubernetes Cinder CSI plugin) in this case for
    example:

    1 [user] Asks nova to attach a volume to a server
    2 ...the user's token expires
    3 [user] Asks cinder if the volume has been attached
    4 [nova] Asks cinder to attach the volume

    In step 3 the token is marked as invalid in the cache and step 4 fails
    even if allow_expired is true

    Closes-Bug: #1987355
    Change-Id: Ice8e34440a5fe1baa370646ed70b5e085c4af70e

[Chris Valean (cvalean)]

Jorge, are there any plans for this fix to be backported to Ussuri as well?

[Pawel Kubica (pawcykca)]

And the same question about Wallaby ;-)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/877398

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/884740

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/884739

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/884738

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Jorge Merlino <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/884740
Reason: Duplicate

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Jorge Merlino <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/884739
Reason: Duplicate

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Jorge Merlino <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/884738
Reason: Test

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Jorge Merlino <email address hidden>" on branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/keystonemiddleware/+/884738
Reason: To force zuul retry

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystonemiddleware/+/873921
Committed: https://opendev.org/openstack/keystonemiddleware/commit/db5cf4e1fd722c7c5889d79808aaa74bb4ac1aa5
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit db5cf4e1fd722c7c5889d79808aaa74bb4ac1aa5
Author: Jorge Merlino <email address hidden>
Date: Wed Oct 5 14:41:06 2022 -0300

    Remove cache invalidation when using expired token

    This can create a race condition for long running services that reuse
    their token (eg. Kubernetes Cinder CSI plugin) in this case for
    example:

    1 [user] Asks nova to attach a volume to a server
    2 ...the user's token expires
    3 [user] Asks cinder if the volume has been attached
    4 [nova] Asks cinder to attach the volume

    In step 3 the token is marked as invalid in the cache and step 4 fails
    even if allow_expired is true

    Closes-Bug: #1987355
    Change-Id: Ice8e34440a5fe1baa370646ed70b5e085c4af70e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystonemiddleware/+/877398
Committed: https://opendev.org/openstack/keystonemiddleware/commit/4f2bb39a5d79f5f8c2e45f3ae782fcb48c55cd27
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 4f2bb39a5d79f5f8c2e45f3ae782fcb48c55cd27
Author: Jorge Merlino <email address hidden>
Date: Wed Oct 5 14:41:06 2022 -0300

    Remove cache invalidation when using expired token

    This can create a race condition for long running services that reuse
    their token (eg. Kubernetes Cinder CSI plugin) in this case for
    example:

    1 [user] Asks nova to attach a volume to a server
    2 ...the user's token expires
    3 [user] Asks cinder if the volume has been attached
    4 [nova] Asks cinder to attach the volume

    In step 3 the token is marked as invalid in the cache and step 4 fails
    even if allow_expired is true

    Closes-Bug: #1987355
    Change-Id: Ice8e34440a5fe1baa370646ed70b5e085c4af70e
    (cherry picked from commit e05466c5f439cd05482f109e6eb97a50ba156698)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/keystonemiddleware/+/889191
Committed: https://opendev.org/openstack/keystonemiddleware/commit/b80dd8cdf848986e71112f16f26d910e447d27fd
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit b80dd8cdf848986e71112f16f26d910e447d27fd
Author: Jorge Merlino <email address hidden>
Date: Wed Oct 5 14:41:06 2022 -0300

    Remove cache invalidation when using expired token

    This can create a race condition for long running services that reuse
    their token (eg. Kubernetes Cinder CSI plugin) in this case for
    example:

    1 [user] Asks nova to attach a volume to a server
    2 ...the user's token expires
    3 [user] Asks cinder if the volume has been attached
    4 [nova] Asks cinder to attach the volume

    In step 3 the token is marked as invalid in the cache and step 4 fails
    even if allow_expired is true

    Closes-Bug: #1987355
    (cherry picked from commit e05466c5f439cd05482f109e6eb97a50ba156698)
    Change-Id: I4ee2e01666b167edf8b5e0d04d1c63ea842a691e

[Jorge Merlino (jorge-merlino)]

SRU for Jammy

[Jorge Merlino (jorge-merlino)]

SRU for UCA Zed

[Corey Bryant (corey.bryant)]

Thanks Jorge. New package versions with this fix have been uploaded to the jammy and focal unapproved queues as well as cloud-archive staging ppas.

[Corey Bryant (corey.bryant)]

Jorge, if you're finding the code path hard to execute from your test you might be able to verify the installed code via a unit test, along the lines of what was done here: https://bugs.launchpad.net/masakari/+bug/1773765

[Timo Aaltonen (tjaalton)]

Hello Jorge, or anyone else affected,

Accepted python-keystonemiddleware into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-keystonemiddleware/9.4.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Jorge, or anyone else affected,

Accepted python-keystonemiddleware into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-keystonemiddleware/9.0.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Jorge, or anyone else affected,

Accepted python-keystonemiddleware into zed-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:zed-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-zed-needed to verification-zed-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-zed-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Jorge, or anyone else affected,

Accepted python-keystonemiddleware into yoga-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:yoga-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-yoga-needed to verification-yoga-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-yoga-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Jorge, or anyone else affected,

Accepted python-keystonemiddleware into xena-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:xena-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-xena-needed to verification-xena-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-xena-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Jorge, or anyone else affected,

Accepted python-keystonemiddleware into wallaby-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:wallaby-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-wallaby-needed to verification-wallaby-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-wallaby-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Jorge, or anyone else affected,

Accepted python-keystonemiddleware into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Jorge, or anyone else affected,

Accepted python-keystonemiddleware into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!