oslo.cache's pymemcache backend doesn't recover from socket disconnection

[Herve Beraud]

Description of problem:
When oslo.cache is enabled and configured to target pymemcache (e.g. memcached + TLS-e),
pymemcache is managing the sockets that connect to memcached.

With this configuration, there is no automatic retry in pymemcache on socket error
or socket disconnection. Instead, pymemcache closes the invalid socket and raises
an Exception down the stack. This makes the oslo cache call fail, and any subsequent
calls will also fail until all bad sockets are hit and closed.

Try can consistently been triggered by:
  1. running "openstack service list" on the overcloud to create connection to memcache

  2. restart memcached with "systemctl restart tripleo_memcached" to
     force the connected sockets to close one side of its connection.
     This will leave <x> opened sockets on the controller:
     the keystone service will have its side of the socket still
     opened.

  3. the next call to "openstack service list" will fail because
     pymemcache will hit a half-closed socket, close its side, and
     raise an exception

  4. the keystone service will recover only once the remaining <x>-1 half-closed sockets
     get hit and closed.

Version-Release number of selected component (if applicable):

How reproducible:
Always

Steps to Reproduce:
1. enable keystone cache with pymemcache as backend.

[cache]
backend = dogpile.cache.pymemcache
enabled = true
memcache_servers = 127.0.0.1:11211

2. trigger an API call to that node, e.g.:

openstack service list

3. restart memcache on the node

systemctl restart tripleo_memcached

4. retry the same API call

openstack service list

Actual results:

the last "service list" call will fail with "Internal Server Error (HTTP 500)"

Expected results:

the call should work

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/826569
Committed: https://opendev.org/openstack/oslo.cache/commit/2ad2d52f4ecb63d9edfe3ae64cd9b7dece5330a0
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 2ad2d52f4ecb63d9edfe3ae64cd9b7dece5330a0
Author: Hervé Beraud <email address hidden>
Date: Fri Aug 6 11:49:17 2021 +0200

    Add socket keepalive options to oslo.cache

    This patch specifies a set of options required to setup the
    socket keepalive of the dogpile.cache's pymemcache
    backend [1][2]. This setup from those options can later on
    be passed to this backend.

    This patch also sets up the socket keepalive object based on
    the configuration options passed via oslo.config and adds it
    as an argument to be passed to the selected oslo.cache backend.

    Dogpile.cache will be used as an interface between oslo.cache and
    pymemcache [3].

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    [1] https://github.com/sqlalchemy/dogpile.cache/pull/205
    [2]
    https://github.com/pinterest/pymemcache/commit/b289c87bb89b3ab477bd5d92c8951ab42c923923
    [3]
    https://dogpilecache.sqlalchemy.org/en/latest/api.html?highlight=keepalive#dogpile.cache.backends.memcached.PyMemcacheBackend.params.socket_keepalive

    Partial-Bug: #1959562
    Change-Id: I501100e1a48cdd4e094c08046e2150405dcf371e
    (cherry picked from commit f4fa6aa6fa2aca23a8f4f9e63c5a57dbcd2d1166)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/826593
Committed: https://opendev.org/openstack/oslo.cache/commit/23e8e9a9f45956ef7ea74d3d443e518f13b43477
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 23e8e9a9f45956ef7ea74d3d443e518f13b43477
Author: Hervé Beraud <email address hidden>
Date: Fri Aug 6 14:51:15 2021 +0200

    Add retry mechanisms to oslo.cache

    This patch specifies a set of options required to setup a retry
    context. The context built from those options can later on be
    passed to any of the oslo.cache backends that supports pymemcache's
    retry mechanisms.

    This patch also sets up the retry mechanisms context based on
    the configuration option passed via oslo.config and adds it
    as an argument to be passed to the selected oslo.cache backend.

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    Partial-Bug: #1959562
    Change-Id: I6c1a4872d7cf19e3a55c676ef4b4200f18e08f2c
    (cherry picked from commit 42bf82d5505a0def3d22972d67fde836f5091ed1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/826594
Committed: https://opendev.org/openstack/oslo.cache/commit/55cb199f90b27cef203bca8edb9431f9d7ed841c
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 55cb199f90b27cef203bca8edb9431f9d7ed841c
Author: Hervé Beraud <email address hidden>
Date: Mon Jan 17 14:59:25 2022 +0100

    Expose pymemcache's HashClient public params

    This patch expose a couple of pymemcache's HashClient public
    params that can be useful to configure HA and failover for
    clustered memcached servers.

    These options can be used in addition of the previously added
    retrying mechanismes.

    This patch rely on recent changes [1] of dogpile.cache that
    aim to expose these options too.

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    [1] https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/3528

    Partial-Bug: #1959562
    Depends-On: https://review.opendev.org/c/openstack/requirements/+/826995
    Change-Id: I24fc853db4237c08b87871ddff1b3ced27cc7fce
    (cherry picked from commit cb118d04cea318df23831d7445f6b33c866cd0c6)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/oslo.cache/+/827611

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/oslo.cache/+/827612

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/oslo.cache/+/827613

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/827611
Committed: https://opendev.org/openstack/oslo.cache/commit/7b00cb38a6d08608e5b7b76f18fcdad038490074
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 7b00cb38a6d08608e5b7b76f18fcdad038490074
Author: Hervé Beraud <email address hidden>
Date: Fri Aug 6 11:49:17 2021 +0200

    Add socket keepalive options to oslo.cache

    This patch specifies a set of options required to setup the
    socket keepalive of the dogpile.cache's pymemcache
    backend [1][2]. This setup from those options can later on
    be passed to this backend.

    This patch also sets up the socket keepalive object based on
    the configuration options passed via oslo.config and adds it
    as an argument to be passed to the selected oslo.cache backend.

    Dogpile.cache will be used as an interface between oslo.cache and
    pymemcache [3].

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    [1] https://github.com/sqlalchemy/dogpile.cache/pull/205
    [2]
    https://github.com/pinterest/pymemcache/commit/b289c87bb89b3ab477bd5d92c8951ab42c923923
    [3]
    https://dogpilecache.sqlalchemy.org/en/latest/api.html?highlight=keepalive#dogpile.cache.backends.memcached.PyMemcacheBackend.params.socket_keepalive

    Partial-Bug: #1959562
    Change-Id: I501100e1a48cdd4e094c08046e2150405dcf371e
    (cherry picked from commit f4fa6aa6fa2aca23a8f4f9e63c5a57dbcd2d1166)
    (cherry picked from commit 2ad2d52f4ecb63d9edfe3ae64cd9b7dece5330a0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/827612
Committed: https://opendev.org/openstack/oslo.cache/commit/55b796be5679ffecc64f1940da425cac5c982fc1
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 55b796be5679ffecc64f1940da425cac5c982fc1
Author: Hervé Beraud <email address hidden>
Date: Fri Aug 6 14:51:15 2021 +0200

    Add retry mechanisms to oslo.cache

    This patch specifies a set of options required to setup a retry
    context. The context built from those options can later on be
    passed to any of the oslo.cache backends that supports pymemcache's
    retry mechanisms.

    This patch also sets up the retry mechanisms context based on
    the configuration option passed via oslo.config and adds it
    as an argument to be passed to the selected oslo.cache backend.

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    Partial-Bug: #1959562
    Change-Id: I6c1a4872d7cf19e3a55c676ef4b4200f18e08f2c
    (cherry picked from commit 42bf82d5505a0def3d22972d67fde836f5091ed1)
    (cherry picked from commit 23e8e9a9f45956ef7ea74d3d443e518f13b43477)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/827613
Committed: https://opendev.org/openstack/oslo.cache/commit/d0252f62f3b61253d0bcd953329f7b52e53c6e81
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit d0252f62f3b61253d0bcd953329f7b52e53c6e81
Author: Hervé Beraud <email address hidden>
Date: Mon Jan 17 14:59:25 2022 +0100

    Expose pymemcache's HashClient public params

    This patch expose a couple of pymemcache's HashClient public
    params that can be useful to configure HA and failover for
    clustered memcached servers.

    These options can be used in addition of the previously added
    retrying mechanismes.

    This patch rely on recent changes [1] of dogpile.cache that
    aim to expose these options too.

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    [1] https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/3528

    Partial-Bug: #1959562
    Depends-On: https://review.opendev.org/c/openstack/requirements/+/826995
    Change-Id: I24fc853db4237c08b87871ddff1b3ced27cc7fce
    (cherry picked from commit cb118d04cea318df23831d7445f6b33c866cd0c6)
    (cherry picked from commit 55cb199f90b27cef203bca8edb9431f9d7ed841c)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/oslo.cache/+/828725

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/oslo.cache/+/829493

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/oslo.cache/+/829494

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/828725
Committed: https://opendev.org/openstack/oslo.cache/commit/f34ed73407b93d2c901ec9dc37632a63712c818b
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit f34ed73407b93d2c901ec9dc37632a63712c818b
Author: Hervé Beraud <email address hidden>
Date: Fri Aug 6 11:49:17 2021 +0200

    Add socket keepalive options to oslo.cache

    This patch specifies a set of options required to setup the
    socket keepalive of the dogpile.cache's pymemcache
    backend [1][2]. This setup from those options can later on
    be passed to this backend.

    This patch also sets up the socket keepalive object based on
    the configuration options passed via oslo.config and adds it
    as an argument to be passed to the selected oslo.cache backend.

    Dogpile.cache will be used as an interface between oslo.cache and
    pymemcache [3].

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    Conflicts:
        - requirements.txt
        - doc/requirements.txt
        - oslo_cache/tests/unit/test_cache_basics.py

    NOTE(hberaud): Conflicts are related to requirements that have been
    updated in the youngest branches, however, I removed these reqs updated
    to follow the same logic that have been applied by Moisès in his patches
    related to TLS, namely, appending the dogpile.cache.pymemcache backend
    only if the version of dogpile.cache is higher to a specific version [4].
    TLS changes are at the origin of the bug fixed here.

    [1] https://github.com/sqlalchemy/dogpile.cache/pull/205
    [2]
    https://github.com/pinterest/pymemcache/commit/b289c87bb89b3ab477bd5d92c8951ab42c923923
    [3]
    https://dogpilecache.sqlalchemy.org/en/latest/api.html?highlight=keepalive#dogpile.cache.backends.memcached.PyMemcacheBackend.params.socket_keepalive
    [4] https://opendev.org/openstack/oslo.cache/commit/3e3037886e8021911e10f492c0502951bff5104e

    Partial-Bug: #1959562
    Change-Id: I501100e1a48cdd4e094c08046e2150405dcf371e
    (cherry picked from commit f4fa6aa6fa2aca23a8f4f9e63c5a57dbcd2d1166)
    (cherry picked from commit 2ad2d52f4ecb63d9edfe3ae64cd9b7dece5330a0)
    (cherry picked from commit 7b00cb38a6d08608e5b7b76f18fcdad038490074)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/829493
Committed: https://opendev.org/openstack/oslo.cache/commit/91f61a50c0a4df70ff7e9c729aa3feda82e59582
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 91f61a50c0a4df70ff7e9c729aa3feda82e59582
Author: Hervé Beraud <email address hidden>
Date: Fri Aug 6 14:51:15 2021 +0200

    Add retry mechanisms to oslo.cache

    This patch specifies a set of options required to setup a retry
    context. The context built from those options can later on be
    passed to any of the oslo.cache backends that supports pymemcache's
    retry mechanisms.

    This patch also sets up the retry mechanisms context based on
    the configuration option passed via oslo.config and adds it
    as an argument to be passed to the selected oslo.cache backend.

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    Partial-Bug: #1959562
    Change-Id: I6c1a4872d7cf19e3a55c676ef4b4200f18e08f2c
    (cherry picked from commit 42bf82d5505a0def3d22972d67fde836f5091ed1)
    (cherry picked from commit 23e8e9a9f45956ef7ea74d3d443e518f13b43477)
    (cherry picked from commit 55b796be5679ffecc64f1940da425cac5c982fc1)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/829494
Committed: https://opendev.org/openstack/oslo.cache/commit/84b3519499b55332bfd150d537ad008b6d1c41b5
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 84b3519499b55332bfd150d537ad008b6d1c41b5
Author: Hervé Beraud <email address hidden>
Date: Mon Jan 17 14:59:25 2022 +0100

    Expose pymemcache's HashClient public params

    This patch expose a couple of pymemcache's HashClient public
    params that can be useful to configure HA and failover for
    clustered memcached servers.

    These options can be used in addition of the previously added
    retrying mechanismes.

    This patch rely on recent changes [1] of dogpile.cache that
    aim to expose these options too.

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    [1] https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/3528

    Partial-Bug: #1959562
    Depends-On: https://review.opendev.org/c/openstack/requirements/+/826995
    Change-Id: I24fc853db4237c08b87871ddff1b3ced27cc7fce
    (cherry picked from commit cb118d04cea318df23831d7445f6b33c866cd0c6)
    (cherry picked from commit 55cb199f90b27cef203bca8edb9431f9d7ed841c)
    (cherry picked from commit d0252f62f3b61253d0bcd953329f7b52e53c6e81)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/oslo.cache/+/832773

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/oslo.cache/+/832774

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/oslo.cache/+/832775

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/832773
Committed: https://opendev.org/openstack/oslo.cache/commit/619413326743acb32ed976c9c3596341479978c9
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 619413326743acb32ed976c9c3596341479978c9
Author: Hervé Beraud <email address hidden>
Date: Fri Aug 6 11:49:17 2021 +0200

    Add socket keepalive options to oslo.cache

    This patch specifies a set of options required to setup the
    socket keepalive of the dogpile.cache's pymemcache
    backend [1][2]. This setup from those options can later on
    be passed to this backend.

    This patch also sets up the socket keepalive object based on
    the configuration options passed via oslo.config and adds it
    as an argument to be passed to the selected oslo.cache backend.

    Dogpile.cache will be used as an interface between oslo.cache and
    pymemcache [3].

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    Conflicts:
        - requirements.txt
        - doc/requirements.txt
        - oslo_cache/tests/unit/test_cache_basics.py

    NOTE(hberaud): Conflicts are related to requirements that have been
    updated in the youngest branches, however, I removed these reqs updated
    to follow the same logic that have been applied by Moisès in his patches
    related to TLS, namely, appending the dogpile.cache.pymemcache backend
    only if the version of dogpile.cache is higher to a specific version [4].
    TLS changes are at the origin of the bug fixed here.

    [1] https://github.com/sqlalchemy/dogpile.cache/pull/205
    [2]
    https://github.com/pinterest/pymemcache/commit/b289c87bb89b3ab477bd5d92c8951ab42c923923
    [3]
    https://dogpilecache.sqlalchemy.org/en/latest/api.html?highlight=keepalive#dogpile.cache.backends.memcached.PyMemcacheBackend.params.socket_keepalive
    [4] https://opendev.org/openstack/oslo.cache/commit/3e3037886e8021911e10f492c0502951bff5104e

    This patch also squash parts of the fix submitted with:
    https://review.opendev.org/c/openstack/oslo.cache/+/834267

    Partial-Bug: #1959562
    Change-Id: I501100e1a48cdd4e094c08046e2150405dcf371e
    (cherry picked from commit f4fa6aa6fa2aca23a8f4f9e63c5a57dbcd2d1166)
    (cherry picked from commit 2ad2d52f4ecb63d9edfe3ae64cd9b7dece5330a0)
    (cherry picked from commit 7b00cb38a6d08608e5b7b76f18fcdad038490074)
    (cherry picked from commit f34ed73407b93d2c901ec9dc37632a63712c818b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/832774
Committed: https://opendev.org/openstack/oslo.cache/commit/973616a87323d2d5300d8062e349b6925ece81bd
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 973616a87323d2d5300d8062e349b6925ece81bd
Author: Hervé Beraud <email address hidden>
Date: Fri Aug 6 14:51:15 2021 +0200

    Add retry mechanisms to oslo.cache

    This patch specifies a set of options required to setup a retry
    context. The context built from those options can later on be
    passed to any of the oslo.cache backends that supports pymemcache's
    retry mechanisms.

    This patch also sets up the retry mechanisms context based on
    the configuration option passed via oslo.config and adds it
    as an argument to be passed to the selected oslo.cache backend.

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    Partial-Bug: #1959562
    Change-Id: I6c1a4872d7cf19e3a55c676ef4b4200f18e08f2c
    (cherry picked from commit 42bf82d5505a0def3d22972d67fde836f5091ed1)
    (cherry picked from commit 23e8e9a9f45956ef7ea74d3d443e518f13b43477)
    (cherry picked from commit 55b796be5679ffecc64f1940da425cac5c982fc1)
    (cherry picked from commit 91f61a50c0a4df70ff7e9c729aa3feda82e59582)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/832775
Committed: https://opendev.org/openstack/oslo.cache/commit/708f7ebdad7642d62c40a26db103efcfa8042c26
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 708f7ebdad7642d62c40a26db103efcfa8042c26
Author: Hervé Beraud <email address hidden>
Date: Mon Jan 17 14:59:25 2022 +0100

    Expose pymemcache's HashClient public params

    This patch expose a couple of pymemcache's HashClient public
    params that can be useful to configure HA and failover for
    clustered memcached servers.

    These options can be used in addition of the previously added
    retrying mechanismes.

    This patch rely on recent changes [1] of dogpile.cache that
    aim to expose these options too.

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    [1] https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/3528

    Partial-Bug: #1959562
    Depends-On: https://review.opendev.org/c/openstack/requirements/+/826995
    Change-Id: I24fc853db4237c08b87871ddff1b3ced27cc7fce
    (cherry picked from commit cb118d04cea318df23831d7445f6b33c866cd0c6)
    (cherry picked from commit 55cb199f90b27cef203bca8edb9431f9d7ed841c)
    (cherry picked from commit d0252f62f3b61253d0bcd953329f7b52e53c6e81)
    (cherry picked from commit 84b3519499b55332bfd150d537ad008b6d1c41b5)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/oslo.cache/+/836862

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/oslo.cache/+/836864

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Daniel Bengtsson <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/oslo.cache/+/836864
Reason: Bad commit parent.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/oslo.cache/+/836868

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/oslo.cache/+/836869

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/836862
Committed: https://opendev.org/openstack/oslo.cache/commit/7c8cabe6d842e18655090f37ab5bfbc6cd9b6b8c
Submitter: "Zuul (22348)"
Branch: stable/train

commit 7c8cabe6d842e18655090f37ab5bfbc6cd9b6b8c
Author: Hervé Beraud <email address hidden>
Date: Fri Aug 6 11:49:17 2021 +0200

    Add socket keepalive options to oslo.cache

    This patch specifies a set of options required to setup the
    socket keepalive of the dogpile.cache's pymemcache
    backend [1][2]. This setup from those options can later on
    be passed to this backend.

    This patch also sets up the socket keepalive object based on
    the configuration options passed via oslo.config and adds it
    as an argument to be passed to the selected oslo.cache backend.

    Dogpile.cache will be used as an interface between oslo.cache and
    pymemcache [3].

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    Conflicts:
        - requirements.txt
        - doc/requirements.txt
        - oslo_cache/tests/unit/test_cache_basics.py

    NOTE(hberaud): Conflicts are related to requirements that have been
    updated in the youngest branches, however, I removed these reqs updated
    to follow the same logic that have been applied by Moisès in his patches
    related to TLS, namely, appending the dogpile.cache.pymemcache backend
    only if the version of dogpile.cache is higher to a specific version [4].
    TLS changes are at the origin of the bug fixed here.

    [1] https://github.com/sqlalchemy/dogpile.cache/pull/205
    [2]
    https://github.com/pinterest/pymemcache/commit/b289c87bb89b3ab477bd5d92c8951ab42c923923
    [3]
    https://dogpilecache.sqlalchemy.org/en/latest/api.html?highlight=keepalive#dogpile.cache.backends.memcached.PyMemcacheBackend.params.socket_keepalive
    [4] https://opendev.org/openstack/oslo.cache/commit/3e3037886e8021911e10f492c0502951bff5104e

    This patch also squash parts of the fix submitted with:
    https://review.opendev.org/c/openstack/oslo.cache/+/834267

    Conflits:
        CONFLICT (content): Merge conflict in oslo_cache/tests/test_cache.py

    Partial-Bug: #1959562
    Change-Id: I501100e1a48cdd4e094c08046e2150405dcf371e
    (cherry picked from commit f4fa6aa6fa2aca23a8f4f9e63c5a57dbcd2d1166)
    (cherry picked from commit 2ad2d52f4ecb63d9edfe3ae64cd9b7dece5330a0)
    (cherry picked from commit 7b00cb38a6d08608e5b7b76f18fcdad038490074)
    (cherry picked from commit f34ed73407b93d2c901ec9dc37632a63712c818b)
    (cherry picked from commit 619413326743acb32ed976c9c3596341479978c9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/836868
Committed: https://opendev.org/openstack/oslo.cache/commit/fbcb07575561eb169424687d1150ac4e546d836a
Submitter: "Zuul (22348)"
Branch: stable/train

commit fbcb07575561eb169424687d1150ac4e546d836a
Author: Hervé Beraud <email address hidden>
Date: Fri Aug 6 14:51:15 2021 +0200

    Add retry mechanisms to oslo.cache

    This patch specifies a set of options required to setup a retry
    context. The context built from those options can later on be
    passed to any of the oslo.cache backends that supports pymemcache's
    retry mechanisms.

    This patch also sets up the retry mechanisms context based on
    the configuration option passed via oslo.config and adds it
    as an argument to be passed to the selected oslo.cache backend.

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    Partial-Bug: #1959562
    (cherry picked from commit 42bf82d5505a0def3d22972d67fde836f5091ed1)
    (cherry picked from commit 23e8e9a9f45956ef7ea74d3d443e518f13b43477)
    (cherry picked from commit 55b796be5679ffecc64f1940da425cac5c982fc1)
    (cherry picked from commit 91f61a50c0a4df70ff7e9c729aa3feda82e59582)
    (cherry picked from commit 973616a87323d2d5300d8062e349b6925ece81bd)
    Change-Id: I81e8d1c98726ab2abea984ce8bf46df49beedad3

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/oslo.cache/+/836869
Committed: https://opendev.org/openstack/oslo.cache/commit/79a2b816759c5fc00466174b5c691f1b09f0922f
Submitter: "Zuul (22348)"
Branch: stable/train

commit 79a2b816759c5fc00466174b5c691f1b09f0922f
Author: Hervé Beraud <email address hidden>
Date: Mon Jan 17 14:59:25 2022 +0100

    Expose pymemcache's HashClient public params

    This patch expose a couple of pymemcache's HashClient public
    params that can be useful to configure HA and failover for
    clustered memcached servers.

    These options can be used in addition of the previously added
    retrying mechanismes.

    This patch rely on recent changes [1] of dogpile.cache that
    aim to expose these options too.

    This patch is needed to fix a TLS issue on stable branches introduced by
    pymemcache (since train), where if a cluster node disappear the client
    will fail without retrying to reconnect or to switch to an other node of
    the cluster.

    [1] https://gerrit.sqlalchemy.org/c/sqlalchemy/dogpile.cache/+/3528

    Partial-Bug: #1959562
    Depends-On: https://review.opendev.org/c/openstack/requirements/+/826995
    Change-Id: I24fc853db4237c08b87871ddff1b3ced27cc7fce
    (cherry picked from commit cb118d04cea318df23831d7445f6b33c866cd0c6)
    (cherry picked from commit 55cb199f90b27cef203bca8edb9431f9d7ed841c)
    (cherry picked from commit d0252f62f3b61253d0bcd953329f7b52e53c6e81)
    (cherry picked from commit 84b3519499b55332bfd150d537ad008b6d1c41b5)
    (cherry picked from commit 708f7ebdad7642d62c40a26db103efcfa8042c26)

[Daniel Bengtsson (damani42)]

It's fixed, I close the issue.