Caching with stale data when a server disconnects due to network partition and reconnects

[Morgan Fainberg]

The flush_on_reconnect optional flag is not used. This can cause stale data to be utilized from a cache server that disconnected due to a network partition. This has security concerns as follows:

1* Password changes/user changes may be reverted for the cache TTL
1a* User may get locked out if PCI-DSS is on and the password change happens during the network
    partition.
2* Grant changes may be reverted for the cache TTL
3* Resources (all types) may become "undeleted" for the cache TTL
4* Tokens (KSM) may become valid again during the cache TTL

As noted in the python-memcached library:

    @param flush_on_reconnect: optional flag which prevents a
            scenario that can cause stale data to be read: If there's more
            than one memcached server and the connection to one is
            interrupted, keys that mapped to that server will get
            reassigned to another. If the first server comes back, those
            keys will map to it again. If it still has its data, get()s
            can read stale data that was overwritten on another
            server. This flag is off by default for backwards
            compatibility.

The solution is to explicitly pass flush_on_reconnect as an optional argument. A concern with this model is that the memcached servers may be utilized by other tooling and may lose cache state (in the case the oslo.cache connection is the only thing affected by the network partitioning).

This similarly needs to be addressed in pymemcache when it is utilized in lieu of python-memcached.

[Ben Nemec (bnemec)]

"A concern with this model is that the memcached servers may be utilized by other tooling and may lose cache state (in the case the oslo.cache connection is the only thing affected by the network partitioning)."

That may be, but I'd rather have an empty but consistent cache than a full but incorrect one. Hopefully network partitions aren't a particularly common occurrence anyway.

So I guess +1 on setting this option.

[Jeremy Stanley (fungi)]

Unless there's a way for a malicious actor to trigger and take advantage of this condition, this is probably a class D (security hardening opportunity) report: https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/644774

[Morgan Fainberg (mdrnstm)]

Keystone is fixed with oslo.cache fix, marked as invalid for keystone

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/644774
Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=1192f185a5fd2fa6177655f157146488a3de81d1
Submitter: Zuul
Branch: master

commit 1192f185a5fd2fa6177655f157146488a3de81d1
Author: Morgan Fainberg <email address hidden>
Date: Fri Mar 22 12:35:16 2019 -0700

    Pass `flush_on_reconnect` to memcache pooled backend

    If a memcache server disappears and then reconnects when multiple memcache
    servers are used (specific to the python-memcached based backends) it is
    possible that the server will contain stale data. The default is now to
    supply the ``flush_on_reconnect`` optional argument to the backend. This
    means that when the service connects to a memcache server, it will flush
    all cached data in the server. The pooled backend is more likely to
    run into issues with this as it does not explicitly use a thread.local
    for the client. The non-pooled backend was not touched, it is not
    the recommended production use-case.

    See the help from python-memcached:

        @param flush_on_reconnect: optional flag which prevents a
     scenario that can cause stale data to be read: If there's more
     than one memcached server and the connection to one is
     interrupted, keys that mapped to that server will get
     reassigned to another. If the first server comes back, those
     keys will map to it again. If it still has its data, get()s
     can read stale data that was overwritten on another
     server. This flag is off by default for backwards
     compatibility.

    Change-Id: I3e335261f749ad065e8abe972f4ac476d334e6b3
    closes-bug: #1819957

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.cache 1.34.0 release.

[Matthew Thode (prometheanfire)]

is this going to be backported to stein/rocky? (the oslo.cache fix)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/656419

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/656420

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/656419
Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=aeb95779c7bea4117ccb69b17eab31d452397964
Submitter: Zuul
Branch: stable/stein

commit aeb95779c7bea4117ccb69b17eab31d452397964
Author: Morgan Fainberg <email address hidden>
Date: Fri Mar 22 12:35:16 2019 -0700

    Pass `flush_on_reconnect` to memcache pooled backend

    If a memcache server disappears and then reconnects when multiple memcache
    servers are used (specific to the python-memcached based backends) it is
    possible that the server will contain stale data. The default is now to
    supply the ``flush_on_reconnect`` optional argument to the backend. This
    means that when the service connects to a memcache server, it will flush
    all cached data in the server. The pooled backend is more likely to
    run into issues with this as it does not explicitly use a thread.local
    for the client. The non-pooled backend was not touched, it is not
    the recommended production use-case.

    See the help from python-memcached:

        @param flush_on_reconnect: optional flag which prevents a
     scenario that can cause stale data to be read: If there's more
     than one memcached server and the connection to one is
     interrupted, keys that mapped to that server will get
     reassigned to another. If the first server comes back, those
     keys will map to it again. If it still has its data, get()s
     can read stale data that was overwritten on another
     server. This flag is off by default for backwards
     compatibility.

    Change-Id: I3e335261f749ad065e8abe972f4ac476d334e6b3
    closes-bug: #1819957
    (cherry picked from commit 1192f185a5fd2fa6177655f157146488a3de81d1)
    Signed-off-by: Matthew Thode <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/656420
Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=1d6d08f4911edb50debab4639e6a2825260b9e09
Submitter: Zuul
Branch: stable/rocky

commit 1d6d08f4911edb50debab4639e6a2825260b9e09
Author: Morgan Fainberg <email address hidden>
Date: Fri Mar 22 12:35:16 2019 -0700

    Pass `flush_on_reconnect` to memcache pooled backend

    If a memcache server disappears and then reconnects when multiple memcache
    servers are used (specific to the python-memcached based backends) it is
    possible that the server will contain stale data. The default is now to
    supply the ``flush_on_reconnect`` optional argument to the backend. This
    means that when the service connects to a memcache server, it will flush
    all cached data in the server. The pooled backend is more likely to
    run into issues with this as it does not explicitly use a thread.local
    for the client. The non-pooled backend was not touched, it is not
    the recommended production use-case.

    See the help from python-memcached:

        @param flush_on_reconnect: optional flag which prevents a
     scenario that can cause stale data to be read: If there's more
     than one memcached server and the connection to one is
     interrupted, keys that mapped to that server will get
     reassigned to another. If the first server comes back, those
     keys will map to it again. If it still has its data, get()s
     can read stale data that was overwritten on another
     server. This flag is off by default for backwards
     compatibility.

    Change-Id: I3e335261f749ad065e8abe972f4ac476d334e6b3
    closes-bug: #1819957
    (cherry picked from commit 1192f185a5fd2fa6177655f157146488a3de81d1)
    Signed-off-by: Matthew Thode <email address hidden>

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.cache 1.33.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.cache 1.30.4 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/704508

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/704508
Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=c31dd1aaac0a1dd8ca3f77b6da911ae85de6dc7a
Submitter: Zuul
Branch: stable/queens

commit c31dd1aaac0a1dd8ca3f77b6da911ae85de6dc7a
Author: Morgan Fainberg <email address hidden>
Date: Fri Mar 22 12:35:16 2019 -0700

    Pass `flush_on_reconnect` to memcache pooled backend

    If a memcache server disappears and then reconnects when multiple memcache
    servers are used (specific to the python-memcached based backends) it is
    possible that the server will contain stale data. The default is now to
    supply the ``flush_on_reconnect`` optional argument to the backend. This
    means that when the service connects to a memcache server, it will flush
    all cached data in the server. The pooled backend is more likely to
    run into issues with this as it does not explicitly use a thread.local
    for the client. The non-pooled backend was not touched, it is not
    the recommended production use-case.

    See the help from python-memcached:

        @param flush_on_reconnect: optional flag which prevents a
     scenario that can cause stale data to be read: If there's more
     than one memcached server and the connection to one is
     interrupted, keys that mapped to that server will get
     reassigned to another. If the first server comes back, those
     keys will map to it again. If it still has its data, get()s
     can read stale data that was overwritten on another
     server. This flag is off by default for backwards
     compatibility.

    Change-Id: I3e335261f749ad065e8abe972f4ac476d334e6b3
    closes-bug: #1819957
    (cherry picked from commit 1192f185a5fd2fa6177655f157146488a3de81d1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.cache queens-eol release.