Cache region invalidation works for local CacheRegion object only

Bug #1590779 reported by Alexander Makarov on 2016-06-09
44
This bug affects 8 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Critical
David Stanek
oslo.cache
Undecided
Alexander Makarov

Bug Description

oslo_cache uses dogpile.cache's CacheRegion
which invalidates by setting region object attributes:
- self._hard_invalidated
- self._soft_invalidated
Then it checks these attributes on value get.
So this invalidation works for particular region object only.

If there is a need to invalidate a region so that values in it are no more valid for other instances of CacheRegion (either in the same process or in another one) - it's simply impossible.

Alexander Makarov (amakarov) wrote :

Wasn't added by CI for some reason:
https://review.openstack.org/#/c/327776/

Changed in oslo.cache:
assignee: nobody → Alexander Makarov (amakarov)
status: New → In Progress
Boris Bobrov (bbobrov) wrote :

I am adding keystone because it has some logic for cache invalidation across projects. Also, we ran into this issue originally on keystone.

The code on https://github.com/openstack/keystone/blob/stable/mitaka/keystone/common/cache/core.py#L71 is supposed to proxy calls to cache invalidation. Unfortunately, lines 123 and 124 do not set setters and deleters to _hard_invalidated and _soft_invalidated. This leads to dogpile.cache working like it was not patched.

Boris Bobrov (bbobrov) wrote :

For keystone it means that COMPUTED_CATALOG_REGION and COMPUTED_ASSIGNMENTS_REGION are not invalidated when catalog or assignments get changed.

Boris Bobrov (bbobrov) wrote :

http://paste.openstack.org/show/509291/ -- script to reproduce the issue. Cache must be enabled.

Fix proposed to branch: master
Review: https://review.openstack.org/327848

Changed in keystone:
assignee: nobody → Boris Bobrov (bbobrov)
status: New → In Progress

Fix proposed to branch: master
Review: https://review.openstack.org/327885

Change abandoned by Alexander Makarov (<email address hidden>) on branch: master
Review: https://review.openstack.org/327776
Reason: This will be solved another way: dogpile.cache will provide invalidation strategy interface.

Jeffrey Zhang (jeffrey4l) wrote :

is there any workaround for this? Kolla CI is busted due to this. How about restart the memcached server after all
the endpoint is registered?

Boris Bobrov (bbobrov) wrote :

Restarting memcached will help.

Changed in keystone:
milestone: none → newton-3
importance: Undecided → High
Changed in keystone:
importance: High → Critical
Steve Martinelli (stevemar) wrote :

So, I'm not sure we can do anything about this in Mitaka.

Correct me if I'm wrong:

In Master/Newton this is solved in keystone via oslo.cache (we use dogpile >0.6.2)

In Mitaka the problem exists, and should be fixed in oslo.cache (https://review.openstack.org/354831). But I think would break stable policy since it's a fairly large change. I'm not sure, regardless this does not impact keystone itself.

In Liberty this won't go in since it's not a security or critical fix.

Changed in keystone:
status: In Progress → Invalid
milestone: newton-3 → none
importance: Critical → Undecided
assignee: Boris Bobrov (bbobrov) → nobody
Mathieu Gagné (mgagne) wrote :

> In Liberty this won't go in since it's not a security or critical fix.

It thinks the impact of this bug should be considered a security issue. In Keystone, cache invalidation is used when assignments are updated. This means a user could still get a token with an assignment he is no longer supposed to have, even after an admin removed the assignment. This is how we (on our side) found out about the cache invalidation issue.

Changed in keystone:
assignee: nobody → David Stanek (dstanek)
status: Invalid → In Progress
Changed in keystone:
importance: Undecided → Medium
milestone: none → newton-3
importance: Medium → Critical

Reviewed: https://review.openstack.org/349704
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=42eda48c78f1153081b4c193dc13c88561409fd3
Submitter: Jenkins
Branch: master

commit 42eda48c78f1153081b4c193dc13c88561409fd3
Author: David Stanek <email address hidden>
Date: Mon Aug 1 21:06:50 2016 +0000

    Distributed cache namespace to invalidate regions

    dogpile.cache's region invalidation is not designed to work across
    processes. This patch enables distributed invalidation of keys in a
    region.

    Instead of using a static cache key, we use the original cache key
    and append a dynamic value to it. This value is looked up in
    memcached using the region name as a key. So anytime the value of
    the region key changes the cache keys in that region are
    effectively invalidated.

    Closes-Bug: #1590779
    Change-Id: Ib80d41d43ef815b37282d72ad68e7aa8e1ff354e

Changed in keystone:
status: In Progress → Fix Released

Change abandoned by Boris Bobrov (<email address hidden>) on branch: master
Review: https://review.openstack.org/327885
Reason: not needed any longer

This issue was fixed in the openstack/keystone 10.0.0.0b3 development milestone.

Change abandoned by Boris Bobrov (<email address hidden>) on branch: master
Review: https://review.openstack.org/327848
Reason: was fixed another way

Change abandoned by Eric Brown (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/364593
Reason: Abandoning in favor of https://review.openstack.org/#/c/369618/

Reviewed: https://review.openstack.org/369618
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=f1d9c54ef07c61cb80def5779802cc4daf45f4cb
Submitter: Jenkins
Branch: stable/mitaka

commit f1d9c54ef07c61cb80def5779802cc4daf45f4cb
Author: David Stanek <email address hidden>
Date: Mon Aug 1 21:06:50 2016 +0000

    Distributed cache namespace to invalidate regions

    dogpile.cache's region invalidation is not designed to work across
    processes. This patch enables distributed invalidation of keys in a
    region.

    Instead of using a static cache key, we use the original cache key
    and append a dynamic value to it. This value is looked up in
    memcached using the region name as a key. So anytime the value of
    the region key changes the cache keys in that region are
    effectively invalidated.

    Conflicts:
     keystone/assignment/core.py: imports were fixed
     keystone/catalog/core.py: imports were fixed
     keystone/common/cache/core.py:
                    dogpile.cache doesn't have invalidation strategies
                    in 0.5.8. Because of that, call to
                    region.invalidate was redefined.
     keystone/identity/core.py:
                    there is no per-region cache in id_mapping_api
                    in Mitaka
     keystone/revoke/core.py:
                    there is no per-region cache in revocations
                    in Mitaka
     keystone/server/backends.py:
                    removed configuration of regions which were added
                    only in Newton
     keystone/tests/unit/test_v3_assignment.py:
                    conflict due to freezegun being used in Newton and
                    not used in Mitaka
     keystone/token/provider.py:
                    there is no per-region cache in token providers
                    in Mitaka

    Closes-Bug: #1590779
    Change-Id: Ib80d41d43ef815b37282d72ad68e7aa8e1ff354e
    (cherry picked from commit 42eda48c78f1153081b4c193dc13c88561409fd3)

tags: added: in-stable-mitaka

This issue was fixed in the openstack/keystone 9.2.0 release.

Change abandoned by ChangBo Guo(gcb) (<email address hidden>) on branch: master
Review: https://review.openstack.org/354831
Reason: This is no update more than 10 weeks, feel free to restore it if you want to continue work on this.

Morgan Fainberg (mdrnstm) wrote :

We should confirm if the change was landed in dogpile.cache that makes this easier to handle natively in oslo.cache.

Changed in oslo.cache:
status: In Progress → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers