UPDATE and DELETE operations for keystone entities sometimes don't work

[Yaroslav Lobankov]

Keystone tempest tests for updating various properties of keystone entities (projects, domains, tenants, etc.) sometimes fail on HA OpenStack environments. It is very strange because these tests always worked perfectly. One of the failed tests you can find here http://paste.openstack.org/show/455369/ . Other tests fail with similar error.
Such strange behavior of keystone tests was noticed on MOS-Infra-CI tempest jobs.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "7.0"
  openstack_version: "2015.1.0-7.0"
  api: "1.0"
  build_number: "288"
  build_id: "288"
  nailgun_sha: "93477f9b42c5a5e0506248659f40bebc9ac23943"
  python-fuelclient_sha: "1ce8ecd8beb640f2f62f73435f4e18d1469979ac"
  fuel-agent_sha: "082a47bf014002e515001be05f99040437281a2d"
  fuel-nailgun-agent_sha: "d7027952870a35db8dc52f185bb1158cdd3d1ebd"
  astute_sha: "a717657232721a7fafc67ff5e1c696c9dbeb0b95"
  fuel-library_sha: "121016a09b0e889994118aa3ea42fa67eabb8f25"
  fuel-ostf_sha: "1f08e6e71021179b9881a824d9c999957fcc7045"
  fuelmain_sha: "6b83d6a6a75bf7bca3177fcf63b2eebbf1ad0a85"

ENVIRONMENT:
Ubuntu, Neutron with VLAN segmentation, Ceph for volumes, Ceph for images, Ceph for ephemeral volumes, Sahara, Murano and Ceilo installed, 3 (controller + mongo), 1 (compute + ceph-osd)

[Boris Bobrov (bbobrov)]

the bug disappears when cache is disabled

[Timur Nurlygayanov (tnurlygayanov)]

Priority changed to Critical because some basic Keystone operations failed with enabled cache. The is a workaround - we can disable cache and it will work, but in this case the performance of keystone significantly deteriorates.

Let's check the root of the issue and fix the incorrect cache configuration - it will help to kill to birds with the one stone - we will fix issue with failed keystone operations, fix issue with bad performance and probably it will help to fix issues with wsgi :)

[Alexander Makarov (amakarov)]

Timur, have you tried Boris proposal?

[Yaroslav Lobankov (ylobankov)]

Alexander, I have tried. With cache disabled each test works within 15-20 seconds. With cache enabled each test takes less than 1 second.

[Alexander Makarov (amakarov)]

Now does it affect customer environments?

[Yaroslav Lobankov (ylobankov)]

What do you mean by "Now does it affect customer environments?" ? You are talking about this bug or disabled cache?

[Alexander Makarov (amakarov)]

Sorry, typo: how does it affect?
And I'm about the bug, of course

[Boris Bobrov (bbobrov)]

https://review.openstack.org/#/c/210305/ this needs to be reverted

[Boris Bobrov (bbobrov)]

due to https://review.openstack.org/#/c/210305/ cache was not invalidated on nodes other than the one that processed the request. When an update operation happened on node-1, node-2 and node-3 still had old values.

[Timur Nurlygayanov (tnurlygayanov)]

This issue reproduced on any environment (on hardware nodes too), these tests failed on our MOS QA labs for Tempest and these tests will fail on hardware environments too. In fact we will see the regression issue in Keystone functionality if we will not fix the issue.

[Vladimir Kuklin (vkuklin)]

The fix under discussion was created by mos-puppet team and fixed the bug initially created by MOS keystone folks. So far, sending it to mos-puppet team.

[Denis Egorenko (degorenko)]

For master: https://review.openstack.org/222600
For 7.0: https://review.openstack.org/222604

[Boris Bobrov (bbobrov)]

Steps to reproduce

1. Set up an env with 3 controllers;
2. Create a project;
3. Fetch it 10 times;
4. Update the project's description;
5. Fetch it 10 times. Some of the times the project description will be old.

[Igor Marnat (imarnat)]

Both fixes merged.

For master: https://review.openstack.org/222600
For 7.0: https://review.openstack.org/222604

[Yaroslav Lobankov (ylobankov)]

Verified on ISO #292.

[Dmitriy Kruglov (dkruglov)]

Verified on MOS 8.0. Not reproduced.

ISO info:
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  openstack_version: "2015.1.0-8.0"
  api: "1.0"
  build_number: "138"
  build_id: "138"
  fuel-nailgun_sha: "3a745ee87e659b3ba239bbede21e491292646acb"
  python-fuelclient_sha: "769df968e19d95a4ab4f12b1d2c76d385cf3168c"
  fuel-agent_sha: "84335446172cc6a699252c184076a519ac791ca1"
  fuel-nailgun-agent_sha: "d66f188a1832a9c23b04884a14ef00fc5605ec6d"
  astute_sha: "e99368bd77496870592781f4ba4fb0caacb9f3a7"
  fuel-library_sha: "80c2dcf3e298e576dd50111825041466b0e38d3f"
  fuel-ostf_sha: "983d0e6fe64397d6ff3bd72311c26c44b02de3e8"
  fuel-createmirror_sha: "df6a93f7e2819d3dfa600052b0f901d9594eb0db"
  fuelmain_sha: "4c58b6503fc780be117777182165fd7b037b1a96"