OpenStack Identity (keystone)

shadow user cache is not cleaned when the related idp is deleted.

  1. Series stein
  2. Bug #1810393
Bug #1810393 reported by wangxiyuan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
Fix Released
Low
wangxiyuan
OpenStack Identity (keystone) stein-2
Rocky
Fix Committed
High
Colleen Murphy
Stein
Fix Released
Low
wangxiyuan
OpenStack Identity (keystone) stein-2

Bug Description

This bug is found in keystone tempest CI job when adding the domain clean-up step:https://review.openstack.org/#/c/579063/

tempest error log:
ft1.2: keystone_tempest_plugin.tests.scenario.test_federated_authentication.TestSaml2EcpFederatedAuthentication.test_request_unscoped_token_StringException: pythonlogging:'': {{{
2019-01-03 02:34:45,765 4283 INFO [tempest.lib.common.rest_client] Request (TestSaml2EcpFederatedAuthentication:setUp): 201 PUT http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest 0.130s
2019-01-03 02:34:45,766 4283 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
        Body: {"identity_provider": {"remote_ids": ["https://samltest.id/saml/idp"], "enabled": true}}
    Response - Headers: {u'content-type': 'application/json', u'date': 'Thu, 03 Jan 2019 02:34:45 GMT', u'connection': 'close', u'server': 'Apache/2.4.18 (Ubuntu)', u'x-openstack-request-id': 'req-d596a054-3b42-4580-88e0-d9f6cfe9be8f', u'content-length': '373', 'content-location': 'http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest', u'vary': 'X-Auth-Token', 'status': '201'}
        Body: {"identity_provider": {"description": null, "links": {"self": "http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest", "protocols": "http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest/protocols"}, "enabled": true, "domain_id": "e14d592e135046f180f94931c2f5f339", "id": "samltest", "remote_ids": ["https://samltest.id/saml/idp"]}}

2019-01-03 02:34:45,865 4283 INFO [tempest.lib.common.rest_client] Request (TestSaml2EcpFederatedAuthentication:setUp): 201 PUT http://38.108.68.96/identity/v3/OS-FEDERATION/mappings/8269b21476554bbdb196d7251d8566b9 0.098s
2019-01-03 02:34:45,866 4283 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
        Body: {"mapping": {"rules": [{"remote": [{"type": "uid"}], "local": [{"user": {"name": "{0}"}}, {"group": {"domain": {"name": "federated_domain"}, "name": "federated_users"}}]}]}}
    Response - Headers: {u'content-type': 'application/json', u'date': 'Thu, 03 Jan 2019 02:34:45 GMT', u'connection': 'close', u'server': 'Apache/2.4.18 (Ubuntu)', u'x-openstack-request-id': 'req-424b858c-57d1-4693-a5ea-2fb5a1d13b57', u'content-length': '326', 'content-location': 'http://38.108.68.96/identity/v3/OS-FEDERATION/mappings/8269b21476554bbdb196d7251d8566b9', u'vary': 'X-Auth-Token', 'status': '201'}
        Body: {"mapping": {"rules": [{"remote": [{"type": "uid"}], "local": [{"user": {"name": "{0}"}}, {"group": {"domain": {"name": "federated_domain"}, "name": "federated_users"}}]}], "id": "8269b21476554bbdb196d7251d8566b9", "links": {"self": "http://38.108.68.96/identity/v3/OS-FEDERATION/mappings/8269b21476554bbdb196d7251d8566b9"}}}

2019-01-03 02:34:45,918 4283 INFO [tempest.lib.common.rest_client] Request (TestSaml2EcpFederatedAuthentication:setUp): 201 PUT http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest/protocols/mapped 0.051s
2019-01-03 02:34:45,919 4283 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
        Body: {"protocol": {"mapping_id": "8269b21476554bbdb196d7251d8566b9"}}
    Response - Headers: {u'content-type': 'application/json', u'date': 'Thu, 03 Jan 2019 02:34:45 GMT', u'connection': 'close', u'server': 'Apache/2.4.18 (Ubuntu)', u'x-openstack-request-id': 'req-b4cab609-d78f-43b1-9dd7-4039f2b08182', u'content-length': '259', 'content-location': 'http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest/protocols/mapped', u'vary': 'X-Auth-Token', 'status': '201'}
        Body: {"protocol": {"mapping_id": "8269b21476554bbdb196d7251d8566b9", "id": "mapped", "links": {"self": "http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest/protocols/mapped", "identity_provider": "http://38.108.68.96/identity/v3/samltest"}}}

2019-01-03 02:34:46,210 4283 INFO [tempest.lib.common.rest_client] Request (TestSaml2EcpFederatedAuthentication:_run_cleanups): 204 DELETE http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest/protocols/mapped 0.050s
2019-01-03 02:34:46,210 4283 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'content-location': 'http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest/protocols/mapped', u'x-openstack-request-id': 'req-10dee6dc-dec0-4383-8aea-bbf097c5279b', u'date': 'Thu, 03 Jan 2019 02:34:46 GMT', u'connection': 'close', u'server': 'Apache/2.4.18 (Ubuntu)', u'vary': 'X-Auth-Token', 'status': '204'}
        Body:
2019-01-03 02:34:46,256 4283 INFO [tempest.lib.common.rest_client] Request (TestSaml2EcpFederatedAuthentication:_run_cleanups): 204 DELETE http://38.108.68.96/identity/v3/OS-FEDERATION/mappings/8269b21476554bbdb196d7251d8566b9 0.045s
2019-01-03 02:34:46,257 4283 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'content-location': 'http://38.108.68.96/identity/v3/OS-FEDERATION/mappings/8269b21476554bbdb196d7251d8566b9', u'x-openstack-request-id': 'req-989f407c-9b99-4a05-a92d-34deb01bedc0', u'date': 'Thu, 03 Jan 2019 02:34:46 GMT', u'connection': 'close', u'server': 'Apache/2.4.18 (Ubuntu)', u'vary': 'X-Auth-Token', 'status': '204'}
        Body:
2019-01-03 02:34:46,306 4283 INFO [tempest.lib.common.rest_client] Request (TestSaml2EcpFederatedAuthentication:_run_cleanups): 204 DELETE http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest 0.048s
2019-01-03 02:34:46,306 4283 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'content-location': 'http://38.108.68.96/identity/v3/OS-FEDERATION/identity_providers/samltest', u'x-openstack-request-id': 'req-06795a5c-eddd-49e5-85c9-7ce85942b12e', u'date': 'Thu, 03 Jan 2019 02:34:46 GMT', u'connection': 'close', u'server': 'Apache/2.4.18 (Ubuntu)', u'vary': 'X-Auth-Token', 'status': '204'}
        Body:
2019-01-03 02:34:46,400 4283 INFO [tempest.lib.common.rest_client] Request (TestSaml2EcpFederatedAuthentication:_run_cleanups): 200 PATCH http://38.108.68.96/identity/v3/domains/e14d592e135046f180f94931c2f5f339 0.093s
2019-01-03 02:34:46,400 4283 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
        Body: {"domain": {"enabled": false}}
    Response - Headers: {u'content-type': 'application/json', u'date': 'Thu, 03 Jan 2019 02:34:46 GMT', u'connection': 'close', u'server': 'Apache/2.4.18 (Ubuntu)', u'x-openstack-request-id': 'req-ebfc5cdc-af5e-45fd-bca3-f500012489a1', u'content-length': '306', 'content-location': 'http://38.108.68.96/identity/v3/domains/e14d592e135046f180f94931c2f5f339', u'vary': 'X-Auth-Token', 'status': '200'}
        Body: {"domain": {"description": "Auto generated federated domain for Identity Provider: samltest", "links": {"self": "http://38.108.68.96/identity/v3/domains/e14d592e135046f180f94931c2f5f339"}, "tags": [], "enabled": false, "id": "e14d592e135046f180f94931c2f5f339", "name": "e14d592e135046f180f94931c2f5f339"}}

2019-01-03 02:34:46,656 4283 INFO [tempest.lib.common.rest_client] Request (TestSaml2EcpFederatedAuthentication:_run_cleanups): 204 DELETE http://38.108.68.96/identity/v3/domains/e14d592e135046f180f94931c2f5f339 0.255s
2019-01-03 02:34:46,657 4283 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Accept': 'application/json', 'X-Auth-Token': '<omitted>', 'Content-Type': 'application/json'}
        Body: None
    Response - Headers: {'content-location': 'http://38.108.68.96/identity/v3/domains/e14d592e135046f180f94931c2f5f339', u'x-openstack-request-id': 'req-41df84a1-40f5-4105-9034-1ed63d91dc43', u'date': 'Thu, 03 Jan 2019 02:34:46 GMT', u'connection': 'close', u'server': 'Apache/2.4.18 (Ubuntu)', u'vary': 'X-Auth-Token', 'status': '204'}
        Body:
}}}

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/keystone_tempest_plugin/tests/scenario/test_federated_authentication.py", line 168, in test_request_unscoped_token
    self._request_unscoped_token()
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/keystone_tempest_plugin/tests/scenario/test_federated_authentication.py", line 159, in _request_unscoped_token
    self.assertEqual(http_client.CREATED, resp.status_code)
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/opt/stack/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 201 != 404

The reason is that once the identity protocol is deleted, the related shadow uses are cascading deleted. But the related federation auth cache is not cleaned. So that once the same idp and protocol are created during the caching time, the caching user which is deleted already will be always returned.

wangxiyuan (wangxiyuan)
Changed in keystone:
assignee: nobody → wangxiyuan (wangxiyuan)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (master)

Fix proposed to branch: master
Review: https://review.openstack.org/628132

Changed in keystone:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (master)

Reviewed: https://review.openstack.org/628132
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=3bcd8968e97a8efd8f9788a8840dd008c490cea1
Submitter: Zuul
Branch: master

commit 3bcd8968e97a8efd8f9788a8840dd008c490cea1
Author: wangxiyuan <email address hidden>
Date: Thu Jan 3 17:40:15 2019 +0800

    Invalidate shadow_federated_user cache when deleting protocol

    When delete identity provider protocol, the related
    shadow_federated_user cache should be invalidated as well.

    Change-Id: Ia1a86724b7a6747fc5177476ee462d8d062978e0
    Closes-bug: 1810393

Changed in keystone:
status: In Progress → Fix Released
Lance Bragstad (lbragstad)
Changed in keystone:
importance: Undecided → Low
milestone: none → stein-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystone (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/643599

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystone (stable/rocky)

Reviewed: https://review.openstack.org/643599
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=79594bb4efeeca2cdb137603e2778d4459456e08
Submitter: Zuul
Branch: stable/rocky

commit 79594bb4efeeca2cdb137603e2778d4459456e08
Author: wangxiyuan <email address hidden>
Date: Thu Jan 3 17:40:15 2019 +0800

    Invalidate shadow_federated_user cache when deleting protocol

    When delete identity provider protocol, the related
    shadow_federated_user cache should be invalidated as well.

    Depends-on: https://review.openstack.org/643580

    Change-Id: Ia1a86724b7a6747fc5177476ee462d8d062978e0
    Closes-bug: 1810393
    (cherry picked from commit 3bcd8968e97a8efd8f9788a8840dd008c490cea1)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 15.0.0.0rc1

This issue was fixed in the openstack/keystone 15.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystone 14.1.0

This issue was fixed in the openstack/keystone 14.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.