[tempest] test_update_user_password in tempest.api.identity.admin.v3.test_users fails with 'Object not found' error

[Anna Babich]

The subject's test fails with the trace given below:

test_update_user_password[id-2d223a0e-e457-4a70-9fb1-febe027a0ff9]
traceback
Traceback (most recent call last):
  File "/home/developer/mos-tempest-runner/tempest/tempest/api/identity/admin/v3/test_users.py", line 83, in test_update_user_password
    token_details = self.client.get_token(subject_token)['token']
  File "/home/developer/mos-tempest-runner/tempest/tempest/services/identity/v3/json/identity_client.py", line 290, in get_token
    resp, body = self.get("auth/tokens", headers=headers)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 274, in get
    return self.request('GET', url, extra_headers, headers)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 639, in request
    resp, resp_body)
  File "/home/developer/mos-tempest-runner/.venv/lib/python2.7/site-packages/tempest_lib/common/rest_client.py", line 692, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest_lib.exceptions.NotFound: Object not found
Details: {u'message': u'Failed to validate token', u'code': 404, u'title': u'Not Found'}

[Boris Bobrov (bbobrov)]

Could you please post the version of the ISO you're trying to run this on?

[Anna Babich (ababich)]

Last seen on ISO #212: https://jenkins.mosi.mirantis.net/view/Periodic%20(deployment)/job/periodic-testing-deploy-openstack-8.0-ubuntu-vlan-ceph-swift/51/artifact/mos-deploy/logs/tempest-report.html
https://jenkins.mosi.mirantis.net/view/Periodic%20(deployment)/job/periodic-testing-deploy-openstack-8.0-ubuntu-vxlan-lvm-swift/49/artifact/mos-deploy/logs/tempest-report.html

[Boris Bobrov (bbobrov)]

This is being discussed in upstream. Please see the following bugreports and reviews.

https://review.openstack.org/#/c/249863/
https://review.openstack.org/#/c/231191/
https://launchpad.net/bugs/1473567
https://launchpad.net/bugs/1517697
https://review.openstack.org/#/c/243742/

TL;DR: this is a race condition in keystone/tempest related to revocation events and can be fixed in both keystone or tempest. We can either wait for it to get fixed in upstream or apply patches to our tempest now.

[Timur Nurlygayanov (tnurlygayanov)]

Hi Boris, if it is race condition we need to fix it in keystone anyway.
I saw "fix" for Tempest with sleep(1), it looks weird.

Can we include fix from upstream to MOS 8.0?

[Boris Bobrov (bbobrov)]

> we need to fix it in keystone anyway

No. There are cases when we cannot fix it. The issue was discussed at the summit and we came to the conclusion that there is no general fix for that in HA architecture.

The patch in keystone will lower the number of occurrences of test failures, but will not bring it to zero. The patch with the fix (https://review.openstack.org/#/c/243742/7) also needs to go through the upstream gate: it is complex and I will not backport it without proper review from upstream.

Also, this issue only affects tests. Real-life users will highly unlikely hit this behavior. If they will, the impact will be low -- they'll have to re-auth.

> I saw "fix" for Tempest with sleep(1), it looks weird.

This is the proper fix, as we decided at the summit.

I suggest to apply the fixes with sleep() for tempest with Partial-Bug and then assign me back so that I backport the patches in the future.

[Timur Nurlygayanov (tnurlygayanov)]

We are waiting when this issue will be fixed in Tempest.

[Sergii Turivnyi (sturivnyi)]

Detailed bug description:
ISO: http://paste.openstack.org/show/493111/
Configuration: [9.0][MOSQA] Tempest 9.0 (VLAN_CINDER-TEMPEST,HA,Cinder LVM,Swift(api))

Steps to reproduce:
Deploy ISO in configuration see (Detailed bug description)

Expected results:
Test is passed

Actual result:
Test is Failed

Reproducibility:
See attachment

Workaround:
---

Impact:
---

Description of the environment:
See (Detailed bug description)

Additional information:
Error Message

test failed

Stacktrace

Traceback (most recent call last):
  File "tempest/api/identity/admin/v3/test_users.py", line 94, in test_update_user_password
    token_details = self.client.show_token(subject_token)['token']
  File "tempest/services/identity/v3/json/identity_client.py", line 35, in show_token
    resp, body = self.get("auth/tokens", headers=headers)
  File "tempest/lib/common/rest_client.py", line 274, in get
    return self.request('GET', url, extra_headers, headers)
  File "tempest/lib/common/rest_client.py", line 642, in request
    resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 695, in _error_checker
    raise exceptions.NotFound(resp_body, resp=resp)
tempest.lib.exceptions.NotFound: Object not found
Details: {u'message': u'Failed to validate token', u'code': 404, u'title': u'Not Found'}

[Dina Belova (dbelova)]

Added move-to-10.0 tag due to the fact bug was transferred from 9.0 to 10.0

[Alexander Petrov (apetrov-n)]

The bug is not reproduced on MOS 9.1 due to the following fix
https://review.fuel-infra.org/#/c/26664/

I think this fix has resolved the issue

[Dmitry Mescheryakov (dmitrymex)]

It looks like patch of Boris referenced by Alexander fixed the issue for 9.0. And judging by what Boris wrote above, the issue is also fixed upstream starting from Newton.