Several tempest test for Keystone are failed

[Sofiia Andriichenko]

Detailed bug description:
test_remove_user_role_by_unauthorized_user
test_list_user_roles_by_unauthorized_user
test_remove_user_role_non_existent_tenant
test_remove_user_role_request_without_token
test_role_create_blank_name
test_role_create_duplicate
test_list_roles_request_without_token
test_list_user_roles_request_without_token
test_remove_user_role_non_existent_role
test_user_update_own_password

Configuration:
Settings:
Compute - QEMU.
Network - Neutron with VLAN segmentation.
Storage Backends - LVM
Additional services - Install Ironic, Install Sahara

In tab Settings->Compute check Nova quotas
In tab Settings->OpenStack Services check enable Install Ceilometer and Aodh
In tab Networks->Other check enable Neutron DVR

Nodes: controller, compute, ironic,cinder, Telemetry - MongoDB

Steps to reproduce:
    1. Deploy ISO in configuration see (Detailed bug description)
    2. Navigate to controller node
    3. Install git (use apt-get install git)
    4. Clone script to deploy rally + tempest
       # git clone https://github.com/obutenko/mos-rally-verify.git
    5. Navigate to the https://github.com/obutenko/mos-rally-verify
    6. Execute necessary steps to deploy tempest
    7. Tun test in debug mode
        #rally --debug verify start --regex tempest.api.identity.admin.v2.test_roles_negative.RolesNegativeTestJSON.test_list_user_roles_by_unauthorized_user

Expected results:
Test is passed

Actual result:
Test is Failed
(see comments)

Reproducibility:
See attachment

Workaround:
---

Impact:
---

Description of the environment:
See (Detailed bug description)

Additional information:
Error Message

test failed

Stacktrace

Traceback (most recent call last):
  File "/home/rally/.rally/tempest/for-deployment-aac0638e-270e-44e8-bfd0-5c5bc95e1e0e/tempest/api/identity/admin/v2/test_roles_negative.py", line 227, in test_list_user_roles_by_unauthorized_user
    (user, tenant, role) = self._get_role_params()
  File "/home/rally/.rally/tempest/for-deployment-aac0638e-270e-44e8-bfd0-5c5bc95e1e0e/tempest/api/identity/admin/v2/test_roles_negative.py", line 25, in _get_role_params
    self.data.setup_test_user()
  File "/home/rally/.rally/tempest/for-deployment-aac0638e-270e-44e8-bfd0-5c5bc95e1e0e/tempest/api/identity/base.py", line 254, in setup_test_user
    self.setup_test_tenant()
  File "/home/rally/.rally/tempest/for-deployment-aac0638e-270e-44e8-bfd0-5c5bc95e1e0e/tempest/api/identity/base.py", line 261, in setup_test_tenant
    description=data_utils.rand_name('desc'))['tenant']
  File "/home/rally/.rally/tempest/for-deployment-aac0638e-270e-44e8-bfd0-5c5bc95e1e0e/tempest/services/identity/v2/json/tenants_client.py", line 36, in create_tenant
    resp, body = self.post('tenants', post_body)
  File "/home/rally/.rally/tempest/for-deployment-aac0638e-270e-44e8-bfd0-5c5bc95e1e0e/tempest/lib/common/rest_client.py", line 259, in post
    return self.request('POST', url, extra_headers, headers, body)
  File "/home/rally/.rally/tempest/for-deployment-aac0638e-270e-44e8-bfd0-5c5bc95e1e0e/tempest/lib/common/rest_client.py", line 641, in request
    resp, resp_body)
  File "/home/rally/.rally/tempest/for-deployment-aac0638e-270e-44e8-bfd0-5c5bc95e1e0e/tempest/lib/common/rest_client.py", line 684, in _error_checker
    raise exceptions.Unauthorized(resp_body, resp=resp)
tempest.lib.exceptions.Unauthorized: Unauthorized
Details: {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}

[Timur Nurlygayanov (tnurlygayanov)]

Why this issue is in Incomplete state?

[Timur Nurlygayanov (tnurlygayanov)]

Marked as invalid because it reproduced only once and we don't know the root of the issue.

Please move to Confirmed if the issue will be reproduced.

[Sergii Turivnyi (sturivnyi)]

Can reproduce on CI

[Sofiia Andriichenko (sandriichenko)]

reproduced on CI. 9.0 mos iso #400
Snapshot: https://drive.google.com/a/mirantis.com/file/d/0BxPLDs6wcpbDRUlINWoyNC0xOFE/view?usp=sharing

[Alexander Makarov (amakarov)]

Waiting for revert for https://bugs.launchpad.net/fuel/mitaka/+bug/1587136 to merge.
It seems to be the root cause.

[Boris Bobrov (bbobrov)]

Moving from "in progress" to "confirmed" after discussion at the meeting. It seems that dummy driver is not the only reason.

[Dina Belova (dbelova)]

Incomplete state due to the fact, that this is not stably reproduced. Env with reproduced bug needs to be given to dev team

[Oleksiy Butenko (obutenko)]

can't reproduce on ci

[Sofiia Andriichenko (sandriichenko)]

Reproduce for 9.1 snapshot #51
Tests:
test_list_user_roles_by_unauthorized_user
test_remove_user_role_by_unauthorized_user
test_remove_user_role_non_existent_tenant
test_remove_user_role_request_without_token
test_role_create_blank_name
test_role_create_duplicate
test_list_user_roles_request_without_token
test_remove_user_role_non_existent_role

Configuration:
Settings:
Compute - QEMU.
Network - Neutron with VLAN segmentation.
Storage Backends - LVM
Additional services - Install Ironic, Install Sahara

In tab Settings->Compute check Nova quotas
In tab Settings->OpenStack Services check enable Install Ceilometer and Aodh
In tab Networks->Other check enable Neutron DVR
In tab Settings->Security check enable TLS for OpenStack public endpoints, HTTPS for Horizon

Nodes: controller, compute, ironic,cinder, Telemetry - MongoDB

[Boris Bobrov (bbobrov)]

We were not able to reproduce it on the env provided. Most probably the root cause is https://bugs.launchpad.net/oslo.cache/+bug/1590779

[Sofiia Andriichenko (sandriichenko)]

Reproduce only for CI 9.1 snapshot #119, can't reproduce manual
Tests:
test_list_user_roles_by_unauthorized_user
test_remove_user_role_by_unauthorized_user
test_remove_user_role_non_existent_tenant
test_remove_user_role_request_without_token
test_role_create_blank_name
test_role_create_duplicate
test_create_role_request_without_token
test_delete_role_by_unauthorized_user
test_delete_role_non_existent
test_delete_role_request_without_token
test_list_roles_request_without_token
test_list_user_roles_request_without_token
test_remove_user_role_non_existent_role

Configuration:
Settings:
Compute - QEMU.
Network - Neutron with VLAN segmentation.
Storage Backends - Ceph RBD for volumes (Cinder), Ceph RBD for ephemeral volumes (Nova), Ceph RBD for images (Glance), Ceph RadosGW for objects (Swift API)
Additional services - Install Ironic, Install Sahara

In tab Settings->Compute check Nova quotas
In tab Networks->Other check enable Neutron DVR

Nodes: controller, compute, ironic, ceph-osd (ironic + controller, 2 compute + ceph-osd)

tempest.api.identity.admin.v2.test_roles_negative.RolesNegativeTestJSON.test_list_user_roles_by_unauthorized_user:
Traceback (most recent call last):
  File "/home/rally/.rally/tempest/for-deployment-8d455a20-7a0b-4484-af9f-42c2c4e0f7c6/tempest/api/identity/admin/v2/test_roles_negative.py", line 234, in test_list_user_roles_by_unauthorized_user
    (user, tenant, role) = self._get_role_params()
  File "/home/rally/.rally/tempest/for-deployment-8d455a20-7a0b-4484-af9f-42c2c4e0f7c6/tempest/api/identity/admin/v2/test_roles_negative.py", line 25, in _get_role_params
    user = self.setup_test_user()
  File "/home/rally/.rally/tempest/for-deployment-8d455a20-7a0b-4484-af9f-42c2c4e0f7c6/tempest/api/identity/base.py", line 131, in setup_test_user
    tenant = self.setup_test_tenant()
  File "/home/rally/.rally/tempest/for-deployment-8d455a20-7a0b-4484-af9f-42c2c4e0f7c6/tempest/api/identity/base.py", line 142, in setup_test_tenant
    description=data_utils.rand_name('desc'))['tenant']
  File "/home/rally/.rally/tempest/for-deployment-8d455a20-7a0b-4484-af9f-42c2c4e0f7c6/tempest/lib/services/identity/v2/tenants_client.py", line 31, in create_tenant
    resp, body = self.post('tenants', post_body)
  File "/home/rally/.rally/tempest/for-deployment-8d455a20-7a0b-4484-af9f-42c2c4e0f7c6/tempest/lib/common/rest_client.py", line 273, in post
    return self.request('POST', url, extra_headers, headers, body, chunked)
  File "/home/rally/.rally/tempest/for-deployment-8d455a20-7a0b-4484-af9f-42c2c4e0f7c6/tempest/lib/common/rest_client.py", line 667, in request
    resp, resp_body)
  File "/home/rally/.rally/tempest/for-deployment-8d455a20-7a0b-4484-af9f-42c2c4e0f7c6/tempest/lib/common/rest_client.py", line 755, in _error_checker
    raise exceptions.Unauthorized(resp_body, resp=resp)
tempest.lib.exceptions.Unauthorized: Unauthorized
Details: {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Unauthorized'}

snapshot: https://drive.google.com/a/mirantis.com/file/d/0BxPLDs6wcpbDR3pEVVRyS0hScEU/view?usp=sharing

[Sofiia Andriichenko (sandriichenko)]

Tempest test test_assign_user_role_request_without_token is failed with other.

Traceback (most recent call last):

  File "/home/rally/.rally/tempest/for-deployment-8d455a20-7a0b-4484-af9f-42c2c4e0f7c6/tempest/api/identity/admin/v2/test_roles_negative.py", line 139, in test_assign_user_role_request_without_token

    user['id'], role['id'])

  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 485, in assertRaises

    self.assertThat(our_callable, matcher)

  File "/usr/local/lib/python2.7/dist-packages/testtools/testcase.py", line 498, in assertThat

    raise mismatch_error

testtools.matchers._impl.MismatchError: <bound method RolesClient.create_user_role_on_project of <tempest.lib.services.identity.v2.roles_client.RolesClient object at 0x7fe0e8d8fb90>> returned {u'role': {u'id': u'bb4602eb98d3429783be2a943e14bab3', u'name': u'tempest-test_role-229241840', u'domain_id': None}}

[Alexander Makarov (amakarov)]

Upstream bug https://bugs.launchpad.net/keystone/+bug/1590779
We have David Stanek solution for Newton:
https://review.openstack.org/#/c/349704/
Waiting for backport to Mitaka

[Timur Nurlygayanov (tnurlygayanov)]

Hi Alexander, could we prepare the proper fix for stable/mitaka branch in upstream?

[Alexander Makarov (amakarov)]

Timur, that's problematic: Newton fix uses new dogpile.cache so the fix is invalid for Mitaka and earlier releases. There is another fix proposed, however it's not merged yet.

[Boris Bobrov (bbobrov)]

It won't get into 9.1 because the fix is very complex and is not yet in upstream. Targeting to 9.2.

[Timur Nurlygayanov (tnurlygayanov)]

Ok, let's do not skip the fix in MOS 9.2. We need to fix the issue even the fix is complex.

[Boris Bobrov (bbobrov)]

There is a fix for this in upstream: https://review.openstack.org/#/c/369618/ . I am now waiting for reviews from other people, and if their reviews will be positive i will pull it to downstream branches.

Also I am marking it as fix committed in 10.0 because it was merged in upstream newton.

[Timur Nurlygayanov (tnurlygayanov)]

Status changed to In Progress for 10.0 because fix is on review in upstream

[Boris Bobrov (bbobrov)]

It's fix commited for 10.0 because it was already merged for Newton.

[Timur Nurlygayanov (tnurlygayanov)]

The fix is on review for MOS 9.1:
https://review.fuel-infra.org/#/c/26523/

I want to promote the issue to Criticals because it affects all keystone users and we can get many "unauthorized" errors under the big load.

[Timur Nurlygayanov (tnurlygayanov)]

Verified on MOS 9.1 snapshot #287:

https://mirantis.testrail.com/index.php?/tests/view/16519234
https://mirantis.testrail.com/index.php?/tests/view/16519228