Need policy-awared test

[Yang Youseok]

Hi, tempest team!

We are using rally + tempest quite a while and one problem that we have to deal with is to maintain skip-list which was failed due to our custom policy.

For example, our Neutron policy does not allow network creation by normal user. (Only admin can control about it). There were few more other things that we customized and every test cases that related them were failed.

Our approach is to maintain skip list for the failed cases which results from our customized policy, but I felt it's getting harder to maintain those lists.

I think if tempest can aware of policy which is different from default configuration and skip the related list, it's much better to run tests.

I wonder how other operators handle of this problem since I think changing a policy is quite common.

Thanks!

[Doug Schveninger (ds6901)]

see this bug and comment if you fell that this will help this bug. https://bugs.launchpad.net/tempest/+bug/1738076

[Doug Schveninger (ds6901)]

We handle this by creating a qa domain and running our tests with preprovision creds that we create the project with uuid for the test run. We give all creds admin roles for the test even if they are for the admin,primary or alt cred groups. When the test are done we delete the project and roles assignments. Let me know if this helps and if you want to talk more about this IRC bigdogstl.

[Paras Babbar (pbabbar)]

But how you are dealing with non admin tempest test cases that needs to be executed as non admin users? are they passing as well??

[Doug Schveninger (ds6901)]

A little more detail.

We had over 150 test cases failed due under-permission to our elevated RBAC policy we are using. We use preprovision creds so in the accounts.yaml we tell tempest that we have admin and other (Member in Devstack), for primary and alt but Keystone knows them all as admin only. With this over-permission direction we have about 80 test cases that are failing on test cases that are seeing items that should not in teh functional test cases .

We would like to submit a spec to change the cred provide in tempest to support 3 hash maps, admin, primary and other, over the current 2, admin and other.

I will try to make the Office hours this week to see what other interest exists in this area. https://wiki.openstack.org/wiki/Meetings/QATeamMeeting