Some tests against pre-provisioned networks which should be honored by PreProvisionedCredentialProvider.

[Mitya Eremeev]

1. setup tempest config to use static accounts
2. setup static accounts in projects.
Some projects have pre-provisioned networks and some projects do not have pre-provisioned networks.
3. launch tests tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest.test_server_multi_create_auto_allocate
or
test_server_create_no_allocate
4. tempest picks random project from static accounts file and the project appears to have network
5. tests fail because they expect the project does not have networks
https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/api/compute/admin/test_auto_allocate_network.py#L79

My suggestion is to fix tempest in the way it would be able to separate projects w/o networks for tests which require networks or not.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tempest/+/831784

[Soniya Murlidhar Vyas (svyas)]

Can you modify this bug report elaborating the following details?

>> When you say some tests are against pre-provisioned networks and hence do require pre-provisioned creditial provider -

- Which tests are against the pre-provioned networks?
- What failures are we observing because of the above issues?
- If the failures are happening and they are consistent then how do pre-provisioned credential provider address these?
- Please attach the relevant traceback of the failures so that it becomes easy for us to understand the issue.
- Please attach enough logs of relevant failures

[Mitya Eremeev (mitos)]

Hello,

I didn't write in bug description: "some tests are against pre-provisioned networks and hence do require pre-provisioned creditial provider".
I wrote that some tests require projects without networks if used static accounts.

- Which tests are against the pre-provioned networks?
Listed in bug description: tempest.api.compute.admin.test_auto_allocate_network.AutoAllocateNetworkTest.test_server_multi_create_auto_allocate or test_server_create_no_allocate

- What failures are we observing because of the above issues?
Tests don't pass checks and are not launched.
https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/api/compute/admin/test_auto_allocate_network.py#L79

Tempest finds pre-provisioned networks and fails tests

- If the failures are happening and they are consistent then how do pre-provisioned credential provider address these?
Again, bug description describes that if static accounts are used and a project has network then tests fail. I believe all tests should work correct if static or dynamic accounts are used.

- Please attach the relevant traceback of the failures so that it becomes easy for us to understand the issue.
- Please attach enough logs of relevant failures

From code it's obvious that occurs exception: lib_excs.TempestException because network was found.
https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/api/compute/admin/test_auto_allocate_network.py#L80

Thank you.

[Lukáš Piwowarski (piwowarl)]

Hi, I have tried to reproduce the bug but it is difficult for me to find tempest configuration that ends up with tempest failing on the line you provided [1]. I was only able to reproduce the bug with pre-provisioned credentials once I set force_tenant_isolation = False here [2].

Is there another way how to end up with the "Found tenant networks:" error [1]? (If it is not then it seems that this is more feature request than it is bug report)

[1] https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/api/compute/admin/test_auto_allocate_network.py#L80

[2] https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/api/compute/admin/test_auto_allocate_network.py#L40

[Mitya Eremeev (mitos)]

Hi,

the bug can be reproduced if pre-provisioned credentials are used and "force_tenant_isolation = False".
There is not other way to reproduce it.
It seems to me it's a valid bug.
If "force_tenant_isolation = True" tests always use dynamic accounts no matter what is specified in tempest config. It looks like a bug than proper behavior.

[Lukáš Piwowarski (piwowarl)]

Hi,

to be honest, I find the force_tenant_isolation variable confusing too. (e.g.: I would personally prefer the name force_dynamic_credential, but I guess there is a reason why was this name chosen.)

I believe that using dynamic credentials when 'force_tenant_isolation = True' is intended behavior [1][2].

I see two things we can do here:
1) Extend the pre-provisioned credentials section in tempest documentation with information that there are tests that will try to use dynamic_credentials even though the auth.use_dnamic_credentials = False.

2) And to resolve the issue mentioned in this bug we have to find a way how to make tests aware of the resources of the pre-provisioned credentials.

I am not 100 % sure that the solution in this patch is ok [3]. I think it would fix this issue for now but another one might appear in future. I am thinking of the scenario in which there are pre-provisioned credentials and all of them have a network. What would happen if there is a test that does not need a network however it is ok for the test to have credentials with a network? I think the test would fail but I can be wrong.

[1] https://github.com/openstack/tempest/blob/9625f0d364c79f4444c6ac8f7900a3731fbe2d07/tempest/common/credentials_factory.py#L132
[2] https://github.com/openstack/tempest/blob/7e96c8e854386f43604ad098a6ec7606ee676145/tempest/tests/common/test_credentials_factory.py#L153
[3] https://review.opendev.org/c/openstack/tempest/+/831784

[Martin Kopec (mkopec)]

The tests was added by https://review.opendev.org/c/openstack/tempest/+/327191
Having no resources (including networks) is crucial for the test of the auto-allocate feature - as we test whether a network will be automatically created in case there is none in the current tenant.

In think that that's the reason (a guarantee there are no resources created) why the test uses dynamic credentials no matter what.

The big question here is whether is possible to use preprovisioned credentials with the 2 tests - well, we can simply set force_tenant_isolation = False and preprov creds would be used, however, that's not all - we would also need to edit tempest/cmd/account_generator.py so that it doesn't create networks for all generated accounts (which seems is the default now). Then we would end up with more complex accounts.yaml file (with and without networks) so we would need to implement further logic so that tests can pick accounts with or without networks. As I'm thinking about it, the solution becomes more and more complex until a point I'm thinking whether it's worth it.

[Mitya Eremeev (mitos)]

Martin,

you're right about "so we would need to implement further logic so that tests can pick accounts with or without networks."
It's already implemented in my patch - please use it and enjoy)

I didn't touch "account_generator.py" because it's not mandatory and only tool for account generating.
I use another tools for it.