When accounts.yaml is used neutron_tempest_plugin fails with "Invalid input for tenant_id. Reason: 'None' is not a valid string."

[Hemachandra Reddy]

When accounts.yaml is used Tempest fails with "Invalid input for tenant_id. Reason: 'None' is not a valid string.". However the test cases run fine when dynamic_credentials is set to true.

The test cases that fail are:
neutron_tempest_plugin.api.test_security_groups.SecGroupTest.test_list_security_group_rules_contains_all_rules
neutron_tempest_plugin.api.test_security_groups_negative.NegativeSecGroupQuotaTest.test_create_excess_sg
neutron_tempest_plugin.api.test_security_groups_negative.NegativeSecGroupRulesQuotaTest.test_sg_creation_with_insufficient_sg_rules_quota

In the last two test cases above, the tenant_id in the URL to update quota is set to None.
200 PUT https://network-test.mydevstack.com/v2.0/quotas/None

Here's the trace...
1. When dynamic_credentials=false and accounts.yaml is used
201 POST https://identity-test.mydevstack.com/v3/auth/tokens
 "user": {"id": "f7cfb79d7e3949bca57cdc0a5726fcf6", "name": "reddy-7026"}, "project": {"id": "69999036590244c5a6e7e22299179bfa", "name": "reddy-7026-DO-NOT-DELETE-tempest-proj"},

201 POST https://identity-test.mydevstack.com/v3/auth/tokens
 {"user": "id": "64b91e81a61a4ff9a53921225ef5c688", "name": "reddy-7047"}, "project": {"id": "583b79fb166a49bda12ea53896e79ed6", "name": "reddy-7047-DO-NOT-DELETE-tempest-proj"}

201 POST https://identity-test.mydevstack.com/v3/auth/tokens
 "user": {"id": "64b91e81a61a4ff9a53921225ef5c688", "name": "reddy-7047"}, "project": {"id": "583b79fb166a49bda12ea53896e79ed6", "name": "reddy-7047-DO-NOT-DELETE-tempest-proj"}

Request (NegativeSecGroupQuotaTest:test_create_excess_sg): 200 GET https://network-test.mydevstack.com/v2.0/quotas/None
 Body: b'{"quota": {"network": 100, "subnet": 100, "subnetpool": -1, "port": 500, "router": 10, "floatingip": 50, "rbac_policy": 10, "security_group": 10, "security_group_rule": 100, "trunk": -1}}

200 PUT https://network-test.mydevstack.com/v2.0/quotas/None
 Body: {"quota": {"security_group": 0}}

 Body: b'{"quota": {"network": 100, "subnet": 100, "subnetpool": -1, "port": 500, "router": 10, "floatingip": 50, "rbac_policy": 10, "security_group": 0, "security_group_rule": 100, "trunk": -1}}'

201 POST https://identity-test.mydevstack.com/v3/auth/tokens
 "user": {"id": "f7cfb79d7e3949bca57cdc0a5726fcf6", "name": "reddy-7026"}, "project": {"id": "69999036590244c5a6e7e22299179bfa", "name": "reddy-7026-DO-NOT-DELETE-tempest-proj"}

201 POST https://network-test.mydevstack.com/v2.0/security-group
  Body: {"security_group": {"name": "tempest-NegativeSecGroupQuotaTest-709899598"}}

====
2. With dynamic_credentials=true
201 POST https://identity-test.mydevstack.com/v3/auth/tokens
 user": {"id": "c81d0a78edfd4c62a7baf28cc431ac71", "name": "admin"}, "project": {"id": "0d3df5dd5d794c41b7bde6d026f24fa2", "name": "admin"}

201 POST https://identity-test.mydevstack.com/v3/auth/tokens
 "user": {"id": "c81d0a78edfd4c62a7baf28cc431ac71", "name": "admin"}, "project": {"id": "0d3df5dd5d794c41b7bde6d026f24fa2", "name": "admin"}

200 GET https://identity-test.mydevstack.com/v3/domains?name=Default
 {"domains": [{"id": "default", "name": "Default", "description": "The default domain", "enabled": true, "tags": [], "options": {}, "links": {"self": "https://identity-test.mydevstack.com/v3/domains/default"}}], "links": {"next": null, "self": "https://identity-test.mydevstack.com/v3/domains?name=Default", "previous": null}}

201 POST https://identity-test.mydevstack.com/v3/projects
 {"project": {"description": "tempest-NegativeSecGroupQuotaTest-1755512384-desc", "domain_id": "default", "name": "tempest-NegativeSecGroupQuotaTest-1755512384"}}
 {"project": {"id": "618161cd2c524aaba5ff107126941a89", "name": "tempest-NegativeSecGroupQuotaTest-1755512384", "domain_id": "default", "description": "tempest-NegativeSecGroupQuotaTest-1755512384-desc", "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": {"self": "https://identity-test.mydevstack.com/v3/projects/618161cd2c524aaba5ff107126941a89"}}}

201 POST https://identity-test.mydevstack.com/v3/users
 {"user": {"project_id": "618161cd2c524aaba5ff107126941a89", "email": "<email address hidden>", "id": "95db37d8c5a249eea0793d110b06e6a6", "name": "tempest-NegativeSecGroupQuotaTest-1755512384", "domain_id": "default", "enabled": true, "password_expires_at": null, "options": {}, "links": {"self": "https://identity-test.mydevstack.com/v3/users/95db37d8c5a249eea0793d110b06e6a6"}}}

200 GET https://identity-test.mydevstack.com/v3/roles

204 PUT https://identity-test.mydevstack.com/v3/projects/618161cd2c524aaba5ff107126941a89/users/95db37d8c5a249eea0793d110b06e6a6/roles/59912ab23ae64d109525ffb2486c1492
 Acquired dynamic creds:
 {'domain_id': 'default', 'domain_name': 'Default', 'username': 'tempest-NegativeSecGroupQuotaTest-1755512384', 'project_domain_id': 'default', 'project_domain_name': 'Default', 'project_id': '618161cd2c524aaba5ff107126941a89', 'project_name': 'tempest-NegativeSecGroupQuotaTest-1755512384', 'tenant_id': '618161cd2c524aaba5ff107126941a89', 'tenant_name': 'tempest-NegativeSecGroupQuotaTest-1755512384', 'user_domain_id': 'default', 'user_domain_name': 'Default', 'user_id': '95db37d8c5a249eea0793d110b06e6a6'}, Network: None, Subnet: None, Router: None

 Created isolated network resources for:
 credentials: Credentials: {'domain_id': 'default', 'domain_name': 'Default', 'username': 'tempest-NegativeSecGroupQuotaTest-1755512384', 'project_domain_id': 'default', 'project_domain_name': 'Default', 'project_id': '618161cd2c524aaba5ff107126941a89', 'project_name': 'tempest-NegativeSecGroupQuotaTest-1755512384', 'tenant_id': '618161cd2c524aaba5ff107126941a89', 'tenant_name': 'tempest-NegativeSecGroupQuotaTest-1755512384', 'user_domain_id': 'default', 'user_domain_name': 'Default', 'user_id': '95db37d8c5a249eea0793d110b06e6a6'}, Network: None, Subnet: None, Router: None

201 POST https://identity-test.mydevstack.com/v3/auth/tokens
 "user": {"id": "95db37d8c5a249eea0793d110b06e6a6", "name": "tempest-NegativeSecGroupQuotaTest-1755512384"}, "project": {"id": "618161cd2c524aaba5ff107126941a89", "name": "tempest-NegativeSecGroupQuotaTest-1755512384"}

201 POST https://identity-test.mydevstack.com/v3/projects
 {"project": {"id": "9c0afd3eff3e465e89226cf4179cb4ec", "name": "tempest-NegativeSecGroupQuotaTest-1126709459", "domain_id": "default", "description": "tempest-NegativeSecGroupQuotaTest-1126709459-desc", "enabled": true, "parent_id": "default", "is_domain": false, "tags": [], "options": {}, "links": {"self": "https://identity-test.mydevstack.com/v3/projects/9c0afd3eff3e465e89226cf4179cb4ec"}}}

201 POST https://identity-test.mydevstack.com/v3/users
 '{"user": {"project_id": "9c0afd3eff3e465e89226cf4179cb4ec", "email": "<email address hidden>", "id": "945e1f5da46e4d26b04888b423dd8853", "name": "tempest-NegativeSecGroupQuotaTest-1126709459", "domain_id": "default", "enabled": true, "password_expires_at": null, "options": {}, "links": {"self": "https://identity-test.mydevstack.com/v3/users/945e1f5da46e4d26b04888b423dd8853"}}}

200 GET https://identity-test.mydevstack.com/v3/roles

204 PUT https://identity-test.mydevstack.com/v3/projects/9c0afd3eff3e465e89226cf4179cb4ec/users/945e1f5da46e4d26b04888b423dd8853/roles/59912ab23ae64d109525ffb2486c1492

200 GET https://identity-test.mydevstack.com/v3/roles

204 PUT https://identity-test.mydevstack.com/v3/projects/9c0afd3eff3e465e89226cf4179cb4ec/users/945e1f5da46e4d26b04888b423dd8853/roles/59912ab23ae64d109525ffb2486c1492

Acquired dynamic creds:
 credentials: Credentials: {'domain_id': 'default', 'domain_name': 'Default', 'username': 'tempest-NegativeSecGroupQuotaTest-1126709459', 'project_domain_id': 'default', 'project_domain_name': 'Default', 'project_id': '9c0afd3eff3e465e89226cf4179cb4ec', 'project_name': 'tempest-NegativeSecGroupQuotaTest-1126709459', 'tenant_id': '9c0afd3eff3e465e89226cf4179cb4ec', 'tenant_name': 'tempest-NegativeSecGroupQuotaTest-1126709459', 'user_domain_id': 'default', 'user_domain_name': 'Default', 'user_id': '945e1f5da46e4d26b04888b423dd8853'}, Network: None, Subnet: None, Router: None
2020-09-16 04:21:35.051 25626 INFO tempest.lib.common.dynamic_creds [-] Created isolated network resources for:
 credentials: Credentials: {'domain_id': 'default', 'domain_name': 'Default', 'username': 'tempest-NegativeSecGroupQuotaTest-1126709459', 'project_domain_id': 'default', 'project_domain_name': 'Default', 'project_id': '9c0afd3eff3e465e89226cf4179cb4ec', 'project_name': 'tempest-NegativeSecGroupQuotaTest-1126709459', 'tenant_id': '9c0afd3eff3e465e89226cf4179cb4ec', 'tenant_name': 'tempest-NegativeSecGroupQuotaTest-1126709459', 'user_domain_id': 'default', 'user_domain_name': 'Default', 'user_id': '945e1f5da46e4d26b04888b423dd8853'}, Network: None, Subnet: None, Router: None

201 POST https://identity-test.mydevstack.com/v3/auth/tokens
 user": {"id": "945e1f5da46e4d26b04888b423dd8853", "name": "tempest-NegativeSecGroupQuotaTest-1126709459"},"project": {"id": "9c0afd3eff3e465e89226cf4179cb4ec", "name": "tempest-NegativeSecGroupQuotaTest-1126709459"}

201 POST https://identity-test.mydevstack.com/v3/auth/tokens

[Hemachandra Reddy (hr858f)]

"user": {"id": "945e1f5da46e4d26b04888b423dd8853", "name": "tempest-NegativeSecGroupQuotaTest-1126709459"}, "project": {"id": "9c0afd3eff3e465e89226cf4179cb4ec", "name": "tempest-NegativeSecGroupQuotaTest-1126709459"}

200 GET https://network-test.mydevstack.com/v2.0/quotas/618161cd2c524aaba5ff107126941a89
 '{"quota": {"network": 100, "subnet": 100, "subnetpool": -1, "port": 500, "router": 10, "floatingip": 50, "rbac_policy": 10, "security_group": 10, "security_group_rule": 100, "trunk": -1}}'

200 PUT https://network-test.mydevstack.com/v2.0/quotas/618161cd2c524aaba5ff107126941a89
 Body: {"quota": {"security_group": 0}}
 '{"quota": {"network": 100, "subnet": 100, "subnetpool": -1, "port": 500, "router": 10, "floatingip": 50, "rbac_policy": 10, "security_group": 0, "security_group_rule": 100, "trunk": -1}}'

201 POST https://identity-test.mydevstack.com/v3/auth/tokens
 "user": {"id": "95db37d8c5a249eea0793d110b06e6a6", "name": "tempest-NegativeSecGroupQuotaTest-1755512384"}, "project": {"id": "618161cd2c524aaba5ff107126941a89", "name": "tempest-NegativeSecGroupQuotaTest-1755512384"},

409 POST https://network-test.mydevstack.com/v2.0/security-group

[Martin Kopec (mkopec)]

Changing the component to neutron as the mentioned failed tests belong to neutron_tempest_plugin.

[Brian Haley (brian-haley)]

Do you have a change/review where you are seeing these failures? I don't see any place in the neutron repos where we set the dynamic_credentials option at all. Also, is this on the master branch?

[Hemachandra Reddy (hr858f)]

The code used was from latest on master branch of neutron-tempest-plugin. "use_dynamic_credentials " is part of tempest.conf: https://docs.openstack.org/tempest/latest/_static/tempest.conf.sample

[Brian Haley (brian-haley)]

When you set use_dynamic_credentials=False, did you set the credentials manually? From reading the doc this is required. I can't tell from above if these are being set or just this small number of tests isn't doing the right thing.

Actually, looking at the code, maybe it's two missing credentials lines in the test files. Could you test a patch if I sent it out?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/754782

[Brian Haley (brian-haley)]

Hemachandra - are you able to see if https://review.opendev.org/754782 fixes the failure for you? Thanks

[Hemachandra Reddy (hr858f)]

Hi Brian, Sorry, could not respond sooner. I'm afraid the test cases still fail. Tenant ID substitution still gets "None".

2020-09-30 21:56:14,508 11595 INFO [tempest.lib.common.rest_client] Request (NegativeSecGroupQuotaTest:_run_cleanups): 200 PUT https://network-test.mydevstack.com/v2.0/quotas/None 0.809s

If it worked in your environment, may be we should compare our tempest.conf. Here's mine...

(.tempest) $ cat etc/tempest.conf |grep auth -A10
[auth]
use_dynamic_credentials = false
test_accounts_file = /tmp/tempest/etc/accounts.yaml
tempest_roles = _member_
default_credentials_domain_name = Default
create_isolated_networks = true
admin_username = admin
admin_project_name = admin
admin_password = xxx

[Brian Haley (brian-haley)]

Hi, thanks for the info, I'll have to run through the tests manually to track this down.

[Brian Haley (brian-haley)]

I'm seeing a different failure, doesn't seem like it's picking-up the auth settings correctly from tempest.conf?

[auth]
#use_dynamic_credentials = True
use_dynamic_credentials = False
tempest_roles = member,creator
admin_domain_name = Default
admin_project_name = admin
admin_password = nomoresecret
admin_username = admin

But a lot of tests fail with:

tempest.lib.exceptions.InvalidConfiguration: Invalid Configuration
Details: A valid credential provider is needed

Is there something obvious I'm missing in setting this up?

[Hemachandra Reddy (hr858f)]

have you provided accounts.yaml in tempest.conf?

[auth]
use_dynamic_credentials = false
test_accounts_file = /tmp/tempest/etc/accounts.yaml

There is a way to generate it, though I never used it.

[Mitya Eremeev (mitos)]

Hi, Brian !

more one year without activity in the ticket.
Sorry, I re-assigned to me.
I have fix for it and upload review soon.
Thank you.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/818933

[Mitya Eremeev (mitos)]

The patch also fixes "overquota" error for tests in:
neutron_tempest_plugin.api.test_security_groups
neutron_tempest_plugin.api.test_revisions
neutron_tempest_plugin.scenario.test_security_groups

Example of error:

Traceback (most recent call last):

  File "/var/lib/openstack/lib/python3.6/site-packages/tempest/common/utils/__init__.py", line 108, in wrapper

    return func(*func_args, **func_kwargs)

  File "/var/lib/openstack/lib/python3.6/site-packages/neutron_tempest_plugin/api/test_revisions.py", line 114, in test_update_sg_group_bumps_revision

    security_group = self.create_security_group()

  File "/var/lib/openstack/lib/python3.6/site-packages/neutron_tempest_plugin/api/base.py", line 842, in create_security_group

    security_group = client.create_security_group(name=name, **kwargs)[

  File "/var/lib/openstack/lib/python3.6/site-packages/neutron_tempest_plugin/services/network/json/network_client.py", line 145, in _create

    resp, body = self.post(uri, post_data)

  File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 299, in post

    return self.request('POST', url, extra_headers, headers, body, chunked)

  File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 710, in request

    self._error_checker(resp, resp_body)

  File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 831, in _error_checker

    raise exceptions.Conflict(resp_body, resp=resp)

tempest.lib.exceptions.Conflict: Conflict with state of target resource

Details: {'type': 'OverQuota', 'message': "Quota exceeded for resources: ['security_group'].", 'detail': ''}

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Brian Haley <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/754782
Reason: See https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/818933

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/818933
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/e01cd53d6c6cd9d74ead277c1fcd5eba73625397
Submitter: "Zuul (22348)"
Branch: master

commit e01cd53d6c6cd9d74ead277c1fcd5eba73625397
Author: Mitya_Eremeev <email address hidden>
Date: Tue Nov 23 17:47:21 2021 +0300

    Set SG quota for specific project.

    project_id was initialized too early.
    It was None. The same for tenant_id.
    That's why unlimited SG quota was set for
    "some default project".
    But the project in test had default quota(10).
    If test class created more 10 SG then arised exception
    "OverQuota".
    Also SG quota for default project was reset to default value in cleanup.
    It could harm another tests which were launched in parallel.

    Closes-Bug: 1895876
    Closes-Bug: 1810963
    Change-Id: If7f3789b2791be08088588619f3a1629aeecbda4

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron-tempest-plugin 1.9.0 release.