Insufficient number of users provided error in tempest.api.object_storage tests

[Martin Kopec]

Run the below mentioned tests with CONF.auth.use_dynamic_credentials = False and accounts.yaml file generated with 1 concurrency. The tests are executing serially (NOT in parallel) so the concurrency==1 should be enough.

Note: if the accounts.yaml is generated with higher concurrency (we're using 3 to workaround this issue), the tests pass just fine.

setUpClass (tempest.api.object_storage.test_account_services.AccountTest)",
-------------------------------------------------------------------------",
",
Captured traceback:",
~~~~~~~~~~~~~~~~~~~",
    Traceback (most recent call last):",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/test.py\", line 188, in setUpClass",
    six.reraise(etype, value, trace)",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/.venv/lib/python3.8/site-packages/six.py\", line 719, in reraise",
    raise value",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/test.py\", line 173, in setUpClass",
    cls.setup_credentials()",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/api/object_storage/test_account_services.py\", line 38, in setup_credentials",
    super(AccountTest, cls).setup_credentials()",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/api/object_storage/base.py\", line 77, in setup_credentials",
    super(BaseObjectTest, cls).setup_credentials()",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/test.py\", line 423, in setup_credentials",
    manager = cls.get_client_manager(roles=credentials_type[1:],",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/test.py\", line 695, in get_client_manager",
    creds = cred_provider.get_creds_by_roles(**params)",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/lib/common/preprov_creds.py\", line 315, in get_creds_by_roles",
    net_creds = self._get_creds(roles=roles)",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/lib/common/preprov_creds.py\", line 248, in _get_creds",
    free_hash = self._get_free_hash(useable_hashes)",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/.venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py\", line 360, in inner",
    return f(*args, **kwargs)",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/lib/common/preprov_creds.py\", line 203, in _get_free_hash",
    raise lib_exc.InvalidCredentials(msg)",
",
    tempest.lib.exceptions.InvalidCredentials: Invalid Credentials",
Details: Insufficient number of users provided. AccountTest have allocated all the credentials for this allocation request",
",
",
setUpClass (tempest.api.object_storage.test_container_acl.ObjectTestACLs)",
-------------------------------------------------------------------------",
",
Captured traceback:",
~~~~~~~~~~~~~~~~~~~",
    Traceback (most recent call last):",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/test.py\", line 188, in setUpClass",
    six.reraise(etype, value, trace)",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/.venv/lib/python3.8/site-packages/six.py\", line 719, in reraise",
    raise value",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/test.py\", line 173, in setUpClass",
    cls.setup_credentials()",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/api/object_storage/base.py\", line 77, in setup_credentials",
    super(BaseObjectTest, cls).setup_credentials()",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/test.py\", line 423, in setup_credentials",
    manager = cls.get_client_manager(roles=credentials_type[1:],",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/test.py\", line 695, in get_client_manager",
    creds = cred_provider.get_creds_by_roles(**params)",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/lib/common/preprov_creds.py\", line 315, in get_creds_by_roles",
    net_creds = self._get_creds(roles=roles)",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/lib/common/preprov_creds.py\", line 248, in _get_creds",
    free_hash = self._get_free_hash(useable_hashes)",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/.venv/lib/python3.8/site-packages/oslo_concurrency/lockutils.py\", line 360, in inner",
    return f(*args, **kwargs)",
",
      File \"/home/zuul/src/opendev.org/openinfra/refstack-client/.tempest/tempest/lib/common/preprov_creds.py\", line 203, in _get_free_hash",
    raise lib_exc.InvalidCredentials(msg)",
",
    tempest.lib.exceptions.InvalidCredentials: Invalid Credentials",
Details: Insufficient number of users provided. ObjectTestACLs have allocated all the credentials for this allocation request",

[Martin Kopec (mkopec)]

we hit this in refstack-client (a tool which executes tempest), here's a review where we were able to reproduce it:
https://review.opendev.org/c/openinfra/ansible-role-refstack-client/+/814806
We basically decreased the concurrency (from 3 to 1) used for accounts.yaml generation.

[Martin Kopec (mkopec)]

Seems there was a similar issue in the past: https://bugs.launchpad.net/tempest/+bug/1455561

[Lukáš Piwowarski (piwowarl)]

Hi,

I tried to investigate what's going on here.

Tests in tempest.api.object_storage.test_account_services.AccountTest and tempest.api.object_storage.test_container_acl.ObjectTestACLs require two accounts with member role [1][2].

When tempest account-generator is run with concurrency==1 it generates only one account with member role and that is not enough for tempest to successfully setup resources for tests in AccountTest and ObjectTestACLs (two accounts with member role are required). The mentioned workaround works because tempest account-generator generates more accounts with member role when it is run with concurrency==3.

[1] https://opendev.org/openstack/tempest/src/branch/master/tempest/api/object_storage/test_account_services.py#L31

[2] https://opendev.org/openstack/tempest/src/branch/master/tempest/api/object_storage/test_container_acl.py#L27

[Martin Kopec (mkopec)]

From the documentation (https://docs.openstack.org/tempest/latest/account_generator.html):
"""
-r, --concurrency CONCURRENCY (Optional) Concurrency count (default: 1). The number of accounts required can be estimated as CONCURRENCY x 2. Each user provided in accounts.yaml file will be in a different tenant. This is required to provide isolation between test for running in parallel.
"""

The doc suggest that if concurrency == 1 than the number of the generated accounts should be 2 (1*2).
However, the current code generates only as many accounts as the value of concurrency is: https://opendev.org/openstack/tempest/src/commit/24864bac273ff5d91dfda4723c46e0f0868c686d/tempest/cmd/account_generator.py#L273

So it seems it's a bug in account-generator tool.

[Martin Kopec (mkopec)]

https://review.opendev.org/c/openstack/tempest/+/822434

[Ghanshyam Mann (ghanshyammann)]

Document is a little confusing here, "The number of accounts required can be estimated as
  CONCURRENCY x 2. " this does not say account-generator tool will *generate* two accounts for --concurrency 1 it says required account which also not so clear to me what is required means here?

Any tests can have more than two creds also like any test need project_member, project_adnmin, system_admin etc in that case generating 2 accounts will not be sufficient. We cannot make complete tempest or tempest plugins test run successfully with the default account generated by this tool.

And making concurrency=1 generating two accounts means tempest test need single creds run always in parallel and no way to run them serially. This does not sound good and is more confusing.

As tests need two creds are more cases, we can make the default value of concurrency as 2 (which is 1 currently) and if anyone wants to generate only a single account and run tests-need-single-cred serially they can set it to 1.

[Martin Kopec (mkopec)]

yes, it's confusing and not only the documentation but also the name of the argument - concurrency - as I understand it now, the account-generator with concurrency set to 1 will generate enough credentials for running all tempest tests serially. With concurrency set to 2 it will generate twice as many credentials (as with concurrency set to 1) so that tempest can be run with concurrency == 2 (2 workers executing tests).

So I think we have 2 options here:
1. leave the --concurrency arg as it is and proceed with the proposed patch
2. improve the documentation of the argument as you suggested in the other comment and rename the argument as well - let's go with something like --creds-number

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/tempest/+/822434
Committed: https://opendev.org/openstack/tempest/commit/bd722d366e1787f101dd26e8770e24f22970a4f7
Submitter: "Zuul (22348)"
Branch: master

commit bd722d366e1787f101dd26e8770e24f22970a4f7
Author: lpiwowar <email address hidden>
Date: Tue Dec 21 09:42:28 2021 +0100

    Set default value of --concurrency to 2

    There is a non-negligible number of tests in tempest that need accounts
    generated by account-generator with a concurrency of at least 2. Running
    these tests with accounts generated with the current default value for
    concurrency makes them therefore fail.

    This change sets the default value for concurrency to 2 and updates
    the documentation for --concurrency parameter.

    Closes-Bug: #1948935
    Change-Id: Iada980beff9db3c8cd02c84d0d7602264b66ddf8

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 30.0.0 release.