Credential locks are not released

Bug #2052752 reported by Samuel Allan
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tempest
New
Undecided
Unassigned

Bug Description

When running tempest tests with a test_accounts_file, it very quickly runs out of accounts, resulting in InvalidCredentials errors with "Insufficient number of users provided.":

```
setUpClass (murano_tempest_tests.tests.scenario.application_catalog.test_security_groups.TestSecurityGroups)
------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/lib/python3.10/site-packages/tempest/test.py", line 206, in setUpClass
    raise value.with_traceback(trace)

      File "/lib/python3.10/site-packages/tempest/test.py", line 196, in setUpClass
    cls.setup_clients()

      File "/lib/python3.10/site-packages/murano_tempest_tests/tests/scenario/application_catalog/base.py", line 48, in setup_clients
    creds = cls.get_configured_isolated_creds(type_of_creds='primary')

      File "/lib/python3.10/site-packages/murano_tempest_tests/tests/scenario/application_catalog/base.py", line 107, in get_configured_isolated_creds
    creds = cls.credentials.get_primary_creds()

      File "/lib/python3.10/site-packages/tempest/lib/common/preprov_creds.py", line 315, in get_primary_creds
    net_creds = self._get_creds()

      File "/lib/python3.10/site-packages/tempest/lib/common/preprov_creds.py", line 272, in _get_creds
    free_hash = self._get_free_hash(useable_hashes)

      File "/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 414, in inner
    return f(*args, **kwargs)

      File "/lib/python3.10/site-packages/tempest/lib/common/preprov_creds.py", line 219, in _get_free_hash
    raise lib_exc.InvalidCredentials(msg)

    tempest.lib.exceptions.InvalidCredentials: Invalid Credentials
Details: Insufficient number of users provided. TestRepositorySanity,ServerActionsTestJSON,TestVersioningNegative,BulkNetworkOpsTest,TestEnvironmentTemplatesSanity,TestRepositorySanity,TestRepository,TestCinderVolumes,TestServices,TestEnvironmentTemplates,TestMuranoDeployment,TestCinderVolumeIsolatedAdmin have allocated all the credentials for this allocation request
```

It appears that locks are not released, as the account lock files (names are hashes, content is test class name) in /tmp/tempest-lock/ remain there after a test run is complete.

This may be related to https://bugs.launchpad.net/tempest/+bug/1455561 - it was marked as fixed, but some users later post comments with workarounds, suggesting there are remaining related issues.

description: updated
Revision history for this message
Samuel Allan (samuelallan) wrote (last edit ):

I can reliably reproduce this failure to clean up credentials on my environment with this test:

```
murano_tempest_tests.tests.api.application_catalog.artifacts.test_repository.TestRepositorySanity.test_get_list_packages[id-2818aaa0-6613-4bd9-8abe-02713121357a,smoke]
```

Which I believe is found in git+https://opendev.org/openstack/murano-tempest-plugin.git@2.6.0

In my environment (2023.2/stable sunbeam), this test is skipped with a message:

```
{3} setUpClass (murano_tempest_tests.tests.api.application_catalog.artifacts.test_repository.TestRepositorySanity) ... SKIPPED: Murano is not using GLARE backend. Skipping GL
```

Every time I run tempest with this test, I see another file left in the /tmp/tempest-lock directory, such that after 4 runs, the directory looks like this:

```
root@tempest-k8s-0:/tmp/tempest-lock# grep -r .
533bd1b808f1c89a41790ccb98b31610:TestRepositorySanity
7b03fe17fc92437c7bb7dff60b8bf565:TestRepositorySanity
e5604fdc3b6b8200b13a1bfa6c4bf8fa:TestRepositorySanity
22a805d5d9f003ab3342b9af2401528a:TestRepositorySanity
```

These files accumulate regardless of whether other tests run in the same tempest run or not (eg. running with --regex smoke or running with --regex 'murano_tempest_tests..*artifacts.test_repository.TestRepositorySanity').

These lock files that aren't cleaned up accumulate, and eventually cause tempest to fail to allocate accounts for tests.

Revision history for this message
Samuel Allan (samuelallan) wrote (last edit ):

So far the only tests that are triggering the failed credential lock cleanup are within the murano plugin. I'm wondering if it's related to https://bugs.launchpad.net/tempest/+bug/1846236 , as this plugin also raises skipException in the resource_setup methods.

For example, here is where skipException is raised for the example in the previous comment: https://opendev.org/openstack/murano-tempest-plugin/src/commit/e16b2dc1512854c9a249944a5f288a1065f328fd/murano_tempest_tests/tests/api/application_catalog/artifacts/test_repository.py#L29-L34

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.