test_encrypted_cinder_volumes_luks intermittently fails with "Volume Type luks already exists."

[Matt Riedemann]

I've seen this in a few jobs recently:

http://logs.openstack.org/11/655711/8/check/nova-multi-cell/15bb8e7/job-output.txt.gz#_2019-04-29_22_30_28_586598

2019-04-29 22:30:28.586598 | controller | b'2019-04-29 22:30:24,519 7333 DEBUG [tempest.scenario.manager] Creating a volume type: luks on backend None'
2019-04-29 22:30:28.586797 | controller | b'2019-04-29 22:30:24,903 7333 INFO [tempest.lib.common.rest_client] Request (TestEncryptedCinderVolumes:test_encrypted_cinder_volumes_luks): 409 POST https://38.108.68.217/volume/v3/642a64c616cd422d9fae4943d2b15d39/types 0.383s'
2019-04-29 22:30:28.586970 | controller | b"2019-04-29 22:30:24,903 7333 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}"
2019-04-29 22:30:28.587041 | controller | b' Body: {"volume_type": {"name": "luks", "extra_specs": {}}}'
2019-04-29 22:30:28.587400 | controller | b" Response - Headers: {'date': 'Mon, 29 Apr 2019 22:30:24 GMT', 'server': 'Apache/2.4.29 (Ubuntu)', 'openstack-api-version': 'volume 3.0', 'vary': 'OpenStack-API-Version', 'content-length': '84', 'content-type': 'application/json', 'x-compute-request-id': 'req-a865471b-5ffb-4db8-b69e-11d9f6f07230', 'x-openstack-request-id': 'req-a865471b-5ffb-4db8-b69e-11d9f6f07230', 'connection': 'close', 'status': '409', 'content-location': 'https://38.108.68.217/volume/v3/642a64c616cd422d9fae4943d2b15d39/types'}"
2019-04-29 22:30:28.587494 | controller | b' Body: b\'{"conflictingRequest": {"code": 409, "message": "Volume Type luks already exists."}}\''

It seems something has recently regressed either in cinder or these tempest tests and the tests aren't handling the 409, but I don't see any obvious related changes.

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%7B%5C%5C%5C%22code%5C%5C%5C%22%3A%20409%2C%20%5C%5C%5C%22message%5C%5C%5C%22%3A%20%5C%5C%5C%22Volume%20Type%20luks%20already%20exists.%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d

[Matt Riedemann (mriedem)]

Looks like this is happening most in the nova-next job which runs tempest.api.compute and scenario tests, slow and non-slow, concurrently, so maybe the concurrency on the scenario tests in that job is what is teasing out the race failure since the tempest-slow job runs the scenario tests serially.

[Matt Riedemann (mriedem)]

Looks like before change https://review.opendev.org/#/c/570161/ in tempest the volume type being created for the test was always a random name so we wouldn't hit the conflict, but after that change there is a single luks volume type created which means we could hit the 409.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/656532

[Ghanshyam Mann (ghanshyammann)]

I am not sure if https://review.opendev.org/#/c/570161/ introduced it ? we still use random name for create volume_type

- https://opendev.org/openstack/tempest/src/commit/661a52ed4523ef3efab966e0986ecea918476c50/tempest/scenario/manager.py

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/657144

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/656532
Reason: https://review.opendev.org/#/c/657144/ is the revert and is approved.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/657144
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=514495b1a0a44f03ddb870c7946bbd2101506e8b
Submitter: Zuul
Branch: master

commit 514495b1a0a44f03ddb870c7946bbd2101506e8b
Author: Matt Riedemann <email address hidden>
Date: Sat May 4 17:34:12 2019 +0000

    Revert "Refactor random naming"

    This reverts commit e3e7b2722e6b78216373f392af5bfeba1e2da5ec.

    After that change, if the test creates a volume type with a
    given name, it's not unique and therefore can hit a 409 conflict
    trying to create a volume type with a name that already exists,
    which can happen when running scenario tests concurrently since
    several encrypted volume tests use the same volume type name "luks".

    See Iecf62d411d2da17c4c983915018e3ed9180ecd11 for an
    alternative but this is easier. It's not entirely clear
    why the original change was made anyway - why do we care
    if the tests create long names? They weren't unnecessary
    because they were random to avoid conflicts.

    Change-Id: I9f5a967514f3c79f861d286f65402e9ec2832cc4
    Closes-Bug: #1826953

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 21.0.0 release.