tempest test_verify_created_server_ephemeral_disk sometimes fails due to SG still in use during delete

[Vedamurthy Joshi]

R4.0 Build 35 ubuntu 16.04.2 Newton

If we run tempest test tempest.api.compute.admin.test_create_server.ServersWithSpecificFlavorTestJSON.test_verify_created_server_ephemeral_disk with run_validation field in tempest.conf not set(i.e false), then tempest tries to skip the test and cleans up the objects which it had created for this test.
During cleanup, SG deletion fails saying that the object is in use.

If run_validation is set to true, then, i have not seen this issue

tempest.conf:
[validation]
run_validation = true <<<<<

======================================================================
ERROR: tearDownClass (tempest.api.compute.admin.test_create_server.ServersWithSpecificFlavorTestJSON)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tempest/test.py", line 292, in tearDownClass
    six.reraise(etype, value, trace)
  File "tempest/test.py", line 275, in tearDownClass
    teardown()
  File "tempest/test.py", line 563, in clear_credentials
    cls._creds_provider.clear_creds()
  File "tempest/lib/common/dynamic_creds.py", line 459, in clear_creds
    self._cleanup_default_secgroup(creds.tenant_id)
  File "tempest/lib/common/dynamic_creds.py", line 406, in _cleanup_default_secgroup
    nsg_client.delete_security_group(secgroup['id'])
  File "tempest/lib/services/network/security_groups_client.py", line 62, in delete_security_group
    return self.delete_resource(uri)
  File "tempest/lib/services/network/base.py", line 41, in delete_resource
    resp, body = self.delete(req_uri)
  File "tempest/lib/common/rest_client.py", line 301, in delete
    return self.request('DELETE', url, extra_headers, headers, body)
  File "tempest/lib/common/rest_client.py", line 659, in request
    self._error_checker(resp, resp_body)
  File "tempest/lib/common/rest_client.py", line 780, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: An object with that identifier already exists
Details: {u'message': u'Security Group 81923ef0-aedc-4b78-b295-4ad1232c4bae in use.', u'type': u'SecurityGroupInUse', u'detail': u''}

[Vedamurthy Joshi (vedujoshi)]

It is a timing issue, if i add a few secs sleep just before deleting the SG, it goes through fine

[Vedamurthy Joshi (vedujoshi)]

Other tempest tests with similar issue :

tempest.scenario.test_encrypted_cinder_volumes.TestEncryptedCinderVolumes.test_encrypted_cinder_volumes_luks when volume service is not enabled

tempest.scenario.test_minimum_basic.TestMinimumBasicScenario where floating_ips is not enabled

tempest.scenario.test_shelve_instance.TestShelveInstance when volume is not enabled

tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern when volume is not enabled

[Vedamurthy Joshi (vedujoshi)]

Even with workaround of having a 5 sec sleep before deleting the SG(as done in bug 1621622), few tests continue to fail.

[Sachin Bansal (sbansal)]

Can you provide audit logs immediately after the test is run?
contrail-logs --object-type config --last 10m

I want to see which object is referring to security-group. Is the test trying to create a port?

[Vedamurthy Joshi (vedujoshi)]

Attached is the log-file for "all" cassandra logs after the test tempest.api.compute.admin.test_create_server.ServersWithSpecificFlavorTestJSON

[Vedamurthy Joshi (vedujoshi)]

Attached is the log-file for "all" cassandra logs after the test tempest.api.compute.admin.test_create_server.ServersWithSpecificFlavorTestJSON

[Vedamurthy Joshi (vedujoshi)]

The SG to track would be 30ff3b58-216f-464c-9c80-fa6fef6d50e7