test_aggregate_crud fails intermittently in osc-functional-devstack[-tips]

[Eric Fried]

Affects python-openstackclient. Not sure if anyone else runs these jobs.

Sometimes (maybe once every five or six runs?) openstackclient.tests.functional.compute.v2.test_aggregate.AggregateTests.test_aggregate_crud will fail with a trace like:

ft1.2: openstackclient.tests.functional.compute.v2.test_aggregate.AggregateTests.test_aggregate_crud_StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/python-openstackclient/openstackclient/tests/functional/compute/v2/test_aggregate.py", line 90, in test_aggregate_crud
    self.wait_for_status('aggregate', name2, name2)
  File "/home/zuul/src/opendev.org/openstack/python-openstackclient/openstackclient/tests/functional/compute/v2/test_aggregate.py", line 46, in wait_for_status
    self.assertOutput(desired_status, current_status)
  File "/home/zuul/src/opendev.org/openstack/python-openstackclient/openstackclient/tests/functional/base.py", line 105, in assertOutput
    raise Exception(expected + ' != ' + actual)
Exception: 2aff709cdae3482d91072d0811ed933b != 72b4e51dc0894943b1589e3c1d6c3bb1
Suspect a race on a global variable, but haven't looked in depth.

[Eric Fried (efried)]

e-r query: https://review.opendev.org/693028

[Eric Fried (efried)]

I can't see anything wrong with the test flow. As far as I can tell, we're doing this:

- openstack aggregate create -f json --zone nova --property a=b $name1
- Checking the output for, among other things, name=$name1
- openstack aggregate show -f json $name1
- Checking the output for name=$name1
- openstack aggregate create -f json --zone external $name2
- Checking the output for name=$name2 <== this is fine
- openstack aggregate show -f json $name2
- Checking the output for name=$name2 <== this FAILS. The output is actually yielding name=$name1!

We do that last show/check in a loop about 24 times (5s intervals over 120s). The answer never changes. Somehow `aggregate show $name2` is producing output for $name1.

I couldn't find any obvious places where commands or data could collide. As far as I can tell, each `openstack` command is running in a separate process. And as far as I know, osc doesn't run any kind of daemon or other persistent process (I seem to recall that coming up in a recent ML debate about ways to speed it up).