Cross-AZ test sometimes fails with wrong AZ

[Sean McGinnis]

Occasionally the test test_migrate_volume_generic_cross_az fails due to getting back the wrong AZ. Unable to investigate further at the moment, but it would appear to be some sort of race condition with getting the result.

Example test failure output:

ft1.14: cinder.tests.unit.volume.test_volume_migration.VolumeMigrationTestCase.test_migrate_volume_generic_cross_az_StringException: pythonlogging:'': {{{
2018-07-03 04:49:07,873 INFO [cinder.rpc] Automatically selected cinder-scheduler objects version 1.37 as minimum service version.
2018-07-03 04:49:07,875 INFO [cinder.rpc] Automatically selected cinder-scheduler RPC version 3.11 as minimum service version.
2018-07-03 04:49:07,904 INFO [cinder.volume.manager] Determined volume DB was empty at startup.
2018-07-03 04:49:07,908 INFO [cinder.volume.manager] Service not found for updating active_backend_id, assuming default for driver init.
2018-07-03 04:49:07,910 INFO [cinder.volume.manager] Image-volume cache disabled for host ubuntu-xenial-inap-mtl01-0000507273.
2018-07-03 04:49:07,913 INFO [cinder.rpc] Automatically selected cinder-volume objects version 1.37 as minimum service version.
2018-07-03 04:49:07,915 INFO [cinder.rpc] Automatically selected cinder-volume RPC version 3.15 as minimum service version.
2018-07-03 04:49:07,966 ERROR [cinder.service] Recovered model server connection!
}}}

Traceback (most recent call last):
  File "/home/zuul/src/git.openstack.org/openstack/cinder/.tox/py35/lib/python3.5/site-packages/mock/mock.py", line 1305, in patched
    return func(*args, **keywargs)
  File "/home/zuul/src/git.openstack.org/openstack/cinder/cinder/tests/unit/volume/test_volume_migration.py", line 216, in test_migrate_volume_generic_cross_az
    create_vol.call_args[0][0]['availability_zone'])
  File "/home/zuul/src/git.openstack.org/openstack/cinder/.tox/py35/lib/python3.5/site-packages/testtools/testcase.py", line 411, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/home/zuul/src/git.openstack.org/openstack/cinder/.tox/py35/lib/python3.5/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 'AZ2' != 'nova'

[Sean McGinnis (sean-mcginnis)]

Whatever was causing this appears to be resolved. No sign for the last 7 days according to logstash.

http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22testtools.matchers._impl.MismatchError:%20'AZ2'%20!%3D%20'nova'%5C%22&from=7d