Tempest Tests with multilpe Availability Zones

[Daniel Haupt]

We have different Availability Zones configured and also set "cross_az_attach = false" in nova.conf.
For following tempest tests, we got the error message, that the Instance and Volume are in different AZ (mounting is of course forbidden). So the tests are failing.

Details: {u'message': u'Invalid volume: Instance 1234 and volume 4567 are not in the same availability_zone. Instance is in "AZ1". Volume is in "AZ2"', u'code': 400}

Tempest Tests:
tempest.scenario.test_shelve_instance.TestShelveInstance.test_shelve_volume_backed_instance
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_create_ebs_image_and_check_boot
tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern

To make the tests work again, it would be good to have a new param in tempest.conf to specify the AZ for the Instances and Volumes.

[Ghanshyam Mann (ghanshyammann)]

Nice bug, we actually did not care about cross_az_attach in tempest.

Almost all tests goes with default AZ which is same on nova and cinder side. So we actually never tested the cross AZ things.

may be we can add "cross_az_attach" in tempest conf and test can pass diff AZ if true otherwise same.
This way we can actually tests cross AZ things in our tests.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/408498

[Martin Kopec (mkopec)]

I'm little bit struggling with one detail here. Is it possible to have multiple valid zones configured for cinder and nova?

Example:
nova is in one availability zone called AZ1, cinder is in another zone called AZ2 and cross_az_attach in nova.conf is set to false.
I get that now the tests will fail, because instances will be in one zone (AZ1) and volumes in another (AZ2) as mentioned in the description of this LP.

The proposed change adds a new option of which value will be used as an availability_zone for creating an instance and a volume. This way both the instance and the volume would be in the same zone.
However, based on my testing, when I set the parameter to AZ1, volume creation failed with a message that AZ1 is not a valid zone (because valid zone for cinder is AZ2) and vice versa for nova when the parameter was set to AZ2.

So, the proposed change could work only in case when multiple availability zones can be configured for nova and for cinder.
Example: nova and cinder would be in availability zones called AZ1 and AZ2 and then in order to pass the tests we would use the new option to set the same zone for new instances and volumes created by the tests.

Therefore my final question I can't find the answer to, is it possible to have multiple availability zones for both nova and cinder?

[Martin Kopec (mkopec)]

For the record:
In order to play with this I created a new zone for nova by:

$ openstack aggregate create —zone AZ1 HA1
$ openstack aggregate create --zone AZ1 HA1
$ openstack aggregate add host HA1 compute-1.localdomain
$ openstack host list
+--------------------------+-------------+----------+
| Host Name | Service | Zone |
+--------------------------+-------------+----------+
| <output omitted> |
| compute-1.localdomain | compute | AZ1 |
| compute-0.localdomain | compute | nova |
+--------------------------+-------------+----------+

So I have the following zones:
$ openstack availability zone list --compute
+-----------+-------------+
| Zone Name | Zone Status |
+-----------+-------------+
| internal | available |
| AZ1 | available |
| nova | available |
+-----------+-------------+

$ openstack availability zone list --volume
+-----------+-------------+
| Zone Name | Zone Status |
+-----------+-------------+
| AZ1 | available |
+-----------+-------------+

Based on my testing the proposed patch fixes the issue regarding scenario tests. However, I noticed there are more tests which need to be edited. Basically every test which creates a volume and a server needs to get access to that new option. Otherwise it will fail with the same error the scenario tests failed.
I'm gonna send a follow up patch dependent on the one already proposed to fix also other non-scenario tests.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/663345

[Martin Kopec (mkopec)]

Here is output from my local environment. I was testing if https://review.opendev.org/#/c/408498 fixed the scenario tests.
Note: one test failed due to an unrelated issue.

[Martin Kopec (mkopec)]

Here is output from my local environment. I was testing if https://review.opendev.org/#/c/663345/ fixed the api tests.
Note: one test failed due to an unrelated issue.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/408498
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=979e494da246711751e97236e2681d6db9bd3e4e
Submitter: Zuul
Branch: master

commit 979e494da246711751e97236e2681d6db9bd3e4e
Author: Marc Koderer <email address hidden>
Date: Thu Dec 8 10:07:59 2016 +0100

    Specify availability zone to create instances and volumes

    For clouds with separate AZs (like two independent fire compartments)
    the cross_az_attach feature is typically used. But using this
    feature the scenario tests in tempest will fail since Cinder/Nova
    will schedule the AZ independently.

    This patch introduces a new parameter compute_volume_common_az
    to specify the availability zone to create instances and volumes,
    so that we can confirm that these resources are created in the same
    availability zone.

    Change-Id: I565adbe2a3ebbdd1d73143380b976df1a9502d0f
    Closes-bug: #1647999

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/663345
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=00e6d6c2e9982db5438e21ef29b5e7cee885dfcf
Submitter: Zuul
Branch: master

commit 00e6d6c2e9982db5438e21ef29b5e7cee885dfcf
Author: Martin Kopec <email address hidden>
Date: Wed Jun 5 14:30:06 2019 +0000

    Specify AZ for non-scenario tests

    The review allows related non-scenario tests which create
    volumes and servers to read and use
    CONF.compute.compute_volume_common_az.
    If the tests don't use the option and tested environment has
    multiple availability_zones, the tests fail.

    Related-bug: #1647999

    Change-Id: Ia0a6fd2175635d4375514d7609646b4516836dc0

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tempest 21.0.0 release.