test_instance_failover consistently fails

[Aurelien Lourot]

Visible in this gate (see stein and wallaby):
https://review.opendev.org/c/openstack/charm-masakari/+/814188
https://openstack-ci-reports.ubuntu.com/artifacts/2bc/814188/2/check/bionic-stein-pacemaker-remote-ssl_masakari/2bce27b/job-output.txt

test_instance_failover (zaza.openstack.charm_tests.masakari.tests.MasakariTest)
Test masakari managed guest migration.
 ...
Node juju-a44531-zaza-dbc55be41148-10 of type member is in online state True
Node juju-a44531-zaza-dbc55be41148-11 of type member is in online state True
Node juju-a44531-zaza-dbc55be41148-16.project.serverstack of type remote is in online state True
Node juju-a44531-zaza-dbc55be41148-17.project.serverstack of type remote is in online state True
Node juju-a44531-zaza-dbc55be41148-18.project.serverstack of type remote is in online state True
Node juju-a44531-zaza-dbc55be41148-9 of type member is in online state True
Launching new guest
looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}}
Using keystone API V3 (or later) for overcloud auth
looking at application: {'name': 'keystone', 'type': {'pkg': 'keystone', 'origin_setting': 'openstack-origin'}}
Launching instance zaza-test-instance-failover
Checking instance is active
resource: resource 02be9b1e-789d-442f-98e9-27b039fa83dd in BUILD state, waiting for ACTIVE
resource: resource 02be9b1e-789d-442f-98e9-27b039fa83dd in BUILD state, waiting for ACTIVE
resource: resource 02be9b1e-789d-442f-98e9-27b039fa83dd in BUILD state, waiting for ACTIVE
resource: resource 02be9b1e-789d-442f-98e9-27b039fa83dd in BUILD state, waiting for ACTIVE
resource: resource 02be9b1e-789d-442f-98e9-27b039fa83dd in BUILD state, waiting for ACTIVE
resource: resource 02be9b1e-789d-442f-98e9-27b039fa83dd in ERROR state, waiting for ACTIVE

nova-compute.log:
2021-10-20 16:29:58.808 16835 ERROR nova.compute.manager [req-e52007e0-d6a6-44f8-9135-a957173afac4 e4e7c3a0c3cc475c91290b98eb3d6fd7 32b0e775f31e4c6cb5632b606436b5aa - aaea89edac4840eeb800bc9d594abe0a aaea89edac4840eeb800bc9d594abe0a] [instance: 02be9b1e-789d-442f-98e9-27b039fa83dd] Build of instance 02be9b1e-789d-442f-98e9-27b039fa83dd aborted: Volume a5e177c4-fb0c-465d-997a-2e9d87adbaa9 did not finish being created even after we waited 3 seconds or 1 attempts. And its status is error.: nova.exception.BuildAbortException: Build of instance 02be9b1e-789d-442f-98e9-27b039fa83dd aborted: Volume a5e177c4-fb0c-465d-997a-2e9d87adbaa9 did not finish being created even after we waited 3 seconds or 1 attempts. And its status is error.

cinder-scheduler.log:
2021-10-20 16:29:48.077 12047 INFO cinder.scheduler.base_filter [req-734c7200-55f4-46b8-9497-73d85dac83f8 e4e7c3a0c3cc475c91290b98eb3d6fd7 32b0e775f31e4c6cb5632b606436b5aa - aaea89edac4840eeb800bc9d594abe0a aaea89edac4840eeb800bc9d594abe0a] Filtering removed all hosts for the request with volume ID 'a5e177c4-fb0c-465d-997a-2e9d87adbaa9'. Filter results: AvailabilityZoneFilter: (start: 0, end: 0), CapacityFilter: (start: 0, end: 0), CapabilitiesFilter: (start: 0, end: 0)

[Corey Bryant (corey.bryant)]

This needs some further testing to see why the cinder scheduler filter is removing all hosts.