test_create_volume fails because the created volume is in error state

[Aurelien Lourot]

Visible in this gate:
https://review.opendev.org/c/openstack/charm-cinder-purestorage/+/814298
https://openstack-ci-reports.ubuntu.com/artifacts/987/814298/1/check/bionic-ussuri/987b59e/job-output.txt

test_create_volume (tests.tests_cinder_purestorage.CinderpurestorageTest)
 ...
resource: resource 02096d8b-518b-45b5-a952-ff44ac6fcb38 in error state, waiting for available
resource: resource 02096d8b-518b-45b5-a952-ff44ac6fcb38 in error state, waiting for available
resource: resource 02096d8b-518b-45b5-a952-ff44ac6fcb38 in error state, waiting for available
resource: resource 02096d8b-518b-45b5-a952-ff44ac6fcb38 in error state, waiting for available
resource: resource 02096d8b-518b-45b5-a952-ff44ac6fcb38 in error state, waiting for available
resource: resource 02096d8b-518b-45b5-a952-ff44ac6fcb38 in error state, waiting for available
resource: resource 02096d8b-518b-45b5-a952-ff44ac6fcb38 in error state, waiting for available
resource: resource 02096d8b-518b-45b5-a952-ff44ac6fcb38 in error state, waiting for available
FAIL

NOTE(coreycb): I've found this to be intermittent.

[Aurelien Lourot (aurelien-lourot)]

cinder-volume.log is full of:

2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume [req-925c345a-24e0-4b00-b1fd-52d35dc9df3f - - - - -] Volume service juju-26537c-zaza-419601a9c3f9-2@LVM failed to start.: cinder.exception.VolumeDriverException: Volume driver reported an error: Active-Active configuration is not currently supported by driver cinder.volume.drivers.lvm.LVMVolumeDriver.
2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume Traceback (most recent call last):
2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/cinder/cmd/volume.py", line 104, in _launch_service
2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume cluster=cluster)
2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/cinder/service.py", line 400, in create
2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume cluster=cluster, **kwargs)
2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/cinder/service.py", line 155, in __init__
2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume *args, **kwargs)
2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 273, in __init__
2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume raise exception.VolumeDriverException(message=msg)
2021-10-19 11:48:44.287 31640 ERROR cinder.cmd.volume cinder.exception.VolumeDriverException: Volume driver reported an error: Active-Active configuration is not currently supported by driver cinder.volume.drivers.lvm.LVMVolumeDriver.

[Felipe Reyes (freyes)]

https://github.com/openstack/cinder/commit/0f04930fb365cdb7723bd42e1addda3333e1d499

https://review.opendev.org/q/I53ab2c8dae43d5aef7a65d35ad99e55f0acb7431

This issue is fixed in OpenStack >=Victoria. This is the change the introduced the use of "cluster" instead of "host" for >=Ocata[0], I think the subordinate should inform the principle if it's capable of running in clustered mode just like they inform stateless=True/False.

[0] https://review.opendev.org/c/openstack/charm-cinder/+/811472

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/814845

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder-purestorage/+/814851

[Felipe Reyes (freyes)]

Fix proposed: https://review.opendev.org/c/openstack/charms.openstack/+/814850

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Felipe Reyes <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/814845
Reason: Instead of adding a new key the 'stateless' flag should be used.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

I added a comment in the abandoned change above discussing the solution for this bug

Also note that the LVM driver error indicates that this CI is misconfigured, it shouldn't be trying to initialize the LVM driver.

[Rodrigo Barbieri (rodrigo-barbieri2010)]

We are addressing this bug with the following patches:

[0] https://review.opendev.org/c/openstack/charm-cinder/+/814845
[1] https://review.opendev.org/c/openstack/charm-cinder-purestorage/+/814851

In [0] we are adding a new flag that mixed with stateless allows us to in [1] intentionally maintain the old host=<same_value> behavior that exists today due to [2]. New deployments should use >= Victoria Openstack releases so they will be Active-Active with 'cluster' config, as coded in [1].

[2] https://github.com/openstack/charm-cinder-purestorage/blob/0d3643274a4a14cef8705389a5912b0ca124bd36/src/lib/charm/openstack/cinder_purestorage.py#L14

[Corey Bryant (corey.bryant)]

It seems we're hitting this with test_100_volume_create_extend_delete as well (as seen at https://review.opendev.org/c/openstack/charm-cinder/+/814177):

test_100_volume_create_extend_delete (zaza.openstack.charm_tests.cinder.tests.CinderTests)
Test creating, extending a volume.
 ...
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available
Volume status wait: resource 56746836-6939-4f58-a9a2-bb2465fc7935 in error state, waiting for available

[Corey Bryant (corey.bryant)]

This may be fixed by:

commit c5896a884a54d0171415d79a82a284e2ff6b2ca9
Author: Billy Olsen <email address hidden>
Date: Thu Oct 21 18:32:07 2021 -0700

    Revert "Group cinder-volume services using cluster config"

    This reverts commit 2603a774b112dc6e25f9e7b2071124ea3d3d0012, which
    converted cinder to use cluster configuration for stateless services.
    While well-intentioned, this causes problems due to the lack of a
    proper migration for those backends which are not able to support
    ACTIVE-ACTIVE configuration.

    Configurations that do not support ACTIVE_ACTIVE driver will have the
    volume service fail to start, which will cause a service outage.

    Change-Id: I7bf4baaf80e5bb58b5c1cf55a2065f3bc50dbced
    Related-Bug: #1945239

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder-purestorage/+/821986

[Corey Bryant (corey.bryant)]

All tests ran successfully in https://review.opendev.org/c/openstack/charm-cinder-purestorage/+/821986 so it seems the patch revert shown above in comment #11 fixed the issue.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Corey Bryant <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder-purestorage/+/821986
Reason: This was just a test to see if the gate tests were successful