Dell PowerFlex: Rescue instance if failing

[Jean Pierre Roquesalane]

When issuing a server rescue operation, the process failed with the following error:

Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] Traceback (most recent call last):
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/opt/stack/new/nova/nova/compute/manager.py", line 4583, in rescue_instance
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] self.driver.rescue(context, instance, network_info,
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4290, in rescue
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] xml = self._get_guest_xml(context, instance, network_info, disk_info,
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7506, in _get_guest_xml
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] conf = self._get_guest_config(instance, network_info, image_meta,
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 7021, in _get_guest_config
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] storage_configs = self._get_guest_storage_config(context,
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5589, in _get_guest_storage_config
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] self._connect_volume(context, connection_info, instance)
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1906, in _connect_volume
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] vol_driver.connect_volume(connection_info, instance)
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/opt/stack/new/nova/nova/virt/libvirt/volume/scaleio.py", line 55, in connect_volume
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] device_info = self.connector.connect_volume(connection_info['data'])
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/usr/local/lib/python3.8/dist-packages/os_brick/utils.py", line 184, in trace_logging_wrapper
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] result = f(*args, **kwargs)
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/usr/local/lib/python3.8/dist-packages/os_brick/utils.py", line 300, in change_encrypted
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] res = func(self, connection_properties)
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/base.py", line 78, in inner
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] return f(*args, **kwargs)
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] File "/usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/scaleio.py", line 419, in connect_volume
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] raise exception.BrickException(message=msg)
Mar 31 15:01:52 pflex153 nova-compute[787]: ERROR nova.compute.manager [instance: 9f3bc599-32d0-452a-8f92-9a4cd708d2b2] os_brick.exception.BrickException: Error mapping volume mQjJtVnaQVGnQ3QpUeepyA==: The requested volume is already mapped to the requested Host

That happens only for new PowerFlex 4.0 system as the code responsible for handling a response where we have the volume already attached is expecting an error code which changed from 3.6 to 4.0 system. This code will ignore the error if the RC is 81 and the rest of the rescue will end up as expected. With a different RC returned, the code is not ignoring it and it failed.

In 3.6 system:
Apr 01 00:07:34 l4-pflex134 nova-compute[120445]: DEBUG os_brick.initiator.connectors.scaleio [None req-5f32d2a4-cbb8-4029-90b6-6a985b971711 admin admin] ERROR CODE IS: 81 {{(pid=120445) connect_volume /opt/stack/new/os-brick/os_brick/initiator/connectors/scaleio.py:403}}

In 4.0 system:
Mar 31 20:10:35 pflex153 nova-compute[2534717]: DEBUG os_brick.initiator.connectors.scaleio [None req-77690abc-5e99-44ec-99c7-4f8d2215df77 admin admin] ERROR CODE IS: 4037 {{(pid=2534717) connect_volume /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/scaleio.py:405}}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/os-brick/+/885772

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/885772
Committed: https://opendev.org/openstack/os-brick/commit/42d9a892805225c13b87d5e68b6000fe01b7ae99
Submitter: "Zuul (22348)"
Branch: master

commit 42d9a892805225c13b87d5e68b6000fe01b7ae99
Author: tony-saad <email address hidden>
Date: Fri Jun 9 17:17:41 2023 +0000

    Dell Powerflex: Add new VOLUME_ALREADY_MAPPED_ERROR

    Dell Powerflex 4.x changed the error code of VOLUME_ALREADY_MAPPED_ERROR
    to 4037. This patch adds that error code.

    Closes-Bug: #2013749
    Change-Id: I928c97ea977f6d0a0b654f15c80c00523c141406

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/os-brick/+/896536

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/os-brick/+/896860

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/896536
Committed: https://opendev.org/openstack/os-brick/commit/c8e7dfbb47aeb846e8ca417e79eb56fd908e0b50
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit c8e7dfbb47aeb846e8ca417e79eb56fd908e0b50
Author: tony-saad <email address hidden>
Date: Fri Jun 9 17:17:41 2023 +0000

    Dell Powerflex: Add new VOLUME_ALREADY_MAPPED_ERROR

    Dell Powerflex 4.x changed the error code of VOLUME_ALREADY_MAPPED_ERROR
    to 4037. This patch adds that error code.

    Closes-Bug: #2013749
    Change-Id: I928c97ea977f6d0a0b654f15c80c00523c141406
    (cherry picked from commit 42d9a892805225c13b87d5e68b6000fe01b7ae99)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 6.5.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/896860
Committed: https://opendev.org/openstack/os-brick/commit/0655ebd93fe885d49681f69fdac0e2a9cc592bea
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 0655ebd93fe885d49681f69fdac0e2a9cc592bea
Author: tony-saad <email address hidden>
Date: Fri Jun 9 17:17:41 2023 +0000

    Dell Powerflex: Add new VOLUME_ALREADY_MAPPED_ERROR

    Dell Powerflex 4.x changed the error code of VOLUME_ALREADY_MAPPED_ERROR
    to 4037. This patch adds that error code.

    Closes-Bug: #2013749
    Change-Id: I928c97ea977f6d0a0b654f15c80c00523c141406
    (cherry picked from commit 42d9a892805225c13b87d5e68b6000fe01b7ae99)
    (cherry picked from commit c8e7dfbb47aeb846e8ca417e79eb56fd908e0b50)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 6.4.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/os-brick 6.2.3 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/os-brick/+/926039

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Nilesh Thathagar <email address hidden>" on branch: unmaintained/zed
Review: https://review.opendev.org/c/openstack/os-brick/+/926039

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/926039
Committed: https://opendev.org/openstack/os-brick/commit/e489d5f68d3d1d333721c140717892317c4912f0
Submitter: "Zuul (22348)"
Branch: unmaintained/zed

commit e489d5f68d3d1d333721c140717892317c4912f0
Author: tony-saad <email address hidden>
Date: Fri Jun 9 17:17:41 2023 +0000

    Dell Powerflex: Add new VOLUME_ALREADY_MAPPED_ERROR

    Dell Powerflex 4.x changed the error code of VOLUME_ALREADY_MAPPED_ERROR
    to 4037. This patch adds that error code.

    Closes-Bug: #2013749
    Change-Id: I928c97ea977f6d0a0b654f15c80c00523c141406
    (cherry picked from commit 42d9a892805225c13b87d5e68b6000fe01b7ae99)
    (cherry picked from commit c8e7dfbb47aeb846e8ca417e79eb56fd908e0b50)
    (cherry picked from commit 0655ebd93fe885d49681f69fdac0e2a9cc592bea)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: unmaintained/yoga
Review: https://review.opendev.org/c/openstack/os-brick/+/927374

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/os-brick/+/927374
Committed: https://opendev.org/openstack/os-brick/commit/3205c5663705a5028b7be79f43cb7325c51899e9
Submitter: "Zuul (22348)"
Branch: unmaintained/yoga

commit 3205c5663705a5028b7be79f43cb7325c51899e9
Author: tony-saad <email address hidden>
Date: Fri Jun 9 17:17:41 2023 +0000

    Dell Powerflex: Add new VOLUME_ALREADY_MAPPED_ERROR

    Dell Powerflex 4.x changed the error code of VOLUME_ALREADY_MAPPED_ERROR
    to 4037. This patch adds that error code.

    Closes-Bug: #2013749
    Change-Id: I928c97ea977f6d0a0b654f15c80c00523c141406
    (cherry picked from commit 42d9a892805225c13b87d5e68b6000fe01b7ae99)
    (cherry picked from commit c8e7dfbb47aeb846e8ca417e79eb56fd908e0b50)
    (cherry picked from commit 0655ebd93fe885d49681f69fdac0e2a9cc592bea)
    (cherry picked from commit e489d5f68d3d1d333721c140717892317c4912f0)