Dell PowerFlex: Failed to attach a volume when using self-signed Certificates

[Jean Pierre Roquesalane]

Initially before the change
https://review.opendev.org/c/openstack/os-brick/+/810419 was merged
to close the bug https://bugs.launchpad.net/os-brick/+bug/1929223,
verify_cert was always set to False which can lead to security issues.
It has been decided through this change that this option can be set
to True or False based upon security requirements. This change
introduced a regression failure as the value set to the option
is not part of connection_properties.

Below is the error thrown in the cinder log

Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [None req-c72aa76a-811d-4463-bb54-413f335f31f7 demo admin] [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] Driver failed to attach volume 048a3748-8517-48df-9d37-e16ec2789408 at /dev/vdb: requests.exceptions.SSLError: HTTPSConnectionPool(host='10.234.219.75', port=443): Max retries exceeded with url: /api/instances/Volume::8b323a2100000016/action/addMappedSdc (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] Traceback (most recent call last):
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] httplib_response = self._make_request(
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 386, in _make_request
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] self._validate_conn(conn)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 1042, in _validate_conn
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] conn.connect()
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/urllib3/connection.py", line 414, in connect
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] self.sock = ssl_wrap_socket(
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/urllib3/util/ssl_.py", line 453, in ssl_wrap_socket
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/urllib3/util/ssl_.py", line 495, in _ssl_wrap_socket_impl
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] return ssl_context.wrap_socket(sock)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/eventlet/green/ssl.py", line 446, in wrap_socket
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] return GreenSSLSocket(sock, *a, _context=self, **kw)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/eventlet/green/ssl.py", line 140, in __init__
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] self.do_handshake()
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/eventlet/green/ssl.py", line 312, in do_handshake
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] return self._call_trampolining(
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/eventlet/green/ssl.py", line 162, in _call_trampolining
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] return func(*a, **kw)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] self._sslobj.do_handshake()
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78]
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] During handling of the above exception, another exception occurred:
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78]
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] Traceback (most recent call last):
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 489, in send
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] resp = conn.urlopen(
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] retries = retries.increment(
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/urllib3/util/retry.py", line 592, in increment
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] raise MaxRetryError(_pool, url, error or ResponseError(cause))
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='10.234.219.75', port=443): Max retries exceeded with url: /api/instances/Volume::8b323a2100000016/action/addMappedSdc (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78]
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] During handling of the above exception, another exception occurred:
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78]
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] Traceback (most recent call last):
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/opt/stack/nova/nova/virt/block_device.py", line 657, in _volume_attach
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] virt_driver.attach_volume(
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2158, in attach_volume
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] self._connect_volume(context, connection_info, instance,
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1908, in _connect_volume
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] vol_driver.connect_volume(connection_info, instance)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/opt/stack/nova/nova/virt/libvirt/volume/scaleio.py", line 55, in connect_volume
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] device_info = self.connector.connect_volume(connection_info['data'])
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/os_brick/utils.py", line 184, in trace_logging_wrapper
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] result = f(*args, **kwargs)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/os_brick/utils.py", line 300, in change_encrypted
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] res = func(self, connection_properties)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/base.py", line 78, in inner
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] return f(*args, **kwargs)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/scaleio.py", line 391, in connect_volume
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] r = requests.post(
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/requests/api.py", line 115, in post
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] return request("post", url, data=data, json=json, **kwargs)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/requests/api.py", line 59, in request
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] return session.request(method=method, url=url, **kwargs)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 587, in request
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] resp = self.send(prep, **send_kwargs)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/requests/sessions.py", line 701, in send
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] r = adapter.send(request, **kwargs)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] File "/usr/local/lib/python3.8/dist-packages/requests/adapters.py", line 563, in send
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] raise SSLError(e, request=request)
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] requests.exceptions.SSLError: HTTPSConnectionPool(host='10.234.219.75', port=443): Max retries exceeded with url: /api/instances/Volume::8b323a2100000016/action/addMappedSdc (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.virt.block_device [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78]
Sep 19 11:18:52 SRA-Left-Openstack1 nova-compute[914]: ERROR nova.compute.manager [None req-c72aa76a-811d-4463-bb54-413f335f31f7 demo admin] [instance: 49c01344-f034-4ac1-8fc8-a486dfe24e78] Failed to attach 048a3748-8517-48df-9d37-e16ec2789408 at /dev/vdb: requests.exceptions.SSLError: HTTPSConnectionPool(host='10.234.219.75', port=443): Max retries exceeded with url: /api/instances/Volume::8b323a2100000016/action/addMappedSdc (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)')))

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/858370

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/858370
Committed: https://opendev.org/openstack/cinder/commit/82823ace4d714ac10427ea3c6fed320c27b56f7d
Submitter: "Zuul (22348)"
Branch: master

commit 82823ace4d714ac10427ea3c6fed320c27b56f7d
Author: happystacker <email address hidden>
Date: Mon Sep 19 08:55:24 2022 +0200

    Dell PowerFlex: Additionnal params for enabling self signed certificates

    Initially before the change
    https://review.opendev.org/c/openstack/os-brick/+/810419 was merged
    to close the bug https://bugs.launchpad.net/os-brick/+bug/1929223,
    verify_cert was always set to False which can lead to security issues.
    It has been decided through this change that this option can be set
    to True or False based upon security requirements. This change
    introduced a regression failure as the value set to the option
    is not part of connection_properties.

    This patch adds additional params during initialization so that it
    can be carried over os-brick and get adequate REST API response.

    Closes-Bug: 1990136
    Change-Id: I0d266a57f68221a3b1740a7376e152bb64cac729

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/cinder/+/880492

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 23.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/cinder/+/909251

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/909251
Committed: https://opendev.org/openstack/cinder/commit/8794f84e77435a4d86d0c08980fff18a80151dc2
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit 8794f84e77435a4d86d0c08980fff18a80151dc2
Author: happystacker <email address hidden>
Date: Mon Sep 19 08:55:24 2022 +0200

    Dell PowerFlex: Additionnal params for enabling self signed certificates

    Initially before the change
    https://review.opendev.org/c/openstack/os-brick/+/810419 was merged
    to close the bug https://bugs.launchpad.net/os-brick/+bug/1929223,
    verify_cert was always set to False which can lead to security issues.
    It has been decided through this change that this option can be set
    to True or False based upon security requirements. This change
    introduced a regression failure as the value set to the option
    is not part of connection_properties.

    This patch adds additional params during initialization so that it
    can be carried over os-brick and get adequate REST API response.

    Closes-Bug: 1990136
    Change-Id: I0d266a57f68221a3b1740a7376e152bb64cac729
    (cherry picked from commit 82823ace4d714ac10427ea3c6fed320c27b56f7d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 22.1.2 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/cinder/+/880492
Reason: stable/zed branch of openstack/cinder is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/zed if you want to further work on this patch.