Regression in manila-netapp-dataontap driver TLS handling in Ussuri

[Steve]

Description
===========
As is common in deployments we have an internal root CA used for TLS certificates. We inject trust for this root into the kolla containers running OpenStack per normal kolla-ansible methods. in Train the Manila NetApp DataONTAP driver uses the system TLS trusted CAs and communication is fine (to the NetApp filers).

While testing our upgrade to Ussuri we noticed that manila driver could no longer communicate to the NetApp filers. Upon investigation it looks like it is now overriding the system trust store causing trust to fail

Steps to reproduce
==================
- put our introotCA.crt in /etc/kolla/certificates/ca
- did cert,manila configs in /etc/kolla/globals.yml:
kolla_copy_ca_into_containers: "yes"
enable_manila: "yes"
enable_manila_backend_generic: "yes"
- enabled the netapp driver in /etc/kolla/config/manila.conf:
[DEFAULT]
default_share_type = default_share_type
enabled_share_backends = generic,cdotSingleSVM

[cdotSingleSVM]
share_backend_name=cdotSingleSVM
share_driver = manila.share.drivers.netapp.common.NetAppDriver
...
- do the manila component reconfigure
kolla-ansible -i /etc/kolla/inventory -t manila reconfigure

Expected result
===============
NetApp driver would come up and be an available service

sample 'manila service-list output':
| 28 | manila-share | control01@cdotSingleSVM | nova | enabled | up |
| 31 | manila-share | control02@cdotSingleSVM | nova | enabled | up |
| 34 | manila-share | control03@cdotSingleSVM | nova | enabled | up |

Actual result
=============
NetApp driver is unavailable.

snip of 'manila service-list output':
| 28 | manila-share | control01@cdotSingleSVM | nova | enabled | down |
| 31 | manila-share | control02@cdotSingleSVM | nova | enabled | down |
| 34 | manila-share | control03@cdotSingleSVM | nova | enabled | down |

TLS trust errors in the manila-share log

Environment
===========
1. Exact version of OpenStack Manila you are running. See the following
manila: openstack-manila-10.0.1-1.el8
kolla-ansible: 10.1.0
container OS: CentOS 8

2. Which storage backend did you use?
NetApp DataONTAP 9.7

3. Which networking type did you use?
Neutron with OpenVSwitch

Analysis & Additional Information
=================================
Did a bit of debugging on this...

in:
/usr/lib/python3.6/site-packages/manila/share/drivers/netapp/dataontap/client/api.py

Noticed it is overriding the system and python system defaults to this value:
SSL_CERT_DEFAULT = "/etc/ssl/certs/"

It looks like this is likely related to this change:
https://bugs.launchpad.net/manila/+bug/1878993

That /etc/ssl location (at least on RedHat/CentOS servers) is a legacy location. the new location is /etc/pki/tls/. The OS does make /etc/ssl/certs be a symlink to /etc/pki/tls/certs.

However when you specify the SSL_CERT_DEFAULT (aka in the python code self._session.verify) it looks to be expecting the hash'd symlinks to a CA bundle isntead of the normal OS method. And in the RHEL/CentOS world at least, those hash symlinks were phased out a while back. In either 6 or 7 the new method is to put a CA you want trusted in /etc/pki/ca-trust/source/anchors/ and then run update-ca-trust. Which is what the kolla-ansible deployment code does with the steps listed above.

We did find that if SSL_CERT_DEFAULT is set to 'True' the python code again uses the OS method for CA trust and it works fine.

as an interim we have applied this patch to api.py and injected the updated code into the manila-share containers and have access to the NetApp driver again.

--- orig/api.py 2020-07-30 05:48:45.000000000 +0000
+++ patched/api.py 2020-10-07 22:54:28.107592418 +0000
@@ -62,7 +62,8 @@

     TRANSPORT_TYPE_HTTP = 'http'
     TRANSPORT_TYPE_HTTPS = 'https'
- SSL_CERT_DEFAULT = "/etc/ssl/certs/"
+ #SSL_CERT_DEFAULT = "/etc/ssl/certs/"
+ SSL_CERT_DEFAULT = True
     SERVER_TYPE_FILER = 'filer'
     SERVER_TYPE_DFM = 'dfm'
     URL_FILER = 'servlets/netapp.servlets.admin.XMLrequest_filer'

FYI to ibject we made a copy of the /usr/lib/python3.6/site-packages/manila/share/drivers/netapp/dataontap/client dir, applied the patch above and then added this to globals.yml:
manila_share_extra_volumes:
 - "/etc/ussuri-manila-netapp-client:/usr/lib/python3.6/site-packages/manila/share/drivers/netapp/dataontap/client"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/758641

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/758641
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=70f7859f986a48e53b117a3e16a11cb5454f74e3
Submitter: Zuul
Branch: master

commit 70f7859f986a48e53b117a3e16a11cb5454f74e3
Author: Felipe Rodrigues <email address hidden>
Date: Tue Oct 6 13:39:58 2020 +0000

    [NetApp] Fix hard-coded CA cert path for SSL

    NetApp driver is hard-coding the location of CA certificates for SSL
    verification during HTTPS requests. This location may change depending
    on the environment or/and backend.

    This patch adds the `netapp_ssl_cert_path` configuration, enabling
    each backend to choose the directory with certificates of trusted CA
    or the CA bundle. If set to a directory, it must have been processed
    using the c_rehash utility supplied with OpenSSL. If not informed,
    it will use the Mozilla's carefully curated collection of Root
    Certificates for validating the trustworthiness of SSL certificates.

    Closes-Bug: #1900191
    Change-Id: Idbed4745104de26af99bb16e07c6890637dfcfd1

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 12.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/772730
Committed: https://opendev.org/openstack/manila/commit/ac667519fcfaa62a700be71b7e0f23841ada42a5
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit ac667519fcfaa62a700be71b7e0f23841ada42a5
Author: Felipe Rodrigues <email address hidden>
Date: Tue Oct 6 13:39:58 2020 +0000

    [NetApp] Fix hard-coded CA cert path for SSL

    NetApp driver is hard-coding the location of CA certificates for SSL
    verification during HTTPS requests. This location may change depending
    on the environment or/and backend.

    This patch adds the `netapp_ssl_cert_path` configuration, enabling
    each backend to choose the directory with certificates of trusted CA
    or the CA bundle. If set to a directory, it must have been processed
    using the c_rehash utility supplied with OpenSSL. If not informed,
    it will use the Mozilla's carefully curated collection of Root
    Certificates for validating the trustworthiness of SSL certificates.

    Closes-Bug: #1900191
    Change-Id: Idbed4745104de26af99bb16e07c6890637dfcfd1
    (cherry picked from commit 70f7859f986a48e53b117a3e16a11cb5454f74e3)
    (cherry picked from commit 57edaadac18de5bcf6cc5564e2ca8ece2228d10d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/manila/+/772731
Committed: https://opendev.org/openstack/manila/commit/bf6c1d3fd4867423e763c0616e21af93532f3c69
Submitter: "Zuul (22348)"
Branch: stable/train

commit bf6c1d3fd4867423e763c0616e21af93532f3c69
Author: Felipe Rodrigues <email address hidden>
Date: Tue Oct 6 13:39:58 2020 +0000

    [NetApp] Fix hard-coded CA cert path for SSL

    NetApp driver is hard-coding the location of CA certificates for SSL
    verification during HTTPS requests. This location may change depending
    on the environment or/and backend.

    This patch adds the `netapp_ssl_cert_path` configuration, enabling
    each backend to choose the directory with certificates of trusted CA
    or the CA bundle. If set to a directory, it must have been processed
    using the c_rehash utility supplied with OpenSSL. If not informed,
    it will use the Mozilla's carefully curated collection of Root
    Certificates for validating the trustworthiness of SSL certificates.

    Closes-Bug: #1900191
    Change-Id: Idbed4745104de26af99bb16e07c6890637dfcfd1
    (cherry picked from commit 70f7859f986a48e53b117a3e16a11cb5454f74e3)
    (cherry picked from commit 57edaadac18de5bcf6cc5564e2ca8ece2228d10d)
    (cherry picked from commit ac667519fcfaa62a700be71b7e0f23841ada42a5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 10.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila train-eol release.