python-cephclient disables urllib3 certificate checks

[Daniel Badea]

Brief Description
-----------------
In StarlingX ceph-mgr restful plugin is using self-signed certificate when providing HTTPS access to Ceph REST API. Instead of retrieving and using this certificate python-cephclient takes a shortcut and disables verifying HTTPS requests for the entire requests/urllib3 library. This puts any application using python-cephclient at risk.

Cephclient warns about 'skip checking server certificate' and this message can be seen in the output of config_controller and in sysinv log. It was meant to be temporary shortcut and needs to be replaced with proper handling of ceph-mgr restful plugin HTTPS certificates.

Severity
--------
Minor

Steps to Reproduce
------------------
On active controller run:

python -c 'from cephclient.wrapper import CephWrapper; CephWrapper().osd_df()' 2>&1 | grep WARNING

Expected Behavior
------------------
No warning displayed and urllib3.disable_warnings() not called.

Actual Behavior
----------------
2019-05-09 20:38:26,953 WARNING ceph_client skip checking server certificate

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
All systems.

Branch/Pull Time/Commit
-----------------------
master 2019-05-08

Last Pass
---------
N/A

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Developer Testing.

[Daniel Badea (daniel.badea)]

To confirm urllib3.disable_warnins is not called without relying on log messages:

python <<EOF
import requests.packages.urllib3
from cephclient.wrapper import CephWrapper
def override(calls=[], query=False,
             func=requests.packages.urllib3.disable_warnings):
    if query:
        return len(calls)
    return calls.append(func())
requests.packages.urllib3.disable_warnings=override
CephWrapper().osd_df()
assert(requests.packages.urllib3.disable_warnings(query=True) == 0)
EOF

should not print AssertionError.

(python mock is not available on controller; this snippet patches disable_warnings(), calls CephWrapper() where HTTPS certificate checks are disabled then asserts disable_warnings() was not called)

[Daniel Badea (daniel.badea)]

To retrieve ceph-mgr restful plugin self-signed certificate:

  openssl x509 -in <(openssl s_client -connect controller-0:5001 -prexit 2>/dev/null) > ceph-restful.pem

However the the certificate is bound to 'ceph-restful' host:

  curl --cacert ceph-restful.pem https://controller:5001

fails so a host alias needs to be defined:

  sudo sed -i 's/\(\w\+\s\+controller\)\s*$/\1\tceph-restful/' /etc/hosts

and now (note the new host name):

  curl --cacert ceph-restful.pem https://ceph-restful:5001

is successful.

Another approach is to generate a certificate and configure ceph-mgr restful plugin to use that certificate instead of generating it. But then we need to manage that certificate and make it available on both controllers.

[Daniel Badea (daniel.badea)]

Created upstream bug https://tracker.ceph.com/issues/40013

[Cindy Xie (xxie1)]

change to Low as this is not stx.2.0 gating according to last week's bug triage.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/680915

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/680920

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/680915
Committed: https://git.openstack.org/cgit/starlingx/integ/commit/?id=fcaa49ecaf002b97e450cb9e8d41b9d2c2094d67
Submitter: Zuul
Branch: master

commit fcaa49ecaf002b97e450cb9e8d41b9d2c2094d67
Author: Daniel Badea <email address hidden>
Date: Fri Sep 6 15:12:46 2019 +0000

    ceph: mgr restful plugin set certificate to match host name

    python-cephclient certificate validation fails when connecting
    to ceph-mgr restful plugin because server URL doesn't match
    CommonName (CN) or SubjectAltName (SAN).

    Setting CN to match server hostname fixes this issue but
    raises a warning caused by missing SAN.

    Using CN=ceph-restful and SAN=<hostname> fixes the issue
    and clears the warning.

    Change-Id: I6e8ca93c7b51546d134a6eb221c282961ba50afa
    Closes-bug: 1828470
    Signed-off-by: Daniel Badea <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/680920
Committed: https://git.openstack.org/cgit/starlingx/utilities/commit/?id=9f73cd4f9bf4a8f284642b7940ac41d22b637d3e
Submitter: Zuul
Branch: master

commit 9f73cd4f9bf4a8f284642b7940ac41d22b637d3e
Author: Daniel Badea <email address hidden>
Date: Fri Sep 6 15:12:46 2019 +0000

    python-cephclient: use configured restful api plugin

    ceph-mgr restful plugin is using self-signed certificate when providing
    HTTPS access to Ceph REST API.

    Instead of retrieving and using this certificate python-cephclient is
    currently a shortcut and disables verifying HTTPS requests for the
    entire requests/urllib3 library. This was meant to be temporary shortcut
    until proper handling of ceph-mgr restful plugin HTTPS certificates is
    implemented.

    This commit implements automatic python-cephclient restful plugin
    certificate retrieval such that it is no longer necessary to disable
    requests/urllib3 certificates verification.

    Two options were available:

    1. provide path to certificate file when creating an instance of
       CephClient() or CephWrapper() then use that value when creating
       a request session ('verify' attribute).

       This delegates the responsibility of providing a valid certificate to
       the caller/user of python-cephclient library. Because it implies an
       API update all StarlingX components using python-cephclient need to
       be updated.

       The certificate file itself is created when mgr-restful-plugin
       is started before ceph-mgr restful plugin service is configured
       to use it.

    2. add support for retrieving the certificate by using 'ceph' commands
       similar to how user credentials and restful plugin endpoint are
       discovered.

       If there is an error in getting the certificate then the session
       certificate verification is temporarily disabled until the next
       request is made. This means that if the corresponding Ceph config-key
       'mgr/restful/{hostname}/crt' is removed then python-cephclient will
       incur the overhead of running 'ceph config-key get' before each
       request but this is an unlikely scenario in our case.

    Option #2 was selected because it doesn't change existing API.

    Change-Id: I68acb3e1d2fb8e2bb07c8d67e65b02d55a6716ca
    Depends-on: I6e8ca93c7b51546d134a6eb221c282961ba50afa
    Closes-bug: 1828470
    Signed-off-by: Daniel Badea <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/682472

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682472
Committed: https://git.openstack.org/cgit/starlingx/utilities/commit/?id=613276c944504b972a38dec3618137ea81dc0dc9
Submitter: Zuul
Branch: master

commit 613276c944504b972a38dec3618137ea81dc0dc9
Author: Daniel Badea <email address hidden>
Date: Mon Sep 16 19:51:46 2019 +0000

    python-cephclient: fix python 3 tox

    Fix python 3 tox error caused urlparse import in commit
    9f73cd4f9bf4a8f284642b7940ac41d22b637d3e "python-cephclient:
    use configured restful api plugin"

    Closes-bug: 1828470
    Signed-off-by: Daniel Badea <email address hidden>
    Change-Id: I58c338cec72132aade7a4b4c1e2abeb887b27c86

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/682619

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/682619
Committed: https://git.openstack.org/cgit/starlingx/utilities/commit/?id=2e7d2550b9c64a33910a567c3433a12f3e68f25a
Submitter: Zuul
Branch: master

commit 2e7d2550b9c64a33910a567c3433a12f3e68f25a
Author: Daniel Badea <email address hidden>
Date: Tue Sep 17 11:10:22 2019 +0000

    python-cephclient: fix urlparse function call

    urlparse import for python3 adds urlparse function to the
    global namespace instead of a module name. Existing code
    calling urlparse.urlparse() fails because of that.

    Replace urlparse.urlparse() with urlparse().

    Change-Id: I61c749b9beb8b5a09e82c8bc93087348b2c20d73
    Closes-bug: 1828470
    Signed-off-by: Daniel Badea <email address hidden>