[Train] certificate verify failed for Zaqar websocket when importing baremetal nodes

[Sagi (Sergey) Shnaidman]

When running importing baremetal nodes with tripleo operator, there is error with Zaqar SSL certificate verification.

Operator runs as:

export OS_CLOUD=undercloud
openstack overcloud node import instackenv.json >/home/zuul/overcloud_import_nodes.log 2>&1

https://logserver.rdoproject.org/88/727688/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train-branch/2c503e3/logs/undercloud/home/zuul/tripleo_overcloud_node_import.sh.txt.gz

Output is:

Could not establish a connection to the Zaqar websocket. The command was sent but the answer could not be read.
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/site-packages/cliff/command.py", line 185, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_node.py", line 412, in take_action
    instance_boot_option=parsed_args.instance_boot_option
  File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/baremetal.py", line 62, in register_or_update
    with tripleoclients.messaging_websocket() as ws:
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 216, in messaging_websocket
    cacert=self._instance.cacert)
  File "/usr/lib/python2.7/site-packages/tripleoclient/plugin.py", line 91, in __init__
    self._ws = websocket.create_connection(endpoint)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 511, in create_connection
    websock.connect(url, **options)
  File "/usr/lib/python2.7/site-packages/websocket/_core.py", line 220, in connect
    options.pop('socket', None))
  File "/usr/lib/python2.7/site-packages/websocket/_http.py", line 126, in connect
    sock = _ssl_socket(sock, options.sslopt, hostname)
  File "/usr/lib/python2.7/site-packages/websocket/_http.py", line 256, in _ssl_socket
    sock = ssl.wrap_socket(sock, **sslopt)
  File "/usr/lib64/python2.7/ssl.py", line 934, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib64/python2.7/ssl.py", line 609, in __init__
    self.do_handshake()
  File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

https://logserver.rdoproject.org/88/727688/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-train-branch/2c503e3/logs/undercloud/home/zuul/overcloud_import_nodes.log.txt.gz

It happens on train branch only, master works fine.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/727959

[Rabi Mishra (rabi)]

Looks like we don't have cacert in clouds.yaml which is needed for zaqar's websocket connection.
https://github.com/openstack/python-tripleoclient/blob/stable/train/tripleoclient/plugin.py#L86

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/728358
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=271c82dbd775e6ca6c20655ae3f781593d86ccb4
Submitter: Zuul
Branch: master

commit 271c82dbd775e6ca6c20655ae3f781593d86ccb4
Author: Rabi Mishra <email address hidden>
Date: Fri May 15 10:31:58 2020 +0530

    Add InternalTLSCAFile param when enabling overcloud ssl

    This is used by the services and without setting this
    paramter it would pick up the template default which
    does not work.

    Related-Bug :#1878540
    Change-Id: Ia23e4336752bd639f357e036baad3aa0cf6cbf74

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/727959
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=42cfbbc8bfbaa401d6e774b58cb78f4bbaccc2d9
Submitter: Zuul
Branch: master

commit 42cfbbc8bfbaa401d6e774b58cb78f4bbaccc2d9
Author: Rabi Mishra <email address hidden>
Date: Thu May 14 13:03:58 2020 +0530

    Add cacert to clouds.yaml

    We need to add the cacert for both undercloud and overclud
    in clouds.yaml

    Closes-Bug: #1878540
    Depends-On: https://review.opendev.org/728358
    Change-Id: I1f209bcae7707af2c8653ad21f69097f81ec6947

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/729090

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/729090
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=405ebda2b192168b174c769d09ace2fcd63a706f
Submitter: Zuul
Branch: stable/train

commit 405ebda2b192168b174c769d09ace2fcd63a706f
Author: Rabi Mishra <email address hidden>
Date: Thu May 14 13:03:58 2020 +0530

    Add cacert to clouds.yaml

    We need to add the cacert for both undercloud and overclud
    in clouds.yaml

    Closes-Bug: #1878540
    Depends-On: https://review.opendev.org/728358
    Change-Id: I1f209bcae7707af2c8653ad21f69097f81ec6947
    (cherry picked from commit 42cfbbc8bfbaa401d6e774b58cb78f4bbaccc2d9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.