Zaqar websocket ssl failures

[Jill Rouleau]

ssl requests to zaqar websocket fail.

(undercloud) [stack@undercloud ~]$ openstack overcloud config download
Starting config-download export...
Could not establish a connection to the Zaqar websocket. The command was sent but the answer could not be read.
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)

(undercloud) [stack@undercloud ~]$ openstack messaging queue list
[SSL] PEM lib (_ssl.c:2554)

cert itself is fine,
(undercloud) [stack@undercloud ~]$ openssl s_client -connect 192.168.24.2:9000 -CAfile /etc/pki/ca-trust/source/anchors/cm-local-ca.pem
CONNECTED(00000003)
depth=1 CN = Local Signing Authority, CN = 182a169e-a2b54e0b-aac0bace-de84d39d
verify return:1
depth=0 CN = 192.168.24.2
verify return:1
---
Certificate chain
 0 s:/CN=192.168.24.2
   i:/CN=Local Signing Authority/CN=182a169e-a2b54e0b-aac0bace-de84d39d
 1 s:/CN=Local Signing Authority/CN=182a169e-a2b54e0b-aac0bace-de84d39d
   i:/CN=Local Signing Authority/CN=182a169e-a2b54e0b-aac0bace-de84d39d
---
Server certificate

...

(undercloud) [stack@undercloud ~]$ curl -g -i --cacert "/etc/pki/ca-trust/source/anchors/cm-local-ca.pem" -X GET https://192.168.24.2:9000/v2/queues -H "User-Agent: openstacksdk
/0.16.0 keystoneauth1/3.9.0 python-requests/2.19.1 CPython/2.7.5" -H "X-Auth-Token: gAAAAABbl9unyu8y8caBo5S3KWVgSVVoAzfMjsSlJnBVgxabBTpu1VSUu8JOUYXTCGIuYsIcMkGf2DVjj0tVhoGJA-f3S
T91XyXt0SxMCKdxMp-ZvGct4_oTwBbDXZUCqp0-KPsPombgMb-DjUnGkb_8q5Nru7j2OEeWEesHYq4kx95LUUKh1k4"
HTTP/1.1 200 OK
Server: AutobahnPython/0.10.9
Content-Type: text/html; charset=UTF-8
Content-Length: 820

Source RPM : python-zaqarclient-1.10.0-0.20180806142547.1a50023.el7.src.rpm

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/601677

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/601677
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=6639b9e5643aeaee7a17b7995fddc2fb6288b926
Submitter: Zuul
Branch: master

commit 6639b9e5643aeaee7a17b7995fddc2fb6288b926
Author: Jill Rouleau <email address hidden>
Date: Tue Sep 11 12:01:01 2018 -0600

    Handle tls endpoint for zaqar websocket client

    When creating zaqar websocket client, if the endpoint has tls enabled
    provide the CA to the client.

    Closes-Bug: 1791970
    Change-Id: I09fca4ea80ae8246f136ea6998dfc7ad1c6bb4d2

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/633024

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633024
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=550c1a4325cefc67f6d3b2e7049c5191b0aeca4b
Submitter: Zuul
Branch: stable/rocky

commit 550c1a4325cefc67f6d3b2e7049c5191b0aeca4b
Author: Jill Rouleau <email address hidden>
Date: Tue Sep 11 12:01:01 2018 -0600

    Handle tls endpoint for zaqar websocket client

    When creating zaqar websocket client, if the endpoint has tls enabled
    provide the CA to the client.

    Closes-Bug: 1791970
    Change-Id: I09fca4ea80ae8246f136ea6998dfc7ad1c6bb4d2
    (cherry picked from commit 6639b9e5643aeaee7a17b7995fddc2fb6288b926)

[Mike Bayer (zzzeek)]

this is creating a problem for me when using infrared.

Per https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ssl.html, "However, it is possible to not use certmonger’s local CA.". When I use infrared to make an undercloud, it creates a new self-signed CA defaulting to /etc/pki/ca-trust/source/anchors/undercloud-cacert.pem; this code is at https://github.com/redhat-openstack/infrared/blob/master/plugins/tripleo-undercloud/tasks/ssl.yml#L32 . Then it sets "undercloud_service_certificate = /etc/pki/instack-certs/undercloud.pem" in undercloud.conf and deploys the undercloud, following the instructions in https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/ssl.html.

However, when I go to introspect nodes, I get the above stack trace again:

Could not establish a connection to the Zaqar websocket. The command was sent but the answer could not be read.
# ...
 File "/usr/lib64/python2.7/ssl.py", line 831, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

to repair this, I just copy the undercloud-cacert.pem on top of the cm-local-ca.pem file:

[root@undercloud-0 ~]# cd /etc/pki/ca-trust/source/anchors/
[root@undercloud-0 anchors]# ls
cm-local-ca.pem undercloud-cacert.pem
[root@undercloud-0 anchors]# cp cm-local-ca.pem cm-local-ca.pem.saved
[root@undercloud-0 anchors]# cp undercloud-cacert.pem cm-local-ca.pem
cp: overwrite ‘cm-local-ca.pem’? y

and then it works. But this seems really wrong. What is the relationship supposed to be between the hardcoded path added here and the instructions for using "undercloud_service_certificate" in undercloud.conf ? is the bug on the infrared side? I don't know anything about this area, I'm just following my nose through an error I'm getting as I tend to be using infrared with RDO sooner than other folks.

[Jill Rouleau (jillrouleau)]

@zzzeek, could you open a new bug for your issue please?

[Mike Bayer (zzzeek)]

Sure thing! didnt see your reply

[Mike Bayer (zzzeek)]

created https://bugs.launchpad.net/tripleo/+bug/1817634

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 10.6.1 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/661582

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/661582
Committed: https://git.openstack.org/cgit/openstack/python-tripleoclient/commit/?id=7f8d8999d3b0cdc182d52ae83d7539679e71c1ea
Submitter: Zuul
Branch: stable/queens

commit 7f8d8999d3b0cdc182d52ae83d7539679e71c1ea
Author: Jill Rouleau <email address hidden>
Date: Tue Sep 11 12:01:01 2018 -0600

    Handle tls endpoint for zaqar websocket client

    When creating zaqar websocket client, if the endpoint has tls enabled
    provide the CA to the client.

    Closes-Bug: 1791970
    Change-Id: I09fca4ea80ae8246f136ea6998dfc7ad1c6bb4d2
    (cherry picked from commit 6639b9e5643aeaee7a17b7995fddc2fb6288b926)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/python-tripleoclient 9.3.0 release.