Ssl verify error in heat -> keystone communication

[Kamil Madac]

In 18.1.3 with self-signed https enabled on haproxy, 'openstack stack list -vvvv' ends up with following Internal Server Error:

SSL exception connecting to https://10.5.0.254:5000: HTTPSConnectionPool(host='10.5.0.254', port=5000): Max retries exceeded with url: / (Caused by SSLError(SSLError(\"bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')],)\",),))\n", "type": "SSLError"}, "title": "Internal Server Error"

The reason is that self-signed certificate is not distributed to heat containers. As a workaround I added certificate /etc/ssl/certs/haproxy.cert from haproxy node to /openstack/venvs/heat-18.1.3/lib/python2.7/site-packages/certifi/cacert.pem as requests python library uses certifi package as for validating the trustworthiness of SSL certificates.

The correct way how to solve the bug would be to add self signed certificate to /etc/ssl/certs/ca-certificates.crt and set environment variable REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt in systemd heat services in order to use system certificates.

[Kamil Madac (kamil-madac)]

Here is the patch we did in our Rocky OSA deployment to solve the bug:

http://paste.openstack.org/show/744947/

It basicaly adds env. variable REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt to heat systemd services.

[panic! (thomas-schend)]

+1

[panic! (thomas-schend)]

Just verified this a few minutes ago. Solves the issue for me. Netherless the practice to go to the external API should be questioned.

[panic! (thomas-schend)]

Another note:

The patch fixes the problem with heat service but it does not fix the issue that horizon does not display the orchestration tab under admin - system - system information.

Maybe the horizon container needs the same patch for the systemd service. Not validated yet.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/639601

[masterpe (michiel-y)]

After testing this patch, I get the following error:

TASK [Regenerate certs] ********************************************************
changed: [controller01_heat_api_container-bd6c1c98]

RUNNING HANDLER [os_heat : Restart heat services] ******************************
fatal: [controller01_heat_api_container-bd6c1c98]: FAILED! => {"msg": "'heat_uwsgi_bin' is undefined"}

The task "Distribute self signed ssl cert" notify's the handler. But some how the heat_uwsgi_bin variable isn't loaded

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/672948

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672948
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_heat/commit/?id=288634ce0bf042bed614b3f764753d7b65a7170f
Submitter: Zuul
Branch: master

commit 288634ce0bf042bed614b3f764753d7b65a7170f
Author: Jonathan Rosser <email address hidden>
Date: Fri Jul 26 11:20:48 2019 +0100

    Fix keystone endpoint for heat servers

    This patch changes the heat config so that communication between
    the heat service and the other internal parts of openstack occurs over
    the internal API endpoint, but a new heat configuration option [1] is set
    which ensures that the keystone endpoint written into server configs
    points to the external API endpoint.

    This should address several long running SSL related failures when self
    signed certificates are used, and allows heat to work correctly when the
    internal and external endpoints are on different networks.

    Change-Id: I533ab16557cb83e2791dbb7267a97fb0d44e9ba6
    Fixes-Bug: 1811086
    Fixes-Bug: 1820591
    Related-Bug: 1824646
    Related-Bug: 1814909
    Depends-On: https://review.opendev.org/678062

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/688895

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/688895
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_heat/commit/?id=eccb5c13fec9d9efcccb821dbcf0f344cd515eec
Submitter: Zuul
Branch: stable/stein

commit eccb5c13fec9d9efcccb821dbcf0f344cd515eec
Author: Jonathan Rosser <email address hidden>
Date: Fri Jul 26 11:20:48 2019 +0100

    Fix keystone endpoint for heat servers

    This patch changes the heat config so that communication between
    the heat service and the other internal parts of openstack occurs over
    the internal API endpoint, but a new heat configuration option [1] is set
    which ensures that the keystone endpoint written into server configs
    points to the external API endpoint.

    This should address several long running SSL related failures when self
    signed certificates are used, and allows heat to work correctly when the
    internal and external endpoints are on different networks.

    Change-Id: I533ab16557cb83e2791dbb7267a97fb0d44e9ba6
    Fixes-Bug: 1811086
    Fixes-Bug: 1820591
    Related-Bug: 1824646
    Related-Bug: 1814909
    Depends-On: https://review.opendev.org/688894
    (cherry picked from commit 288634ce0bf042bed614b3f764753d7b65a7170f)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/705555

[OpenStack Infra (hudson-openstack)]

Change abandoned by Shannon Mitchell (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/705555
Reason: Submitting a separate bug just for the public/private url fix.