tripleo

Deploying a HA overcloud + public TLS endpoints fails in container-puppet-haproxy with invalid pem file

Bug #1872834 reported by Damien Ciabrini on 2020-04-14

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	tripleo	Invalid	Undecided	Unassigned	tripleo ussuri-rc3 "tripleo ussuri-rc3"

Bug Description

I'm trying to deploy a HA overcloud with public TLS endpoints:

openstack overcloud deploy --override-ansible-cfg /home/stack/custom_ansible.cfg \
--templates /usr/share/openstack-tripleo-heat-templates \
--libvirt-type qemu --timeout 90 --ntp-server clock.redhat.com -e /home/stack/cloud-names.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml -e /home/stack/containers-prepare-parameter.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e /home/stack/network-environment.yaml -e /home/stack/overcloud_storage_params.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -e /home/stack/enable-tls.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml -e /home/stack/inject-trust-anchor.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml --validation-warnings-fatal -e /home/stack/overcloud-topology-config.yaml -e /home/stack/overcloud-selinux-config.yaml -e /usr/share/openstack-tripleo-heat-templates/ci/environments/ovb-ha.yaml

The deployment fails early during configuration of haproxy service in container-puppet-haproxy

TASK [tripleo_container_manage : Print failing containers] *********************
Tuesday 14 April 2020 16:46:17 +0000 (0:00:00.218) 0:13:07.871 *********
ESC[0;31mfatal: [overcloud-controller-0]: FAILED! => changed=false ESC[0m
ESC[0;31m msg: 'Container(s) with bad ExitCode: [''container-puppet-haproxy''], check logs in /var/log/containers/stdouts/'ESC[0m

The error seems to be related to a pem file which cannot be parsed correctly by haproxy:

<13>Apr 14 21:03:13 puppet-user: Notice: Compiled catalog for overcloud-controller-0.localdomain in environment production in 0.94 seconds
<13>Apr 14 21:03:13 puppet-user: Error: Execution of '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg20200414-17-mcrnj -c' returned 1: [ALERT] 104/210313 (53) : parsing [/etc/haproxy/haproxy.cfg20200414-17-mcrnj:27]
: 'bind 10.0.0.5:13776' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
<13>Apr 14 21:03:13 puppet-user: [ALERT] 104/210313 (53) : parsing [/etc/haproxy/haproxy.cfg20200414-17-mcrnj:42] : 'bind 10.0.0.5:13292' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcl
oud_endpoint.pem'.
<13>Apr 14 21:03:13 puppet-user: [ALERT] 104/210313 (53) : parsing [/etc/haproxy/haproxy.cfg20200414-17-mcrnj:64] : 'bind 10.0.0.5:13004' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcl
oud_endpoint.pem'.
[...]

on the controller, /etc/pki/tls/private/overcloud_endpoint.pem seems to be generated with escaped \n, which confuses parsing:

[root@overcloud-controller-0 ~]# openssl x509 -in /etc/pki/tls/private/overcloud_endpoint.pem -text
unable to load certificate
139864179402560:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
[root@overcloud-controller-0 ~]# cat /etc/pki/tls/private/overcloud_endpoint.pem
-----BEGIN CERTIFICATE-----\nMIIDTzCCAjegAwIBAgIBATANBgkqhkiG9w0BAQsFADBhMQswCQYDVQQGEwJVUzEL\nMAkGA1UECAwCTkMxEDAOBgNVBAcMB1JhbGVpZ2gxEDAOBgNVBAoMB1JlZCBIYXQx\nDTALBgNVBAsMBE9PT1ExEjAQBgNVBAMMCW92ZXJjbG91ZDAeFw0yMDA0MTQwOTA2\nNDZaFw0yMTA0MTQwOTA2NDZaMGAxCzAJBgNVBAYTAlVTMQswCQYDVQQIDAJOQzEQ\nMA4GA1UEBwwHUmFsZWlnaDEQMA4GA1UECgwHUmVkIEhhdDENMAsGA1UECwwET09P\nUTERMA8GA1UEAwwIMTAuMC4wLjUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK\nAoIBAQDfZli2bMohbME4Si1HaxbH4TKfaWMeeIsFoLw/8C7spKmkGQjrZYnMKMpJ\nUHWzO7p/JweWQ/l9OJgbYNgI6f9S2ijFgPkp1wI8ZW0TE1/U6e6tWLjp3fuARL+i\nxVKseIe7He4mPy9gULZfFVkQvIYs37w0baXtmJ9ohT0oGhOxGfvuQjZJqvQI7Wfb\nEQ3+UFgNRpiNhQiTNjer/v+FITs2u3xvMDr0oQBNRXPh7UMnrWX3qiJJ5wtZ6Sg3\n2Yb1al6KVbnOgFElx0VmQtcXlrrAdZ2m79Ur1/CM8Uy5nryIW+2gxVlWqOMnWvQ3\napv5FZC2jEin8exDfv9ztH9KVAIJAgMBAAGjEzARMA8GA1UdEQQIMAaHBAoAAAUw\nDQYJKoZIhvcNAQELBQADggEBABekCRlzNsGYkjYrfwOrfHeMbj1J33+32jGpdXqL\nNQ6uz9iInuuEPhKseG1Dymx/pzeHRk1oqv8WSYsjy7jCTEhi35cnS8DraP7Wh6D/\nxTmHi45Jl2nWPMhr4yndBAS0lAQENLkt4v4M97gNEru74g4Tjp4f5YviS3+ha8ba\n9fM3JlKFXrXbbuTFRMQP8I699dm2bvuYgC7mtL3RHqVfQcsJgYZb582RizPuLngT\nol7TxmhoN8jBU3uD0cmG8Xgqje3p38luuq2DLeGGfcrdyPFO0eM0SVwIIPYKCOpZ\n3qE6W5IP7FxzNmv9DEl25aB6AyhDL5B/ZTxlqNL+BJN+3tg=\n-----END CERTIFICATE-----\n

Revision history for this message

Rabi Mishra (rabi) wrote on 2020-04-15:

There seems to be an issue with ansible-runner which treats every extravar as string and there is no proper serialization for multiline strings like private key.

I've submitted PR[1] for that few days back. However, there is a workaround in tripleoclient[2] that already merged to fix this. I think your python-tripleoclient is old and does not have it.

[1] https://github.com/ansible/ansible-runner/pull/444
[2] https://github.com/openstack/python-tripleoclient/commit/a947b57094f2fd5da6d54e1c98b2aa947bb37914#diff-3d0be95f0f9b73cbde278ac20e8e41ff

Revision history for this message

Damien Ciabrini (dciabrin) wrote on 2020-04-15:

Thanks Rabi,

I didn't realize I deployed with a tag from last week, so your fix wasn't picked up.
Closing this bug accordingly.

Changed in tripleo:
status:	New → Fix Released
status:	Fix Released → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.