ssl container setups hangs in heat db sync

Bug #1715132 reported by Attila Fazekas on 2017-09-05
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Critical
Juan Antonio Osorio Robles

Bug Description

heat db sync forever tries to connect to the mysql vip, but haproxy is not running.

openstack-haproxy-docker haproxy_init_bundle log contains errors like:

Error: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/content: change from {md5}1f337186b0e1ba5ee82760cb437fb810 to {md5}a2f2a8d54d1068d962337a23798234be failed: Execution of '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg20170905-8-bg0qsd -c' returned 1: [ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:26] : 'bind 10.0.0.101:13042' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:37] : 'bind 10.0.0.101:13776' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:48] : 'bind 10.0.0.101:13292' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:59] : 'bind 10.0.0.101:13041' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:77] : 'bind 10.0.0.101:13004' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:90] : 'bind 10.0.0.101:13005' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:103] : 'bind 10.0.0.101:13003' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:116] : 'bind 10.0.0.101:443' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:118] : 'bind 172.17.1.11:443' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:139] : 'bind 10.0.0.101:13000' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:160] : 'bind 10.0.0.101:13696' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:176] : 'bind 10.0.0.101:13080' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:186] : 'bind 10.0.0.101:13774' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:197] : 'bind 10.0.0.101:13778' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:208] : 'bind 10.0.0.101:13779' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:229] : 'bind 10.0.0.101:13386' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:237] : 'bind 10.0.0.101:13808' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg20170905-8-bg0qsd
[ALERT] 247/071151 (3236) : Proxy 'aodh': no SSL certificate specified for bind '10.0.0.101:13042' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:26] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'cinder': no SSL certificate specified for bind '10.0.0.101:13776' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:37] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'glance_api': no SSL certificate specified for bind '10.0.0.101:13292' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:48] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'gnocchi': no SSL certificate specified for bind '10.0.0.101:13041' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:59] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'heat_api': no SSL certificate specified for bind '10.0.0.101:13004' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:77] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'heat_cfn': no SSL certificate specified for bind '10.0.0.101:13005' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:90] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'heat_cloudwatch': no SSL certificate specified for bind '10.0.0.101:13003' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:103] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'horizon': no SSL certificate specified for bind '10.0.0.101:443' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:116] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'horizon': no SSL certificate specified for bind '172.17.1.11:443' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:118] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'keystone_public': no SSL certificate specified for bind '10.0.0.101:13000' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:139] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'neutron': no SSL certificate specified for bind '10.0.0.101:13696' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:160] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'nova_novncproxy': no SSL certificate specified for bind '10.0.0.101:13080' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:176] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'nova_osapi': no SSL certificate specified for bind '10.0.0.101:13774' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:186] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'nova_placement': no SSL certificate specified for bind '10.0.0.101:13778' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:197] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'panko': no SSL certificate specified for bind '10.0.0.101:13779' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:208] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'sahara': no SSL certificate specified for bind '10.0.0.101:13386' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:229] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'swift_proxy_server': no SSL certificate specified for bind '10.0.0.101:13808' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:237] (use 'crt').
[ALERT] 247/071151 (3236) : Fatal errors found in configuration.

Similar settings work without ssl, also it worked without containers.

Fix proposed to branch: master
Review: https://review.openstack.org/500779

Changed in tripleo:
assignee: nobody → Juan Antonio Osorio Robles (juan-osorio-robles)
status: New → In Progress
Changed in tripleo:
importance: Undecided → Critical
milestone: none → pike-rc2
Luigi Toscano (ltoscano) wrote :

This pre-existing bug seems to be another outcome of the same issue:

https://bugs.launchpad.net/tripleo/+bug/1715029

Changed in tripleo:
importance: Critical → High
Changed in tripleo:
milestone: pike-rc2 → queens-1
Changed in tripleo:
importance: High → Critical
milestone: queens-1 → pike-rc2

Reviewed: https://review.openstack.org/500779
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=03622e89ac3037b4d69d913586823e689b210688
Submitter: Jenkins
Branch: master

commit 03622e89ac3037b4d69d913586823e689b210688
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Tue Sep 5 14:12:58 2017 +0300

    Mount public certificate in haproxy init container

    It's being mounted on the actual haproxy container, but not the init
    one.

    Change-Id: I66b69e0bb3642dbfeec767ef5216d515786b5b19
    Closes-Bug: #1715132

Changed in tripleo:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/501127
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6bed7525fcfe32108ab797e88c5bf73c0f8fff83
Submitter: Jenkins
Branch: stable/pike

commit 6bed7525fcfe32108ab797e88c5bf73c0f8fff83
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Tue Sep 5 14:12:58 2017 +0300

    Mount public certificate in haproxy init container

    It's being mounted on the actual haproxy container, but not the init
    one.

    Change-Id: I66b69e0bb3642dbfeec767ef5216d515786b5b19
    Closes-Bug: #1715132
    (cherry picked from commit 03622e89ac3037b4d69d913586823e689b210688)

tags: added: in-stable-pike

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc2 release candidate.

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers