tripleo

ssl container setups hangs in heat db sync

Bug #1715132 reported by Attila Fazekas
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Juan Antonio Osorio Robles
tripleo pike-rc2

Bug Description

heat db sync forever tries to connect to the mysql vip, but haproxy is not running.

openstack-haproxy-docker haproxy_init_bundle log contains errors like:

Error: /Stage[main]/Haproxy/Haproxy::Instance[haproxy]/Haproxy::Config[haproxy]/Concat[/etc/haproxy/haproxy.cfg]/File[/etc/haproxy/haproxy.cfg]/content: change from {md5}1f337186b0e1ba5ee82760cb437fb810 to {md5}a2f2a8d54d1068d962337a23798234be failed: Execution of '/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg20170905-8-bg0qsd -c' returned 1: [ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:26] : 'bind 10.0.0.101:13042' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:37] : 'bind 10.0.0.101:13776' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:48] : 'bind 10.0.0.101:13292' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:59] : 'bind 10.0.0.101:13041' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:77] : 'bind 10.0.0.101:13004' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:90] : 'bind 10.0.0.101:13005' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:103] : 'bind 10.0.0.101:13003' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:116] : 'bind 10.0.0.101:443' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:118] : 'bind 172.17.1.11:443' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:139] : 'bind 10.0.0.101:13000' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:160] : 'bind 10.0.0.101:13696' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:176] : 'bind 10.0.0.101:13080' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:186] : 'bind 10.0.0.101:13774' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:197] : 'bind 10.0.0.101:13778' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:208] : 'bind 10.0.0.101:13779' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:229] : 'bind 10.0.0.101:13386' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : parsing [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:237] : 'bind 10.0.0.101:13808' : unable to load SSL private key from PEM file '/etc/pki/tls/private/overcloud_endpoint.pem'.
[ALERT] 247/071151 (3236) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg20170905-8-bg0qsd
[ALERT] 247/071151 (3236) : Proxy 'aodh': no SSL certificate specified for bind '10.0.0.101:13042' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:26] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'cinder': no SSL certificate specified for bind '10.0.0.101:13776' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:37] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'glance_api': no SSL certificate specified for bind '10.0.0.101:13292' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:48] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'gnocchi': no SSL certificate specified for bind '10.0.0.101:13041' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:59] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'heat_api': no SSL certificate specified for bind '10.0.0.101:13004' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:77] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'heat_cfn': no SSL certificate specified for bind '10.0.0.101:13005' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:90] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'heat_cloudwatch': no SSL certificate specified for bind '10.0.0.101:13003' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:103] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'horizon': no SSL certificate specified for bind '10.0.0.101:443' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:116] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'horizon': no SSL certificate specified for bind '172.17.1.11:443' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:118] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'keystone_public': no SSL certificate specified for bind '10.0.0.101:13000' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:139] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'neutron': no SSL certificate specified for bind '10.0.0.101:13696' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:160] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'nova_novncproxy': no SSL certificate specified for bind '10.0.0.101:13080' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:176] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'nova_osapi': no SSL certificate specified for bind '10.0.0.101:13774' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:186] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'nova_placement': no SSL certificate specified for bind '10.0.0.101:13778' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:197] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'panko': no SSL certificate specified for bind '10.0.0.101:13779' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:208] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'sahara': no SSL certificate specified for bind '10.0.0.101:13386' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:229] (use 'crt').
[ALERT] 247/071151 (3236) : Proxy 'swift_proxy_server': no SSL certificate specified for bind '10.0.0.101:13808' at [/etc/haproxy/haproxy.cfg20170905-8-bg0qsd:237] (use 'crt').
[ALERT] 247/071151 (3236) : Fatal errors found in configuration.

Similar settings work without ssl, also it worked without containers.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/500779

Changed in tripleo:
assignee: nobody → Juan Antonio Osorio Robles (juan-osorio-robles)
status: New → In Progress
Juan Antonio Osorio Robles (juan-osorio-robles)
Changed in tripleo:
importance: Undecided → Critical
milestone: none → pike-rc2
Revision history for this message
Luigi Toscano (ltoscano) wrote :

This pre-existing bug seems to be another outcome of the same issue:

https://bugs.launchpad.net/tripleo/+bug/1715029

Emilien Macchi (emilienm)
Changed in tripleo:
importance: Critical → High
Emilien Macchi (emilienm)
Changed in tripleo:
milestone: pike-rc2 → queens-1
Juan Antonio Osorio Robles (juan-osorio-robles)
Changed in tripleo:
importance: High → Critical
milestone: queens-1 → pike-rc2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/501127

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/500779
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=03622e89ac3037b4d69d913586823e689b210688
Submitter: Jenkins
Branch: master

commit 03622e89ac3037b4d69d913586823e689b210688
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Tue Sep 5 14:12:58 2017 +0300

    Mount public certificate in haproxy init container

    It's being mounted on the actual haproxy container, but not the init
    one.

    Change-Id: I66b69e0bb3642dbfeec767ef5216d515786b5b19
    Closes-Bug: #1715132

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/pike)

Reviewed: https://review.openstack.org/501127
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=6bed7525fcfe32108ab797e88c5bf73c0f8fff83
Submitter: Jenkins
Branch: stable/pike

commit 6bed7525fcfe32108ab797e88c5bf73c0f8fff83
Author: Juan Antonio Osorio Robles <email address hidden>
Date: Tue Sep 5 14:12:58 2017 +0300

    Mount public certificate in haproxy init container

    It's being mounted on the actual haproxy container, but not the init
    one.

    Change-Id: I66b69e0bb3642dbfeec767ef5216d515786b5b19
    Closes-Bug: #1715132
    (cherry picked from commit 03622e89ac3037b4d69d913586823e689b210688)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 7.0.0.0rc2

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 8.0.0.0b1

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.