Juniper Openstack

fab setup_all throws SQL connection failed continuously if SSL is enabled in HA setup

Bug #1649787 reported by musharani on 2016-12-14

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R3.1	Fix Committed	Critical	Ignatious Johnson Christopher	Juniper Openstack r3.1.2.0
R3.2	Fix Committed	Critical	Ignatious Johnson Christopher	Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"
Trunk	Fix Committed	Critical	Ignatious Johnson Christopher	Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"

Bug Description

while running fab setup_all throws a warning message 'SQL connection failed' continuously for ssl enabled ha setup.

setup details:
env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6],
    'cfgm': [host1, host2, host3],
    'openstack': [host1, host2, host3],
    'webui': [host3],
    'control': [host1, host2, host3],
    'compute': [host4, host5, host6],
    'collector': [host1, host2],
    'database': [host1, host2, host3],
    'build': [host_build],
}

env.hostnames = {
'all': ['nodea35', 'nodea34', 'nodec53', 'nodec54', 'nodec55', 'nodec56']
}

env.ha = {
'internal_vip' : '22.22.22.36',
}
ha_setup = True

ha_proxy.log:
[ALERT] 348/121102 (30898) : parsing [/etc/haproxy/haproxy.cfg:65] : 'bind *:5000' : unable to load SSL private key from PEM file '/etc/keystone/ssl/certs/keystonecertbundle.pem'.^M
[ALERT] 348/121102 (30898) : parsing [/etc/haproxy/haproxy.cfg:99] : 'bind *:35357' : unable to load SSL private key from PEM file '/etc/keystone/ssl/certs/keystonecertbundle.pem'.^M
[ALERT] 348/121102 (30898) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg^M
[ALERT] 348/121102 (30898) : Proxy 'openstack-keystone': no SSL certificate specified for bind '*:5000' at [/etc/haproxy/haproxy.cfg:65] (use 'crt').^M
[ALERT] 348/121102 (30898) : Proxy 'openstack-keystone-admin': no SSL certificate specified for bind '*:35357' at [/etc/haproxy/haproxy.cfg:99] (use 'crt').^M
[WARNING] 348/121102 (30898) : config : missing timeouts for proxy 'memcached'.^M
   | While not properly invalid, you will certainly encounter various problems^M
   | with such a configuration. To fix this, please ensure that all following^M
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.^M
[WARNING] 348/121102 (30898) : config : missing timeouts for proxy 'rabbitmq'.^M
   | While not properly invalid, you will certainly encounter various problems^M
   | with such a configuration. To fix this, please ensure that all following^M
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.^M
[WARNING] 348/121102 (30898) : config : missing timeouts for proxy 'mysql'.^M
   | While not properly invalid, you will certainly encounter various problems^M
   | with such a configuration. To fix this, please ensure that all following^M
   | timeouts are set to a non-zero value: 'client', 'connect', 'server'.^M
[ALERT] 348/121102 (30898) : Fatal errors found in configuration.^M

fab setup_all log:
2016-12-14 12:23:14:051921: [root@10.204.216.31] out: + sed -i -e 's|^#* *\(\(sql_\)*connection *= *mysql://heat\):.*@|\1:c0ntrail123@|' /etc/heat/heat.conf
2016-12-14 12:23:14:051994: [root@10.204.216.31] out: + echo 'Initializing the heat database, please wait...'
2016-12-14 12:23:14:055732: [root@10.204.216.31] out: Initializing the heat database, please wait...
2016-12-14 12:23:14:055879: [root@10.204.216.31] out: + '[' heat = nova ']'
2016-12-14 12:23:14:055962: [root@10.204.216.31] out: + '[' heat = cinder ']'
2016-12-14 12:23:14:056043: [root@10.204.216.31] out: + '[' heat = nova_api ']'
2016-12-14 12:23:14:056114: [root@10.204.216.31] out: + db_sync=db_sync
2016-12-14 12:23:14:056186: [root@10.204.216.31] out: + '[' 0 -eq 1 ']'
2016-12-14 12:23:14:056259: [root@10.204.216.31] out: + '[' 1 -eq 1 ']'
2016-12-14 12:23:14:056328: [root@10.204.216.31] out: + '[' heat = nova_api ']'
2016-12-14 12:23:14:056398: [root@10.204.216.31] out: + heat-manage db_sync
2016-12-14 12:23:14:056468: [root@10.204.216.31] out: 2016-12-14 12:23:17.178 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -1 attempts left.
2016-12-14 12:23:17:192491: [root@10.204.216.31] out: 2016-12-14 12:23:25.172 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -2 attempts left.
2016-12-14 12:23:25:203727: [root@10.204.216.31] out: 2016-12-14 12:23:38.185 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -3 attempts left.
2016-12-14 12:23:38:221452: [root@10.204.216.31] out: 2016-12-14 12:23:45.189 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -4 attempts left.
2016-12-14 12:23:45:194235: [root@10.204.216.31] out: 2016-12-14 12:23:56.189 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -5 attempts left.
2016-12-14 12:23:56:208304: [root@10.204.216.31] out: 2016-12-14 12:24:02.192 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -6 attempts left.
2016-12-14 12:24:02:216265: [root@10.204.216.31] out: 2016-12-14 12:24:12.188 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -7 attempts left.
2016-12-14 12:24:12:193170: [root@10.204.216.31] out: 2016-12-14 12:24:20.192 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -8 attempts left.
2016-12-14 12:24:20:203964: [root@10.204.216.31] out: 2016-12-14 12:24:34.192 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -9 attempts left.
2016-12-14 12:24:34:221834: [root@10.204.216.31] out: 2016-12-14 12:24:41.199 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -10 attempts left.
2016-12-14 12:24:41:231332: [root@10.204.216.31] out: 2016-12-14 12:24:52.196 7156 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -11 attempts left.

Tags:

Jeba Paulaiyan (jebap) on 2016-12-14

tags:

added: blocker