Openstack HA : Provisioning is failing in mitaka , not able to connect to mysql db

[venu kolli]

Build 2740 , Mitaka .

Please find the setup details @ 10.84.24.32/33/34`0

find complete log attached.

2016-07-11 11:27:41:486887: [root@10.84.24.32] out: + echo 'SELECT 1;'
2016-07-11 11:27:41:486912: [root@10.84.24.32] out: + mysql -u root --password=0fca3cc4cd3cf1eeb17d
2016-07-11 11:27:41:486936: [root@10.84.24.32] out: + echo 'Verified connectivity to MySQL.'
2016-07-11 11:27:41:486959: [root@10.84.24.32] out: Verified connectivity to MySQL.
2016-07-11 11:27:41:486981: [root@10.84.24.32] out: + '[' init = init ']'
2016-07-11 11:27:41:487005: [root@10.84.24.32] out: ++ echo 'SELECT COUNT(*) FROM INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME='\''heat'\'';'
2016-07-11 11:27:41:487028: [root@10.84.24.32] out: ++ mysql -u root --password=0fca3cc4cd3cf1eeb17d
2016-07-11 11:27:41:487052: [root@10.84.24.32] out: ++ tail -n+2
2016-07-11 11:27:41:487075: [root@10.84.24.32] out: + dbs=0
2016-07-11 11:27:41:488543: [root@10.84.24.32] out: + '[' 0 '!=' 0 ']'
2016-07-11 11:27:41:488614: [root@10.84.24.32] out: ++ echo 'SELECT COUNT(*) FROM mysql.user WHERE User = '\''heat'\'';'
2016-07-11 11:27:41:488668: [root@10.84.24.32] out: ++ mysql -u root --password=0fca3cc4cd3cf1eeb17d
2016-07-11 11:27:41:488719: [root@10.84.24.32] out: ++ tail -n+2
2016-07-11 11:27:41:488780: [root@10.84.24.32] out: + users=0
2016-07-11 11:27:41:492189: [root@10.84.24.32] out: + '[' 0 '!=' 0 ']'
2016-07-11 11:27:41:492243: [root@10.84.24.32] out: + '[' init = init ']'
2016-07-11 11:27:41:492272: [root@10.84.24.32] out: + echo 'Creating '\''heat'\'' database.'
2016-07-11 11:27:41:492295: [root@10.84.24.32] out: Creating 'heat' database.
2016-07-11 11:27:41:492317: [root@10.84.24.32] out: + '[' heat = nova_api ']'
2016-07-11 11:27:41:492338: [root@10.84.24.32] out: + mysql -u root --password=0fca3cc4cd3cf1eeb17d
2016-07-11 11:27:41:492358: [root@10.84.24.32] out: + cat
2016-07-11 11:27:41:492381: [root@10.84.24.32] out: + '[' init = init ']'
2016-07-11 11:27:41:495744: [root@10.84.24.32] out: + '[' c0ntrail123 '!=' heat ']'
2016-07-11 11:27:41:495798: [root@10.84.24.32] out: + echo 'Updating '\''heat'\'' database password in /etc/heat/heat.conf'
2016-07-11 11:27:41:495826: [root@10.84.24.32] out: Updating 'heat' database password in /etc/heat/heat.conf
2016-07-11 11:27:41:495852: [root@10.84.24.32] out: + '[' heat = nova_api ']'
2016-07-11 11:27:41:495875: [root@10.84.24.32] out: + sed -i -e 's|^#* *\(\(sql_\)*connection *= *mysql://heat\):.*@|\1:c0ntrail123@|' /etc/heat/heat.conf
2016-07-11 11:27:41:495897: [root@10.84.24.32] out: + echo 'Initializing the heat database, please wait...'
2016-07-11 11:27:41:499235: [root@10.84.24.32] out: Initializing the heat database, please wait...
2016-07-11 11:27:41:499287: [root@10.84.24.32] out: + '[' heat = nova ']'
2016-07-11 11:27:41:499317: [root@10.84.24.32] out: + '[' heat = cinder ']'
2016-07-11 11:27:41:499348: [root@10.84.24.32] out: + '[' heat = nova_api ']'
2016-07-11 11:27:41:499368: [root@10.84.24.32] out: + db_sync=db_sync
2016-07-11 11:27:41:499389: [root@10.84.24.32] out: + '[' 0 -eq 1 ']'
2016-07-11 11:27:41:499411: [root@10.84.24.32] out: + '[' 1 -eq 1 ']'
2016-07-11 11:27:41:499432: [root@10.84.24.32] out: + '[' heat = nova_api ']'
2016-07-11 11:27:41:499452: [root@10.84.24.32] out: + heat-manage db_sync
2016-07-11 11:27:41:499472: [root@10.84.24.32] out: 2016-07-11 11:27:43.954 15783 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -1 attempts left.
2016-07-11 11:27:43:968345: [root@10.84.24.32] out: 2016-07-11 11:27:51.952 15783 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -2 attempts left.
2016-07-11 11:27:51:987285: [root@10.84.24.32] out: 2016-07-11 11:28:04.952 15783 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -3 attempts left.
2016-07-11 11:28:04:976125: [root@10.84.24.32] out: 2016-07-11 11:28:17.977 15783 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -4 attempts left.
2016-07-11 11:28:18:007141: [root@10.84.24.32] out: 2016-07-11 11:28:24.980 15783 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -5 attempts left.
2016-07-11 11:28:24:987355: [root@10.84.24.32] out: 2016-07-11 11:28:31.980 15783 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -6 attempts left.
2016-07-11 11:28:32:003591: [root@10.84.24.32] out: 2016-07-11 11:28:39.982 15783 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -7 attempts left.
2016-07-11 11:28:39:988578: [root@10.84.24.32] out: 2016-07-11 11:28:47.981 15783 WARNING oslo_db.sqlalchemy.engines [-] SQL connection failed. -8 attempts left.

[amudhar (amudha)]

Fixing through bug #1582932 (tracking bug for mitaka support).

https://review.opencontrail.org/#/c/21894/ is the review for setting api_database in nova.conf

[Ritam Gangopadhyay (ritam)]

Can you please checkin a fix for R3.1 branch as well, this review is for mainline https://review.opencontrail.org/#/c/21894/

[amudhar (amudha)]

Fix committed into R3.1 through https://review.opencontrail.org/22032 (#1582932).

[venu kolli (vkolli)]

Issue is still present with the way haproxy is started