upgrade_contrail failed in a HA setup

[Vedamurthy Joshi]

R2.20 Build 30 Ubuntu 14.04 Juno Multi-node setup

In the below setup with HA enabled, upgrade from 2.20 30 to 2.20 40 failed.
Ranjeet looked into it. Mysql had stopped on 10.204.216.58. After starting mysql on it , waiting for few mins, and rerunning upgrade_contrail, things worked fine.

2015-06-04 20:13:58:959465: [root@10.204.216.59] out: [localhost] local: service supervisor-openstack restart
2015-06-04 20:13:58:959509: [root@10.204.216.59] out: stop: Unknown instance:
2015-06-04 20:13:58:962922: [root@10.204.216.59] out: supervisor-openstack start/running, process 10015
2015-06-04 20:13:59:026634: [root@10.204.216.59] out:
2015-06-04 20:13:59:127074:
2015-06-04 20:13:59:127552: [root@10.204.216.58] Executing task 'setup_cmon_schema'
2015-06-04 20:13:59:148835: [root@10.204.216.58] sudo: cat /etc/contrail/mysql.token
2015-06-04 20:13:59:149016: [root@10.204.216.58] out: e6a0b37bb8b95acb8af4
2015-06-04 20:13:59:183452: [root@10.204.216.58] out:
2015-06-04 20:13:59:183733:
2015-06-04 20:13:59:184061: [root@10.204.216.58] sudo: python -c 'from platform import linux_distribution; print linux_distribution()'
2015-06-04 20:13:59:184277: [root@10.204.216.58] out: ('Ubuntu', '14.04', 'trusty')
2015-06-04 20:13:59:249519: [root@10.204.216.58] out:
2015-06-04 20:13:59:252901:
2015-06-04 20:13:59:281879: [root@10.204.216.58] sudo: mysql -u root -pe6a0b37bb8b95acb8af4 -e "CREATE SCHEMA IF NOT EXISTS cmon"
2015-06-04 20:13:59:282066: [root@10.204.216.58] out: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)
2015-06-04 20:13:59:306708: [root@10.204.216.58] out:
2015-06-04 20:13:59:306888:

2015-06-04 20:13:59:307107: Fatal error: sudo() received nonzero return code 1 while executing!
2015-06-04 20:13:59:307107:
2015-06-04 20:13:59:307107: Requested: mysql -u root -pe6a0b37bb8b95acb8af4 -e "CREATE SCHEMA IF NOT EXISTS cmon"
2015-06-04 20:13:59:307107: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "mysql -u root -pe6a0b37bb8b95acb8af4 -e \"CREATE SCHEMA IF NOT EXISTS cmon\""
2015-06-04 20:13:59:307107:
2015-06-04 20:13:59:307142: Aborting.
2015-06-04 20:13:59:307142: 2015-06-04 20:13:59:307005: Disconnecting from 10.204.216.59... done.
2015-06-04 20:13:59:370819: Disconnecting from 10.204.216.58... done.
2015-06-04 20:13:59:402372: Disconnecting from 10.204.216.60... done.
2015-06-04 20:13:59:465938: root@nodec1:/opt/contrail/utils#

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6, host7],
    'cfgm': [host1,host2,host3],
    'openstack': [host1,host2,host3],
    'control': [host1,host2,host3],
    'compute': [host4,host5, host6, host7],
    'collector': [host1,host2,host3],
    'webui': [host1],
    'database': [host1,host2,host3],
    'toragent': [host6, host7],
    'tsn': [host6, host7],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodec1', 'nodec2', 'nodec3', 'nodek1', 'nodek2', 'nodek3', 'nodeg11']
}

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/11384
Submitter: Ranjeet R (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/11384
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/f3b7d8af093fa337c822b2d928fffa95eca13069
Submitter: Zuul
Branch: R2.20

commit f3b7d8af093fa337c822b2d928fffa95eca13069
Author: Ranjeet R <email address hidden>
Date: Mon Jun 8 12:33:28 2015 -0700

Fixes: upgrade_contrail failed in a HA setup

In 2.2, we are re-creating the cmon_schema. The Galera cluster
was forming and hence when we try to setup cmon_schema it fails.

Before setting up cmon schema, make sure that mysql galera status
is up and running, and if its in joining state - wait for it to
completely sync.

Change-Id: Iec0e9ee92eed90f4ed5ae8dc4b680e1f9ce070ed
ClosesBug: 1462209