Mysql has gone away after deploy

[Dmitry Kalashnik]

Steps to reproduce:
            1. Create cluster
            2. Add 3 node with controller and mongo roles
            3. Add 2 node with compute and cinder roles
            4. Deploy the cluster
            5. Run network verification
            6. Run ostf suits: 'ha', 'smoke', 'sanity'

Expected result:
OSTF passed

Actual result:
Multiple failures because of unexpected issue with mysql:

<30>Sep 16 01:34:49 node-2 ocf-mysql-wss: INFO: p_mysqld: validate_gtid(): GTID OK: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4137
<30>Sep 16 01:34:49 node-2 ocf-mysql-wss: INFO: p_mysqld: get_node_gtid(): Galera GTID: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4137
<30>Sep 16 01:34:49 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() The most seen GTID is: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9
<30>Sep 16 01:34:49 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() Node's node-2.test.domain.local score: 100, GTID/SEQNUM: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4202
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() Node's node-3.test.domain.local score: 100, GTID/SEQNUM: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4233
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() Node's node-5.test.domain.local score: 100, GTID/SEQNUM: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4137
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() Possible masters: node-2.test.domain.local node-3.test.domain.local node-5.test.domain.local
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() Choosed master: node-3.test.domain.local with GTID: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4233
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: validate_gtid(): GTID OK: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4233
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: get_node_gtid(): Galera GTID: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4233
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_galera_pc(): My neighbour is Primary Component with GTID: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4233
<27>Sep 16 01:34:50 node-2 ocf-mysql-wss: ERROR: p_mysqld: check_if_galera_pc(): But I'm running a new cluster, PID:11714, this is a split-brain!
<27>Sep 16 01:34:50 node-2 ocf-mysql-wss: ERROR: p_mysqld: mysql_monitor(): I'm a master, and my GTID: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4202, which was not expected
<28>Sep 16 01:34:51 node-2 ocf-mysql-wss: WARNING: p_mysqld: proc_stop(): pid param is not a file or a number, try match by mysqld.*/var/lib/mysql
<30>Sep 16 01:34:51 node-2 ocf-mysql-wss: INFO: p_mysqld: proc_stop(): Stopping mysqld.*/var/lib/mysql by PID none
<28>Sep 16 01:34:53 node-2 ocf-mysql-wss: WARNING: p_mysqld: proc_kill(): Failed to stop mysqld.*/var/lib/mysql with SIGTERM
<30>Sep 16 01:34:55 node-2 ocf-mysql-wss: INFO: p_mysqld: proc_stop(): Stopped mysqld.*/var/lib/mysql
<30>Sep 16 01:34:55 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_cleanup(): Cleaning up gtid attribute
<30>Sep 16 01:34:56 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): PIDFile /var/run/resource-agents/mysql-wss/mysql-wss.pid of MySQL server not found. Sleeping for 2 seconds. 0 retries left
<30>Sep 16 01:34:58 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL is not running
<27>Sep 16 01:34:58 node-2 ocf-mysql-wss: ERROR: p_mysqld: mysql_status(): PIDFile /var/run/resource-agents/mysql-wss/mysql-wss.pid of MySQL server not found. Sleeping for 2 seconds. 0 retries left
<27>Sep 16 01:35:00 node-2 ocf-mysql-wss: ERROR: p_mysqld: mysql_status(): MySQL is not running
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: get_node_gtid(): No GTID for node-2.test.domain.local
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: validate_gtid(): GTID OK: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4287
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: update_node_gtid(): Galera GTID: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4287
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_start(): Starting MySQL
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): No signs of SST found
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): PIDFile /var/run/resource-agents/mysql-wss/mysql-wss.pid of MySQL server not found. Sleeping for 2 seconds. 0 retries left
<30>Sep 16 01:35:12 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL is not running
<30>Sep 16 01:35:15 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): MySQL process 11008 found
<30>Sep 16 01:35:15 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): No signs of SST found
<30>Sep 16 01:35:15 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): PIDFile /var/run/resource-agents/mysql-wss/mysql-wss.pid of MySQL server not found. Sleeping for 2 seconds. 0 retries left
<30>Sep 16 01:35:17 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL is not running
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): MySQL process 11008 found
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): No signs of SST found
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL PID found
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_start(): MySQL started
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL PID found
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: validate_gtid(): GTID OK: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4287
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: get_node_gtid(): Galera GTID: a0dfb853-7ba7-11e6-a28b-4afcbc77a9d9:4287

Failed cases: 5

https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.cic_maintenance_mode/62

[Maksim Malchuk (mmalchuk)]

2016-09-16T01:34:50.924169+00:00 err: ERROR: p_mysqld: check_if_galera_pc(): But I'm running a new cluster, PID:11714, this is a split-brain!