Mysql has gone away after deploy
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Confirmed
|
Critical
|
Fuel Sustaining |
Bug Description
Steps to reproduce:
1. Create cluster
2. Add 3 node with controller and mongo roles
3. Add 2 node with compute and cinder roles
4. Deploy the cluster
5. Run network verification
6. Run ostf suits: 'ha', 'smoke', 'sanity'
Expected result:
OSTF passed
Actual result:
Multiple failures because of unexpected issue with mysql:
<30>Sep 16 01:34:49 node-2 ocf-mysql-wss: INFO: p_mysqld: validate_gtid(): GTID OK: a0dfb853-
<30>Sep 16 01:34:49 node-2 ocf-mysql-wss: INFO: p_mysqld: get_node_gtid(): Galera GTID: a0dfb853-
<30>Sep 16 01:34:49 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() The most seen GTID is: a0dfb853-
<30>Sep 16 01:34:49 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() Node's node-2.
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() Node's node-3.
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() Node's node-5.
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() Possible masters: node-2.
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: get_master() Choosed master: node-3.
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: validate_gtid(): GTID OK: a0dfb853-
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: get_node_gtid(): Galera GTID: a0dfb853-
<30>Sep 16 01:34:50 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_
<27>Sep 16 01:34:50 node-2 ocf-mysql-wss: ERROR: p_mysqld: check_if_
<27>Sep 16 01:34:50 node-2 ocf-mysql-wss: ERROR: p_mysqld: mysql_monitor(): I'm a master, and my GTID: a0dfb853-
<28>Sep 16 01:34:51 node-2 ocf-mysql-wss: WARNING: p_mysqld: proc_stop(): pid param is not a file or a number, try match by mysqld.
<30>Sep 16 01:34:51 node-2 ocf-mysql-wss: INFO: p_mysqld: proc_stop(): Stopping mysqld.
<28>Sep 16 01:34:53 node-2 ocf-mysql-wss: WARNING: p_mysqld: proc_kill(): Failed to stop mysqld.
<30>Sep 16 01:34:55 node-2 ocf-mysql-wss: INFO: p_mysqld: proc_stop(): Stopped mysqld.
<30>Sep 16 01:34:55 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_cleanup(): Cleaning up gtid attribute
<30>Sep 16 01:34:56 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): PIDFile /var/run/
<30>Sep 16 01:34:58 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL is not running
<27>Sep 16 01:34:58 node-2 ocf-mysql-wss: ERROR: p_mysqld: mysql_status(): PIDFile /var/run/
<27>Sep 16 01:35:00 node-2 ocf-mysql-wss: ERROR: p_mysqld: mysql_status(): MySQL is not running
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: get_node_gtid(): No GTID for node-2.
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: validate_gtid(): GTID OK: a0dfb853-
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: update_node_gtid(): Galera GTID: a0dfb853-
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_start(): Starting MySQL
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): No signs of SST found
<30>Sep 16 01:35:10 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): PIDFile /var/run/
<30>Sep 16 01:35:12 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL is not running
<30>Sep 16 01:35:15 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): MySQL process 11008 found
<30>Sep 16 01:35:15 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): No signs of SST found
<30>Sep 16 01:35:15 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): PIDFile /var/run/
<30>Sep 16 01:35:17 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL is not running
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): MySQL process 11008 found
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: check_if_sst(): No signs of SST found
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL PID found
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_start(): MySQL started
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: mysql_status(): MySQL PID found
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: validate_gtid(): GTID OK: a0dfb853-
<30>Sep 16 01:35:20 node-2 ocf-mysql-wss: INFO: p_mysqld: get_node_gtid(): Galera GTID: a0dfb853-
Failed cases: 5
https:/
tags: | added: area-library |
Changed in fuel: | |
status: | New → Confirmed |
2016-09- 16T01:34: 50.924169+ 00:00 err: ERROR: p_mysqld: check_if_ galera_ pc(): But I'm running a new cluster, PID:11714, this is a split-brain!