GTID replications fails to resume after master disconnect

[monty solomon]

The slave was unable to connect to the master due to network issues. Once the network issues were resolved the slave did not continue replication even though SHOW SLAVE STATUS listed "Waiting for master to send event"

We sometimes see similar issues when the master is restarted.

The issue appears to be specific to GTID replication.

Repeated executions of SHOW SLAVE STATUS\G contained the following excerpt.

               Slave_IO_State: Waiting for master to send event
                Connect_Retry: 15
              Master_Log_File: bin.010862
          Read_Master_Log_Pos: 2528103
               Relay_Log_File: relay.001675
                Relay_Log_Pos: 307
        Relay_Master_Log_File: bin.010862
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
          Exec_Master_Log_Pos: 2528103
              Relay_Log_Space: 657
        Seconds_Behind_Master: 0
             Master_Server_Id: 1156476169
                  Master_UUID: 9f9bf362-83e9-11e4-a45f-0acf4ba07bb2
      Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
           Master_Retry_Count: 86400
           Retrieved_Gtid_Set: 9f9bf362-83e9-11e4-a45f-0acf4ba07bb2:1-1233067
            Executed_Gtid_Set: 90fe53a8-1a31-11e5-b854-122ff10e879b:1-6,
9f9bf362-83e9-11e4-a45f-0acf4ba07bb2:1-1233067

After executing "STOP SLAVE; START SLAVE;" the slave resumed replication and Seconds_Behind_Master changed from 0 to 3510

               Slave_IO_State: Waiting for master to send event
                Connect_Retry: 15
              Master_Log_File: bin.010864
          Read_Master_Log_Pos: 11834054
               Relay_Log_File: relay.001679
                Relay_Log_Pos: 16023
        Relay_Master_Log_File: bin.010863
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
          Exec_Master_Log_Pos: 15825
              Relay_Log_Space: 24322598
        Seconds_Behind_Master: 3510
             Master_Server_Id: 1156476169
                  Master_UUID: 9f9bf362-83e9-11e4-a45f-0acf4ba07bb2
      Slave_SQL_Running_State: System lock
           Master_Retry_Count: 86400
           Retrieved_Gtid_Set: 9f9bf362-83e9-11e4-a45f-0acf4ba07bb2:1-1239713
            Executed_Gtid_Set: 90fe53a8-1a31-11e5-b854-122ff10e879b:1-6,
9f9bf362-83e9-11e4-a45f-0acf4ba07bb2:1-1233654

[monty solomon (monty+launchpad)]

mysql Ver 14.14 Distrib 5.6.25-73.1, for Linux (x86_64) using 6.0
Server version: 5.6.25-73.1-log Percona Server (GPL), Release 73.1, Revision 07b797f
CentOS release 6.5 (Final)

[monty solomon (monty+launchpad)]

excerpt from slave error log

2015-09-02 15:02:41 22681 [Note] Error reading relay log event: slave SQL thread was killed
2015-09-02 15:02:41 22681 [ERROR] Error reading packet from server: Lost connection to MySQL server during query ( server_errno=2013)
2015-09-02 15:02:41 22681 [Note] Slave I/O thread killed while reading event
2015-09-02 15:02:41 22681 [Note] Slave I/O thread exiting, read up to log 'bin.010862', position 2528103
2015-09-02 15:02:41 22681 [Warning] Storing MySQL user name or password information in the master info repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START SLAVE; see the 'START SLAVE Syntax' in the MySQL Manual for more information.
2015-09-02 15:02:41 22681 [Note] Slave SQL thread initialized, starting replication in log 'bin.010862' at position 2528103, relay log 'relay.001676' position: 307
2015-09-02 15:02:41 22681 [Note] Slave I/O thread: connected to master 'SUSR_Repl@blaurock:3306',replication started in log 'bin.010862' at position 2528103

excerpt from master error log

2015-09-02 15:02:41 14983 [Note] Start binlog_dump to master_thread_id(4528175) slave_server(2354230027), pos(, 4)

[Valerii Kravchuk (valerii-kravchuk)]

Please, send your my.cnf file content from slave.

[monty solomon (monty+launchpad)]

Attaching my.cnf and 001-basic-config.cnf

my.cnf includes 001-basic-config.cnf

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3307