Manoj:
I ran the two commands on the one container where mysqld had crashed.
ubuntu@juju-6db2c9-0-lxd-5:~$ sudo myisamchk -r -q /var/lib/percona-xtradb-cluster/mysql/db - check record delete-chain - recovering (with sort) MyISAM-table '/var/lib/percona-xtradb-cluster/mysql/db' Data records: 22 - Fixing index 1 - Fixing index 2 ubuntu@juju-6db2c9-0-lxd-5:~$ sudo myisamchk -r -q /var/lib/percona-xtradb-cluster/mysql/user - check record delete-chain - recovering (with sort) MyISAM-table '/var/lib/percona-xtradb-cluster/mysql/user' Data records: 26 - Fixing index 1
Then I stopped and started that container.
THe daemon did not stay up long. It went down pretty soon with this in the error.log:
170829 21:58:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/percona-xtradb-cluster 170829 21:58:15 mysqld_safe Skipping wsrep-recover for ee3cf6b7-89c8-11e7-80c5-2aec0eea34fa:28447 pair 170829 21:58:15 mysqld_safe Assigning ee3cf6b7-89c8-11e7-80c5-2aec0eea34fa:28447 to wsrep_start_position 2017-08-29 21:58:15 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2017-08-29 21:58:15 0 [Note] /usr/sbin/mysqld (mysqld 5.6.34-79.1-79.1) starting as process 1279 ... 2017-08-29 21:58:15 1279 [Note] WSREP: Read nil XID from storage engines, skipping position init 2017-08-29 21:58:15 1279 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/libgalera_smm.so' 2017-08-29 21:58:15 1279 [Note] WSREP: wsrep_load(): Galera 3.19(rXXXX) by Codership Oy <email address hidden> loaded successfully. 2017-08-29 21:58:15 1279 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm. 2017-08-29 21:58:15 1279 [Note] WSREP: Found saved state: ee3cf6b7-89c8-11e7-80c5-2aec0eea34fa:28447, safe_to_bootsrap: 0 2017-08-29 21:58:15 1279 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/percona-xtradb-cluster/; base_host = 172.29.239.12; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 2; evs.view_forget_timeout = PT24H; gcache.dir = /var/lib/percona-xtradb-cluster/; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/percona-xtradb-cluster//galera.cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segme 2017-08-29 21:58:15 1279 [Note] WSREP: GCache history reset: old(ee3cf6b7-89c8-11e7-80c5-2aec0eea34fa:0) -> new(ee3cf6b7-89c8-11e7-80c5-2aec0eea34fa:28447) 2017-08-29 21:58:15 1279 [Note] WSREP: Assign initial position for certification: 28447, protocol version: -1 2017-08-29 21:58:15 1279 [Note] WSREP: wsrep_sst_grab() 2017-08-29 21:58:15 1279 [Note] WSREP: Start replication 2017-08-29 21:58:15 1279 [Note] WSREP: Setting initial position to ee3cf6b7-89c8-11e7-80c5-2aec0eea34fa:28447 2017-08-29 21:58:15 1279 [Note] WSREP: protonet asio version 0 2017-08-29 21:58:15 1279 [Note] WSREP: Using CRC-32C for message checksums. 2017-08-29 21:58:15 1279 [Note] WSREP: backend: asio 2017-08-29 21:58:15 1279 [Note] WSREP: gcomm thread scheduling priority set to other:0 2017-08-29 21:58:15 1279 [Warning] WSREP: access file(/var/lib/percona-xtradb-cluster//gvwstate.dat) failed(No such file or directory) 2017-08-29 21:58:15 1279 [Note] WSREP: restore pc from disk failed 2017-08-29 21:58:15 1279 [Note] WSREP: GMCast version 0 2017-08-29 21:58:15 1279 [Note] WSREP: (28a10b0e, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567 2017-08-29 21:58:15 1279 [Note] WSREP: (28a10b0e, 'tcp://0.0.0.0:4567') multicast: , ttl: 1 2017-08-29 21:58:15 1279 [Note] WSREP: EVS version 0 2017-08-29 21:58:15 1279 [Note] WSREP: gcomm: connecting to group 'juju_cluster', peer '172.29.239.12:,172.29.239.17:,172.29.239.32:' 2017-08-29 21:58:15 1279 [Note] WSREP: (28a10b0e, 'tcp://0.0.0.0:4567') connection established to 28a10b0e tcp://172.29.239.12:4567 2017-08-29 21:58:15 1279 [Warning] WSREP: (28a10b0e, 'tcp://0.0.0.0:4567') address 'tcp://172.29.239.12:4567' points to own listening address, blacklisting 2017-08-29 21:58:18 1279 [Warning] WSREP: no nodes coming from prim view, prim not possible 2017-08-29 21:58:18 1279 [Note] WSREP: view(view_id(NON_PRIM,28a10b0e,1) memb { 28a10b0e,0 } joined { } left { } partitioned { }) 2017-08-29 21:58:18 1279 [Note] WSREP: (28a10b0e, 'tcp://0.0.0.0:4567') connection to peer 28a10b0e with addr tcp://172.29.239.12:4567 timed out, no messages seen in PT3S 2017-08-29 21:58:18 1279 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50272S), skipping check 2017-08-29 21:58:48 1279 [Note] WSREP: view((empty)) 2017-08-29 21:58:48 1279 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out) at gcomm/src/pc.cpp:connect():158 2017-08-29 21:58:48 1279 [ERROR] WSREP: gcs/src/gcs_core.cpp:gcs_core_open():208: Failed to open backend connection: -110 (Connection timed out) 2017-08-29 21:58:48 1279 [ERROR] WSREP: gcs/src/gcs.cpp:gcs_open():1391: Failed to open channel 'juju_cluster' at 'gcomm://172.29.239.12,172.29.239.17,172.29.239.32': -110 (Connection timed out) 2017-08-29 21:58:48 1279 [ERROR] WSREP: gcs connect failed: Connection timed out 2017-08-29 21:58:48 1279 [ERROR] WSREP: wsrep::connect(gcomm://172.29.239.12,172.29.239.17,172.29.239.32) failed: 7 2017-08-29 21:58:48 1279 [ERROR] Aborting
2017-08-29 21:58:48 1279 [Note] WSREP: Service disconnected. 2017-08-29 21:58:49 1279 [Note] WSREP: Some threads may fail to exit. 2017-08-29 21:58:49 1279 [Note] Binlog end 2017-08-29 21:58:49 1279 [Note] /usr/sbin/mysqld: Shutdown complete
170829 21:58:49 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Manoj:
I ran the two commands on the one container where mysqld had crashed.
ubuntu@ juju-6db2c9- 0-lxd-5: ~$ sudo myisamchk -r -q /var/lib/ percona- xtradb- cluster/ mysql/db percona- xtradb- cluster/ mysql/db' juju-6db2c9- 0-lxd-5: ~$ sudo myisamchk -r -q /var/lib/ percona- xtradb- cluster/ mysql/user percona- xtradb- cluster/ mysql/user'
- check record delete-chain
- recovering (with sort) MyISAM-table '/var/lib/
Data records: 22
- Fixing index 1
- Fixing index 2
ubuntu@
- check record delete-chain
- recovering (with sort) MyISAM-table '/var/lib/
Data records: 26
- Fixing index 1
Then I stopped and started that container.
THe daemon did not stay up long. It went down pretty soon with this in the error.log:
170829 21:58:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/ percona- xtradb- cluster 89c8-11e7- 80c5-2aec0eea34 fa:28447 pair 89c8-11e7- 80c5-2aec0eea34 fa:28447 to wsrep_start_ position defaults_ for_timestamp server option (see documentation for more details). libgalera_ smm.so' 89c8-11e7- 80c5-2aec0eea34 fa:28447, safe_to_bootsrap: 0 percona- xtradb- cluster/ ; base_host = 172.29.239.12; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_ keep_period = PT30S; evs.inactive_ check_period = PT0.5S; evs.inactive_ timeout = PT15S; evs.join_ retrans_ period = PT1S; evs.max_ install_ timeouts = 3; evs.send_window = 4; evs.stats_ report_ period = PT1M; evs.suspect_timeout = PT5S; evs.user_ send_window = 2; evs.view_ forget_ timeout = PT24H; gcache.dir = /var/lib/ percona- xtradb- cluster/ ; gcache. keep_pages_ count = 0; gcache. keep_pages_ size = 0; gcache.mem_size = 0; gcache.name = /var/lib/ percona- xtradb- cluster/ /galera. cache; gcache.page_size = 128M; gcache.recover = no; gcache.size = 128M; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_ q_hard_ limit = 922337203685477 5807; gcs.recv_ q_soft_ limit = 0.25; gcs.sync_donor = no; gmcast.segme 89c8-11e7- 80c5-2aec0eea34 fa:0) -> new(ee3cf6b7- 89c8-11e7- 80c5-2aec0eea34 fa:28447) 89c8-11e7- 80c5-2aec0eea34 fa:28447 lib/percona- xtradb- cluster/ /gvwstate. dat) failed(No such file or directory) 0.0.0.0: 4567') listening at tcp://0.0.0.0:4567 0.0.0.0: 4567') multicast: , ttl: 1 239.12: ,172.29. 239.17: ,172.29. 239.32: ' 0.0.0.0: 4567') connection established to 28a10b0e tcp://172. 29.239. 12:4567 0.0.0.0: 4567') address 'tcp:// 172.29. 239.12: 4567' points to own listening address, blacklisting id(NON_ PRIM,28a10b0e, 1) memb { 0.0.0.0: 4567') connection to peer 28a10b0e with addr tcp://172. 29.239. 12:4567 timed out, no messages seen in PT3S pc.cpp: connect( ):158 gcs_core. cpp:gcs_ core_open( ):208: Failed to open backend connection: -110 (Connection timed out) gcs.cpp: gcs_open( ):1391: Failed to open channel 'juju_cluster' at 'gcomm: //172.29. 239.12, 172.29. 239.17, 172.29. 239.32' : -110 (Connection timed out) connect( gcomm:/ /172.29. 239.12, 172.29. 239.17, 172.29. 239.32) failed: 7
170829 21:58:15 mysqld_safe Skipping wsrep-recover for ee3cf6b7-
170829 21:58:15 mysqld_safe Assigning ee3cf6b7-
2017-08-29 21:58:15 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_
2017-08-29 21:58:15 0 [Note] /usr/sbin/mysqld (mysqld 5.6.34-79.1-79.1) starting as process 1279 ...
2017-08-29 21:58:15 1279 [Note] WSREP: Read nil XID from storage engines, skipping position init
2017-08-29 21:58:15 1279 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/
2017-08-29 21:58:15 1279 [Note] WSREP: wsrep_load(): Galera 3.19(rXXXX) by Codership Oy <email address hidden> loaded successfully.
2017-08-29 21:58:15 1279 [Note] WSREP: CRC-32C: using "slicing-by-8" algorithm.
2017-08-29 21:58:15 1279 [Note] WSREP: Found saved state: ee3cf6b7-
2017-08-29 21:58:15 1279 [Note] WSREP: Passing config to GCS: base_dir = /var/lib/
2017-08-29 21:58:15 1279 [Note] WSREP: GCache history reset: old(ee3cf6b7-
2017-08-29 21:58:15 1279 [Note] WSREP: Assign initial position for certification: 28447, protocol version: -1
2017-08-29 21:58:15 1279 [Note] WSREP: wsrep_sst_grab()
2017-08-29 21:58:15 1279 [Note] WSREP: Start replication
2017-08-29 21:58:15 1279 [Note] WSREP: Setting initial position to ee3cf6b7-
2017-08-29 21:58:15 1279 [Note] WSREP: protonet asio version 0
2017-08-29 21:58:15 1279 [Note] WSREP: Using CRC-32C for message checksums.
2017-08-29 21:58:15 1279 [Note] WSREP: backend: asio
2017-08-29 21:58:15 1279 [Note] WSREP: gcomm thread scheduling priority set to other:0
2017-08-29 21:58:15 1279 [Warning] WSREP: access file(/var/
2017-08-29 21:58:15 1279 [Note] WSREP: restore pc from disk failed
2017-08-29 21:58:15 1279 [Note] WSREP: GMCast version 0
2017-08-29 21:58:15 1279 [Note] WSREP: (28a10b0e, 'tcp://
2017-08-29 21:58:15 1279 [Note] WSREP: (28a10b0e, 'tcp://
2017-08-29 21:58:15 1279 [Note] WSREP: EVS version 0
2017-08-29 21:58:15 1279 [Note] WSREP: gcomm: connecting to group 'juju_cluster', peer '172.29.
2017-08-29 21:58:15 1279 [Note] WSREP: (28a10b0e, 'tcp://
2017-08-29 21:58:15 1279 [Warning] WSREP: (28a10b0e, 'tcp://
2017-08-29 21:58:18 1279 [Warning] WSREP: no nodes coming from prim view, prim not possible
2017-08-29 21:58:18 1279 [Note] WSREP: view(view_
28a10b0e,0
} joined {
} left {
} partitioned {
})
2017-08-29 21:58:18 1279 [Note] WSREP: (28a10b0e, 'tcp://
2017-08-29 21:58:18 1279 [Warning] WSREP: last inactive check more than PT1.5S ago (PT3.50272S), skipping check
2017-08-29 21:58:48 1279 [Note] WSREP: view((empty))
2017-08-29 21:58:48 1279 [ERROR] WSREP: failed to open gcomm backend connection: 110: failed to reach primary view: 110 (Connection timed out)
at gcomm/src/
2017-08-29 21:58:48 1279 [ERROR] WSREP: gcs/src/
2017-08-29 21:58:48 1279 [ERROR] WSREP: gcs/src/
2017-08-29 21:58:48 1279 [ERROR] WSREP: gcs connect failed: Connection timed out
2017-08-29 21:58:48 1279 [ERROR] WSREP: wsrep::
2017-08-29 21:58:48 1279 [ERROR] Aborting
2017-08-29 21:58:48 1279 [Note] WSREP: Service disconnected.
2017-08-29 21:58:49 1279 [Note] WSREP: Some threads may fail to exit.
2017-08-29 21:58:49 1279 [Note] Binlog end
2017-08-29 21:58:49 1279 [Note] /usr/sbin/mysqld: Shutdown complete
170829 21:58:49 mysqld_safe mysqld from pid file /var/run/ mysqld/ mysqld. pid ended