Stuck in "ERROR 1047 (08S01): WSREP has not yet prepared node for application use" state

[Evan]

Using XtraDB Cluster version:

ii libmysqlclient18 5.6.x all Dummy package to satisfy dependencies when libperconaserverclient18 is installed
ii libperconaserverclient18.1 5.6.35-26.20-2.wheezy amd64 Percona XtraDB Cluster database client library
ii percona-xtrabackup 2.3.7-2.wheezy amd64 Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-client-5.6 5.6.35-26.20-2.wheezy amd64 Percona XtraDB Cluster database client binaries
ii percona-xtradb-cluster-common-5.6 5.6.35-26.20-2.wheezy amd64 Percona XtraDB Cluster database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-3 3.20-2.wheezy amd64 Metapackage for latest version of galera3.
ii percona-xtradb-cluster-galera-3.x 3.20-2.wheezy amd64 Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.6 5.6.35-26.20-2.wheezy amd64 Percona XtraDB Cluster database server binaries
ii xtrabackup

on Ubuntu 16.04 and sometime when you restart a cluster member it gets stuck in Preparing state trying to apply the transactions it has missed when it was stopped. You can see in the process list that is is stuck applying the same transactions forever. My transactions are small.

Once stuck I ran:

mysql -e "SHOW FULL PROCESSLIST;SHOW GLOBAL VARIABLES;SHOW STATUS LIKE 'wsrep%';SHOW ENGINE InnoDB STATUS;"

and the output was:

1 system user NULL Sleep 289 wsrep aborter idle NULL 0 0
2 system user NULL Sleep 580 applied write set 7038 NULL 0 0
3 system user NULL Sleep 287 NULL NULL 0 0
4 system user NULL Sleep 580 applied write set 7037 NULL 0 0
5 system user NULL Sleep 287 NULL NULL 0 0
6 system user NULL Sleep 287 NULL NULL 0 0
7 system user NULL Sleep 579 applied write set 7040 NULL 0 0
8 system user NULL Sleep 287 NULL NULL 0 0
9 system user NULL Sleep 580 rolled back 7036 NULL 0 0
10 system user NULL Sleep 580 committed 7035 NULL 0 0
11 system user NULL Sleep 580 committed 7034 NULL 0 0
12 system user NULL Sleep 580 committed 7033 NULL 0 0
13 system user NULL Sleep 287 NULL NULL 0 0
14 system user NULL Sleep 287 NULL NULL 0 0
15 system user NULL Sleep 287 NULL NULL 0 0
16 system user NULL Sleep 287 NULL NULL 0 0
17 system user NULL Sleep 287 NULL NULL 0 0
2100 root localhost NULL Query 0 init SHOW FULL PROCESSLIST 0 0
auto_increment_increment 1
auto_increment_offset 1
autocommit ON
automatic_sp_privileges ON
avoid_temporal_upgrade OFF
back_log 150
basedir /usr/
big_tables OFF
bind_address 0.0.0.0
binlog_cache_size 32768
binlog_checksum CRC32
binlog_direct_non_transactional_updates OFF
binlog_error_action IGNORE_ERROR
binlog_format ROW
binlog_gtid_simple_recovery OFF
binlog_max_flush_queue_time 0
binlog_order_commits ON
binlog_row_image FULL
binlog_rows_query_log_events OFF
binlog_stmt_cache_size 32768
binlogging_impossible_mode IGNORE_ERROR
block_encryption_mode aes-128-ecb
bulk_insert_buffer_size 16777216
character_set_client latin1
character_set_connection latin1
character_set_database latin1
character_set_filesystem binary
character_set_results latin1
character_set_server latin1
character_set_system utf8
character_sets_dir /usr/share/mysql/charsets/
collation_connection latin1_swedish_ci
collation_database latin1_swedish_ci
collation_server latin1_swedish_ci
completion_type NO_CHAIN
concurrent_insert ALWAYS
connect_timeout 5
core_file OFF
csv_mode
datadir /var/lib/mysql/
date_format %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
default_storage_engine InnoDB
default_tmp_storage_engine InnoDB
default_week_format 0
delay_key_write ON
delayed_insert_limit 100
delayed_insert_timeout 300
delayed_queue_size 1000
disconnect_on_expired_password ON
div_precision_increment 4
end_markers_in_json OFF
enforce_gtid_consistency OFF
enforce_storage_engine
eq_range_index_dive_limit 10
event_scheduler OFF
expand_fast_index_creation OFF
expire_logs_days 2
explicit_defaults_for_timestamp ON
extra_max_connections 1
extra_port 0
flush OFF
flush_time 0
foreign_key_checks ON
ft_boolean_syntax + -><()~*:""&|
ft_max_word_len 84
ft_min_word_len 4
ft_query_expansion_limit 20
ft_stopword_file (built-in)
general_log OFF
general_log_file /var/lib/mysql/dp1.log
group_concat_max_len 1024
gtid_deployment_step OFF
gtid_executed
gtid_mode OFF
gtid_owned
gtid_purged
have_backup_locks YES
have_backup_safe_binlog_info YES
have_compress YES
have_crypt YES
have_dynamic_loading YES
have_elliptic_curve_crypto YES
have_geometry YES
have_openssl YES
have_profiling YES
have_query_cache YES
have_rtree_keys YES
have_snapshot_cloning YES
have_ssl YES
have_statement_timeout YES
have_symlink YES
have_tlsv1_2 YES
host_cache_size 628
hostname dp1
ignore_builtin_innodb OFF
ignore_db_dirs
init_connect
init_file
init_slave
innodb_adaptive_flushing ON
innodb_adaptive_flushing_lwm 10
innodb_adaptive_hash_index ON
innodb_adaptive_hash_index_partitions 1
innodb_adaptive_max_sleep_delay 150000
innodb_additional_mem_pool_size 8388608
innodb_api_bk_commit_interval 5
innodb_api_disable_rowlock OFF
innodb_api_enable_binlog OFF
innodb_api_enable_mdl OFF
innodb_api_trx_level 0
innodb_autoextend_increment 64
innodb_autoinc_lock_mode 2
innodb_buffer_pool_dump_at_shutdown OFF
innodb_buffer_pool_dump_now OFF
innodb_buffer_pool_filename ib_buffer_pool
innodb_buffer_pool_instances 8
innodb_buffer_pool_load_abort OFF
innodb_buffer_pool_load_at_startup OFF
innodb_buffer_pool_load_now OFF
innodb_buffer_pool_populate OFF
innodb_buffer_pool_size 1039138816
innodb_change_buffer_max_size 25
innodb_change_buffering all
innodb_checksum_algorithm innodb
innodb_checksums ON
innodb_cleaner_lsn_age_factor high_checkpoint
innodb_cmp_per_index_enabled OFF
innodb_commit_concurrency 0
innodb_compressed_columns_threshold 96
innodb_compressed_columns_zip_level 6
innodb_compression_failure_threshold_pct 5
innodb_compression_level 6
innodb_compression_pad_pct_max 50
innodb_concurrency_tickets 5000
innodb_corrupt_table_action assert
innodb_data_file_path ibdata1:12M:autoextend
innodb_data_home_dir
innodb_disable_sort_file_cache OFF
innodb_disallow_writes OFF
innodb_doublewrite ON
innodb_empty_free_list_algorithm backoff
innodb_fast_shutdown 1
innodb_file_format Antelope
innodb_file_format_check ON
innodb_file_format_max Antelope
innodb_file_per_table OFF
innodb_flush_log_at_timeout 1
innodb_flush_log_at_trx_commit 2
innodb_flush_method O_DIRECT
innodb_flush_neighbors 1
innodb_flushing_avg_loops 30
innodb_force_load_corrupted OFF
innodb_force_recovery 0
innodb_foreground_preflush exponential_backoff
innodb_ft_aux_table
innodb_ft_cache_size 8000000
innodb_ft_enable_diag_print OFF
innodb_ft_enable_stopword ON
innodb_ft_max_token_size 84
innodb_ft_min_token_size 3
innodb_ft_num_word_optimize 2000
innodb_ft_result_cache_limit 2000000000
innodb_ft_server_stopword_table
innodb_ft_sort_pll_degree 2
innodb_ft_total_cache_size 640000000
innodb_ft_user_stopword_table
innodb_io_capacity 400
innodb_io_capacity_max 2000
innodb_kill_idle_transaction 0
innodb_large_prefix OFF
innodb_lock_wait_timeout 50
innodb_locks_unsafe_for_binlog OFF
innodb_log_arch_dir ./
innodb_log_arch_expire_sec 0
innodb_log_archive OFF
innodb_log_block_size 512
innodb_log_buffer_size 8388608
innodb_log_checksum_algorithm innodb
innodb_log_compressed_pages ON
innodb_log_file_size 268435456
innodb_log_files_in_group 2
innodb_log_group_home_dir ./
innodb_lru_scan_depth 1024
innodb_max_bitmap_file_size 104857600
innodb_max_changed_pages 1000000
innodb_max_dirty_pages_pct 75
innodb_max_dirty_pages_pct_lwm 0
innodb_max_purge_lag 0
innodb_max_purge_lag_delay 0
innodb_mirrored_log_groups 1
innodb_monitor_disable
innodb_monitor_enable
innodb_monitor_reset
innodb_monitor_reset_all
innodb_numa_interleave OFF
innodb_old_blocks_pct 37
innodb_old_blocks_time 1000
innodb_online_alter_log_max_size 134217728
innodb_open_files 400
innodb_optimize_fulltext_only OFF
innodb_page_size 16384
innodb_print_all_deadlocks OFF
innodb_purge_batch_size 300
innodb_purge_threads 1
innodb_random_read_ahead OFF
innodb_read_ahead_threshold 56
innodb_read_io_threads 4
innodb_read_only OFF
innodb_replication_delay 0
innodb_rollback_on_timeout OFF
innodb_rollback_segments 128
innodb_sched_priority_cleaner 19
innodb_show_locks_held 10
innodb_show_verbose_locks 0
innodb_sort_buffer_size 1048576
innodb_spin_wait_delay 6
innodb_stats_auto_recalc ON
innodb_stats_include_delete_marked OFF
innodb_stats_method nulls_equal
innodb_stats_on_metadata OFF
innodb_stats_persistent ON
innodb_stats_persistent_sample_pages 20
innodb_stats_sample_pages 8
innodb_stats_transient_sample_pages 8
innodb_status_output OFF
innodb_status_output_locks OFF
innodb_strict_mode OFF
innodb_support_xa ON
innodb_sync_array_size 1
innodb_sync_spin_loops 30
innodb_table_locks ON
innodb_thread_concurrency 0
innodb_thread_sleep_delay 10000
innodb_tmpdir
innodb_track_changed_pages OFF
innodb_undo_directory .
innodb_undo_logs 128
innodb_undo_tablespaces 0
innodb_use_atomic_writes OFF
innodb_use_global_flush_log_at_trx_commit ON
innodb_use_native_aio ON
innodb_use_sys_malloc ON
innodb_version 5.6.35-80.0
innodb_write_io_threads 4
interactive_timeout 28800
join_buffer_size 262144
keep_files_on_create OFF
key_buffer_size 134217728
key_cache_age_threshold 300
key_cache_block_size 1024
key_cache_division_limit 100
kill_idle_transaction 0
large_files_support ON
large_page_size 0
large_pages OFF
lc_messages en_US
lc_messages_dir /usr/share/mysql/
lc_time_names en_US
license GPL
local_infile ON
lock_wait_timeout 31536000
locked_in_memory OFF
log_bin ON
log_bin_basename /var/log/mysql/mysql-bin
log_bin_index /var/log/mysql/mysql-bin.index
log_bin_trust_function_creators OFF
log_bin_use_v1_row_events OFF
log_error
log_output FILE
log_queries_not_using_indexes OFF
log_slave_updates OFF
log_slow_admin_statements OFF
log_slow_filter
log_slow_rate_limit 1
log_slow_rate_type session
log_slow_slave_statements OFF
log_slow_sp_statements ON
log_slow_verbosity query_plan
log_throttle_queries_not_using_indexes 0
log_warnings 2
log_warnings_suppress
long_query_time 10.000000
low_priority_updates OFF
lower_case_file_system OFF
lower_case_table_names 0
master_info_repository FILE
master_verify_checksum OFF
max_allowed_packet 134217728
max_binlog_cache_size 18446744073709547520
max_binlog_files 0
max_binlog_size 104857600
max_binlog_stmt_cache_size 18446744073709547520
max_connect_errors 100
max_connections 500
max_delayed_threads 20
max_digest_length 1024
max_error_count 64
max_heap_table_size 33554432
max_insert_delayed_threads 20
max_join_size 18446744073709551615
max_length_for_sort_data 1024
max_prepared_stmt_count 16382
max_relay_log_size 0
max_seeks_for_key 18446744073709551615
max_slowlog_files 0
max_slowlog_size 0
max_sort_length 1024
max_sp_recursion_depth 0
max_statement_time 0
max_tmp_tables 32
max_user_connections 0
max_write_lock_count 18446744073709551615
metadata_locks_cache_size 1024
metadata_locks_hash_instances 8
min_examined_row_limit 0
multi_range_count 256
myisam_data_pointer_size 6
myisam_max_sort_file_size 9223372036853727232
myisam_mmap_size 18446744073709551615
myisam_recover_options BACKUP
myisam_repair_threads 1
myisam_sort_buffer_size 536870912
myisam_stats_method nulls_unequal
myisam_use_mmap OFF
net_buffer_length 16384
net_read_timeout 30
net_retry_count 10
net_write_timeout 60
new OFF
old OFF
old_alter_table OFF
old_passwords 0
open_files_limit 1000000
optimizer_prune_level 1
optimizer_search_depth 62
optimizer_switch index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,subquery_materialization_cost_based=on,use_index_extensions=on
optimizer_trace enabled=off,one_line=off
optimizer_trace_features greedy_search=on,range_optimizer=on,dynamic_range=on,repeated_subselect=on
optimizer_trace_limit 1
optimizer_trace_max_mem_size 16384
optimizer_trace_offset -1
performance_schema ON
performance_schema_accounts_size 100
performance_schema_digests_size 10000
performance_schema_events_stages_history_long_size 10000
performance_schema_events_stages_history_size 10
performance_schema_events_statements_history_long_size 10000
performance_schema_events_statements_history_size 10
performance_schema_events_waits_history_long_size 10000
performance_schema_events_waits_history_size 10
performance_schema_hosts_size 100
performance_schema_max_cond_classes 80
performance_schema_max_cond_instances 3300
performance_schema_max_digest_length 1024
performance_schema_max_file_classes 50
performance_schema_max_file_handles 32768
performance_schema_max_file_instances 1538462
performance_schema_max_mutex_classes 200
performance_schema_max_mutex_instances 11000
performance_schema_max_rwlock_classes 40
performance_schema_max_rwlock_instances 5000
performance_schema_max_socket_classes 10
performance_schema_max_socket_instances 1020
performance_schema_max_stage_classes 150
performance_schema_max_statement_classes 181
performance_schema_max_table_handles 800
performance_schema_max_table_instances 12500
performance_schema_max_thread_classes 50
performance_schema_max_thread_instances 1100
performance_schema_session_connect_attrs_size 512
performance_schema_setup_actors_size 100
performance_schema_setup_objects_size 100
performance_schema_users_size 100
pid_file /var/run/mysqld/mysqld.pid
plugin_dir /usr/lib/mysql/plugin/
port 3306
preload_buffer_size 32768
profiling OFF
profiling_history_size 15
protocol_version 10
proxy_protocol_networks
query_alloc_block_size 8192
query_cache_limit 131072
query_cache_min_res_unit 4096
query_cache_size 67108864
query_cache_strip_comments OFF
query_cache_type OFF
query_cache_wlock_invalidate OFF
query_prealloc_size 8192
range_alloc_block_size 4096
read_buffer_size 2097152
read_only OFF
read_rnd_buffer_size 1048576
relay_log
relay_log_basename
relay_log_index
relay_log_info_file relay-log.info
relay_log_info_repository FILE
relay_log_purge ON
relay_log_recovery OFF
relay_log_space_limit 0
report_host
report_password
report_port 3306
report_user
rpl_stop_slave_timeout 31536000
secure_auth ON
secure_file_priv /var/lib/mysql-files/
server_id 1
server_id_bits 32
server_uuid 9befe204-7b4f-11e7-bc47-0800277797ec
sha256_password_private_key_path private_key.pem
sha256_password_public_key_path public_key.pem
show_old_temporals OFF
simplified_binlog_gtid_recovery OFF
skip_external_locking ON
skip_name_resolve OFF
skip_networking OFF
skip_show_database OFF
slave_allow_batching OFF
slave_checkpoint_group 512
slave_checkpoint_period 300
slave_compressed_protocol OFF
slave_exec_mode STRICT
slave_load_tmpdir /tmp
slave_max_allowed_packet 1073741824
slave_net_timeout 3600
slave_parallel_workers 0
slave_pending_jobs_size_max 16777216
slave_rows_search_algorithms TABLE_SCAN,INDEX_SCAN
slave_skip_errors OFF
slave_sql_verify_checksum ON
slave_transaction_retries 10
slave_type_conversions
slow_launch_time 2
slow_query_log OFF
slow_query_log_always_write_time 10.000000
slow_query_log_file /var/log/mysql/mariadb-slow.log
slow_query_log_timestamp_always OFF
slow_query_log_timestamp_precision second
slow_query_log_use_global_control
socket /var/run/mysqld/mysqld.sock
sort_buffer_size 4194304
sql_auto_is_null OFF
sql_big_selects ON
sql_buffer_result OFF
sql_log_bin ON
sql_log_off OFF
sql_mode NO_ENGINE_SUBSTITUTION
sql_notes ON
sql_quote_show_create ON
sql_safe_updates OFF
sql_select_limit 18446744073709551615
sql_slave_skip_counter 0
sql_warnings OFF
ssl_ca /etc/mysql/certs/ca-cert.pem
ssl_capath
ssl_cert /etc/mysql/certs/server-cert.pem
ssl_cipher
ssl_crl
ssl_crlpath
ssl_key /etc/mysql/certs/server-key.pem
storage_engine InnoDB
stored_program_cache 256
super_read_only OFF
sync_binlog 0
sync_frm ON
sync_master_info 10000
sync_relay_log 10000
sync_relay_log_info 10000
system_time_zone UTC
table_definition_cache 600
table_open_cache 400
table_open_cache_instances 1
thread_cache_size 16
thread_concurrency 10
thread_handling one-thread-per-connection
thread_pool_high_prio_mode transactions
thread_pool_high_prio_tickets 4294967295
thread_pool_idle_timeout 60
thread_pool_max_threads 100000
thread_pool_oversubscribe 3
thread_pool_size 2
thread_pool_stall_limit 500
thread_stack 262144
thread_statistics OFF
time_format %H:%i:%s
time_zone SYSTEM
timed_mutexes OFF
tls_version TLSv1.1,TLSv1.2
tmp_table_size 33554432
tmpdir /tmp
transaction_alloc_block_size 8192
transaction_prealloc_size 4096
tx_isolation READ-COMMITTED
tx_read_only OFF
unique_checks ON
updatable_views_with_limit YES
userstat OFF
version 5.6.35-80.0-56-log
version_comment Percona XtraDB Cluster (GPL), Release rel80.0, Revision 62752df, WSREP version 26.20, wsrep_26.20
version_compile_machine x86_64
version_compile_os debian-linux-gnu
wait_timeout 600
wsrep_OSU_method TOI
wsrep_auto_increment_control OFF
wsrep_causal_reads OFF
wsrep_certify_nonPK ON
wsrep_cluster_address gcomm://192.168.56.101,192.168.56.102
wsrep_cluster_name Devportal Cluster
wsrep_convert_LOCK_to_trx OFF
wsrep_data_home_dir /var/lib/mysql/
wsrep_dbug_option
wsrep_debug OFF
wsrep_desync OFF
wsrep_dirty_reads OFF
wsrep_drupal_282555_workaround OFF
wsrep_forced_binlog_format NONE
wsrep_load_data_splitting ON
wsrep_log_conflicts OFF
wsrep_max_ws_rows 0
wsrep_max_ws_size 2147483647
wsrep_mysql_replication_bundle 0
wsrep_node_address 192.168.56.101
wsrep_node_incoming_address AUTO
wsrep_node_name dp1
wsrep_notify_cmd /home/admin/bin/apim_wsrep_notify
wsrep_preordered OFF
wsrep_provider /usr/lib/libgalera_smm.so
wsrep_provider_options base_dir = /var/lib/mysql/; base_host = 192.168.56.101; base_port = 4567; cert.log_conflicts = no; debug = no; evs.auto_evict = 0; evs.causal_keepalive_period = PT1S; evs.debug_log_mask = 0x1; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.info_log_mask = 0; evs.install_timeout = PT7.5S; evs.join_retrans_period = PT1S; evs.keepalive_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 4; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.use_aggregate = true; evs.user_send_window = 2; evs.version = 0; evs.view_forget_timeout = P1D; gcache.dir = /var/lib/mysql/; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 3G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 16; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.listen_addr = ssl://0.0.0.0:4567; gmcast.mcast_addr = ; gmcast.mcast_ttl = 1; gmcast.peer_timeout = PT3S; gmcast.segment = 0; gmcast.time_wait = PT5S; gmcast.version = 0; ist.recv_addr = 192.168.56.101; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.linger = PT20S; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 7; socket.checksum = 2; socket.recv_buf_size = 212992; socket.ssl = YES; socket.ssl_ca = /etc/mysql/certs/ca-cert.pem; socket.ssl_cert = /etc/mysql/certs/server-cert.pem; socket.ssl_cipher = AES128-SHA; socket.ssl_compression = YES; socket.ssl_key = /etc/mysql/certs/server-key.pem;
wsrep_recover OFF
wsrep_reject_queries NONE
wsrep_replicate_myisam OFF
wsrep_restart_slave OFF
wsrep_retry_autocommit 5
wsrep_slave_FK_checks ON
wsrep_slave_UK_checks OFF
wsrep_slave_threads 16
wsrep_sst_auth ********
wsrep_sst_donor
wsrep_sst_donor_rejects_queries OFF
wsrep_sst_method xtrabackup-v2
wsrep_sst_receive_address AUTO
wsrep_start_position 1e0346fe-7b4d-11e7-8377-12ad9b3068ea:7029
wsrep_sync_wait 0
wsrep_local_state_uuid 1e0346fe-7b4d-11e7-8377-12ad9b3068ea
wsrep_protocol_version 7
wsrep_last_committed 7035
wsrep_replicated 0
wsrep_replicated_bytes 0
wsrep_repl_keys 0
wsrep_repl_keys_bytes 0
wsrep_repl_data_bytes 0
wsrep_repl_other_bytes 0
wsrep_received 1
wsrep_received_bytes 202
wsrep_local_commits 0
wsrep_local_cert_failures 0
wsrep_local_replays 0
wsrep_local_send_queue 0
wsrep_local_send_queue_max 2
wsrep_local_send_queue_min 0
wsrep_local_send_queue_avg 0.333333
wsrep_local_recv_queue 451
wsrep_local_recv_queue_max 451
wsrep_local_recv_queue_min 0
wsrep_local_recv_queue_avg 224.502212
wsrep_local_cached_downto 0
wsrep_flow_control_paused_ns 0
wsrep_flow_control_paused 0.000000
wsrep_flow_control_sent 0
wsrep_flow_control_recv 0
wsrep_cert_deps_distance 0.000000
wsrep_apply_oooe 0.700000
wsrep_apply_oool 0.000000
wsrep_apply_window 6.900000
wsrep_commit_oooe 0.000000
wsrep_commit_oool 0.000000
wsrep_commit_window 3.000000
wsrep_local_state 1
wsrep_local_state_comment Joining: receiving State Transfer
wsrep_cert_index_size 0
wsrep_cert_bucket_count 22
wsrep_gcache_pool_size 61130905
wsrep_causal_reads 0
wsrep_cert_interval 0.000000
wsrep_incoming_addresses 192.168.56.102:3306,192.168.56.101:3306
wsrep_desync_count 0
wsrep_evs_delayed
wsrep_evs_evict_list
wsrep_evs_repl_latency 0/0/0/0/0
wsrep_evs_state OPERATIONAL
wsrep_gcomm_uuid 2e08bbed-7b50-11e7-97dd-8f1d9334154c
wsrep_cluster_conf_id 2
wsrep_cluster_size 2
wsrep_cluster_state_uuid 1e0346fe-7b4d-11e7-8377-12ad9b3068ea
wsrep_cluster_status Primary
wsrep_connected ON
wsrep_local_bf_aborts 0
wsrep_local_index 1
wsrep_provider_name Galera
wsrep_provider_vendor Codership Oy <email address hidden>
wsrep_provider_version 3.20(r7e383f7)
wsrep_ready OFF
InnoDB
=====================================
2017-08-07 09:14:45 7f87b3f7d700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 24 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 2 srv_active, 0 srv_shutdown, 284 srv_idle
srv_master_thread log flush and writes: 286
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 17
OS WAIT ARRAY INFO: signal count 14
Mutex spin waits 16, rounds 345, OS waits 9
RW-shared spins 6, rounds 151, OS waits 5
RW-excl spins 1, rounds 60, OS waits 2
Spin rounds per wait: 21.56 mutex, 25.17 RW-shared, 60.00 RW-excl
------------
TRANSACTIONS
------------
Trx id counter 33315
Purge done for trx's n:o < 33315 undo n:o < 0 state: running but idle
History list length 2358
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 0, not started
MySQL thread id 2100, OS thread handle 0x7f87b3f7d700, query id 6201 localhost root init
SHOW ENGINE InnoDB STATUS
---TRANSACTION 33302, not started
MySQL thread id 11, OS thread handle 0x7f87bc9a1700, query id 13 committed 7034
---TRANSACTION 33301, not started
MySQL thread id 10, OS thread handle 0x7f87bc91f700, query id 12 committed 7035
---TRANSACTION 33298, not started
MySQL thread id 9, OS thread handle 0x7f87bc85c700, query id 10 rolled back 7036
---TRANSACTION 33291, not started
MySQL thread id 12, OS thread handle 0x7f87bc9e2700, query id 5 committed 7033
---TRANSACTION 33306, ACTIVE 287 sec
2 lock struct(s), heap size 360, 1 row lock(s), undo log entries 1
MySQL thread id 4, OS thread handle 0x7f87bca23700, query id 14 applied write set 7037
---TRANSACTION 33300, ACTIVE 287 sec
2 lock struct(s), heap size 360, 1 row lock(s), undo log entries 1
MySQL thread id 7, OS thread handle 0x7f87bc89d700, query id 11 applied write set 7040
---TRANSACTION 33296, ACTIVE 287 sec
1 lock struct(s), heap size 360, 0 row lock(s), undo log entries 1
MySQL thread id 2, OS thread handle 0x7f880dc44700, query id 8 applied write set 7038
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: 0 [0, 0, 0, 0] , aio writes: 0 [0, 0, 0, 0] ,
 ibuf aio reads: 0, log i/o's: 0, sync i/o's: 0
Pending flushes (fsync) log: 0; buffer pool: 0
544 OS file reads, 64 OS file writes, 17 OS fsyncs
0.00 reads/s, 0 avg bytes/read, 0.00 writes/s, 0.00 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
 insert 0, delete mark 0, delete 0
discarded operations:
 insert 0, delete mark 0, delete 0
0.00 hash searches/s, 0.00 non-hash searches/s
---
LOG
---
Log sequence number 73099936
Log flushed up to 73099936
Pages flushed up to 73099936
Last checkpoint at 73099936
Max checkpoint age 434154333
Checkpoint age target 420587011
Modified age 0
Checkpoint age 0
0 pending log writes, 0 pending chkp writes
26 log i/o's done, 0.00 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total memory allocated 1081769984; in additional pool allocated 0
Total memory allocated by read views 112
Internal hash tables (constant factor + variable factor)
    Adaptive hash index 18334688 (18314104 + 20584)
    Page hash 1145912 (buffer pool 0 only)
    Dictionary cache 4862613 (4580624 + 281989)
    File system 348112 (82672 + 265440)
    Lock system 2574424 (2571544 + 2880)
    Recovery system 0 (0 + 0)
Dictionary memory allocated 281989
Buffer pool size 63423
Buffer pool size, bytes 1039122432
Free buffers 62908
Database pages 514
Old database pages 209
Modified db pages 0
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 510, created 4, written 41
0.00 reads/s, 0.00 creates/s, 0.00 writes/s
No buffer pool page gets since the last printout
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 514, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
3 RW transactions active inside InnoDB
0 RO transactions active inside InnoDB
3 out of 1000 descriptors used
Main thread process no. 24110, id 140220943857408, state: sleeping
Number of rows inserted 4, updated 2, deleted 3, read 5
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================

[Evan (eskinner)]

It is possible that this only happens if ntp jumps the clock forward by a significant amount (several minutes perhaps) whilst mysqld is running.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-2004