Crash of Percona XtraDB Cluster 5.6 node on UPDATE

[Vitalii Klapkiv]

Crash of Percona XtraDB Cluster 5.6 node on UPDATE

Linux serv 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 GNU/Linux

percona-xtrabackup 2.1.8-733-1.wheezy amd64
percona-xtradb-cluster-client-5.6 5.6.15-25.5-759.wheezy amd64
percona-xtradb-cluster-common-5.6 5.6.15-25.5-759.wheezy amd64
percona-xtradb-cluster-galera-3.x 213.wheezy amd64
percona-xtradb-cluster-server-5.6 5.6.15-25.5-759.wheezy amd64

Node permanently crashes on UPDATE query :

backtrace :

Apr 17 21:30:58 web177 mysqld: 2014-04-17 21:30:58 21598 [Note] WSREP: Created page /var/lib/mysql/gcache.page.000000 of size
134217728 bytes
Apr 17 21:31:07 web177 mysqld: 18:31:07 UTC - mysqld got signal 6 ;
Apr 17 21:31:07 web177 mysqld: This could be because you hit a bug. It is also possible that this binary
Apr 17 21:31:07 web177 mysqld: or one of the libraries it was linked against is corrupt, improperly built,
Apr 17 21:31:07 web177 mysqld: or misconfigured. This error can also be caused by malfunctioning hardware.
Apr 17 21:31:07 web177 mysqld: We will try our best to scrape up some info that will hopefully help
Apr 17 21:31:07 web177 mysqld: diagnose the problem, but since we have already crashed,
Apr 17 21:31:07 web177 mysqld: something is definitely wrong and this may fail.
Apr 17 21:31:07 web177 mysqld: Please help us make Percona XtraDB Cluster better by reporting any
Apr 17 21:31:07 web177 mysqld: bugs at https://bugs.launchpad.net/percona-xtradb-cluster
Apr 17 21:31:07 web177 mysqld:
Apr 17 21:31:07 web177 mysqld: key_buffer_size=268435456
Apr 17 21:31:07 web177 mysqld: read_buffer_size=33554432
Apr 17 21:31:07 web177 mysqld: max_used_connections=87
Apr 17 21:31:07 web177 mysqld: max_threads=602
Apr 17 21:31:07 web177 mysqld: thread_count=69
Apr 17 21:31:07 web177 mysqld: connection_count=69
Apr 17 21:31:07 web177 mysqld: It is possible that mysqld could use up to
Apr 17 21:31:07 web177 mysqld: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 39725325 K bytes of memo
ry
Apr 17 21:31:07 web177 mysqld: Hope that's ok; if not, decrease some variables in the equation.
Apr 17 21:31:07 web177 mysqld:
Apr 17 21:31:07 web177 mysqld: Thread pointer: 0x7fab6c6d6f40
Apr 17 21:31:07 web177 mysqld: Attempting backtrace. You can use the following information to find out
Apr 17 21:31:07 web177 mysqld: where mysqld died. If you see no messages after this, something went
Apr 17 21:31:07 web177 mysqld: terribly wrong...
Apr 17 21:31:07 web177 mysqld: stack_bottom = 7fab88bafe30 thread_stack 0x30000
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x8ed9fe]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(handle_fatal_signal+0x4a1)[0x6b8181]
Apr 17 21:31:07 web177 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0xf030)[0x7fac9dfa9030]
Apr 17 21:31:07 web177 mysqld: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7fac9c3ef475]
Apr 17 21:31:07 web177 mysqld: /lib/x86_64-linux-gnu/libc.so.6(abort+0x180)[0x7fac9c3f26f0]
Apr 17 21:31:07 web177 mysqld: /usr/lib/libgalera_smm.so(_ZN6galera13Certification16purge_for_trx_v3EPNS_9TrxHandleE+0x255)[0x
7fac7d051645]
Apr 17 21:31:07 web177 mysqld: /usr/lib/libgalera_smm.so(_ZNK6galera13Certification15PurgeAndDiscardclERSt4pairIKlPNS_9TrxHand
leEE+0xd1)[0x7fac7d059211]
Apr 17 21:31:07 web177 mysqld: /usr/lib/libgalera_smm.so(_ZSt8for_eachISt17_Rb_tree_iteratorISt4pairIKlPN6galera9TrxHandleEEEN
S3_13Certification15PurgeAndDiscardEET0_T_SB_SA_+0x2c)[0x7fac7d05945c]
Apr 17 21:31:07 web177 mysqld: /usr/lib/libgalera_smm.so(_ZN6galera13Certification16purge_trxs_upto_Elb+0x72)[0x7fac7d052d82]
Apr 17 21:31:07 web177 mysqld: /usr/lib/libgalera_smm.so(_ZN6galera13Certification10append_trxEPNS_9TrxHandleE+0x24e)[0x7fac7d
056cae]
Apr 17 21:31:07 web177 mysqld: /usr/lib/libgalera_smm.so(_ZN6galera13ReplicatorSMM4certEPNS_9TrxHandleE+0x8a)[0x7fac7d07ecfa]
Apr 17 21:31:07 web177 mysqld: /usr/lib/libgalera_smm.so(_ZN6galera13ReplicatorSMM10pre_commitEPNS_9TrxHandleEP14wsrep_trx_meta+0x59)[0x7fac7d07f839]
Apr 17 21:31:07 web177 mysqld: /usr/lib/libgalera_smm.so(galera_pre_commit+0x140)[0x7fac7d08f9d0]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(_Z22wsrep_run_wsrep_commitP3THDP10handlertonb+0x5fd)[0x7c154d]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(_Z14ha_prepare_lowP3THDb+0x84)[0x617fc4]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(_Z15ha_commit_transP3THDbb+0x156)[0x6174a6]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(_Z17trans_commit_stmtP3THD+0x37)[0x7a8c17]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x5d0)[0x7296b0]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5b8)[0x72f3b8]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld[0x72f4b8]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xbaa)[0x73058a]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(_Z10do_commandP3THD+0x169)[0x7316b9]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x167)[0x703f67]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(handle_one_connection+0x47)[0x7040d7]
Apr 17 21:31:07 web177 mysqld: /usr/sbin/mysqld(pfs_spawn_thread+0x143)[0x985e13]
Apr 17 21:31:07 web177 mysqld: /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7fac9dfa0b50]
Apr 17 21:31:07 web177 mysqld: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fac9c4990ed]
Apr 17 21:31:07 web177 mysqld:
Apr 17 21:31:07 web177 mysqld: Trying to get some variables.
Apr 17 21:31:07 web177 mysqld: Some pointers may be invalid and cause the dump to abort.
Apr 17 21:31:07 web177 mysqld: Query (9706530): UPDATE n5chk2_k2_items
Apr 17 21:31:07 web177 mysqld: SET `hits` = (`hits` + 1)
Apr 17 21:31:07 web177 mysqld: WHERE `id` = '336616'
Apr 17 21:31:07 web177 mysqld: Connection ID (thread ID): 55272
Apr 17 21:31:07 web177 mysqld: Status: NOT_KILLED
Apr 17 21:31:07 web177 mysqld:
Apr 17 21:31:07 web177 mysqld: You may download the Percona XtraDB Cluster operations manual by visiting
Apr 17 21:31:07 web177 mysqld: http://www.percona.com/software/percona-xtradb-cluster/. You may find information
Apr 17 21:31:07 web177 mysqld: in the manual which will help you identify the cause of the crash.
Apr 17 21:31:07 web177 mysqld_safe: Number of processes running now: 0
Apr 17 21:31:07 web177 mysqld_safe: WSREP: not restarting wsrep node automatically
Apr 17 21:31:07 web177 mysqld_safe: mysqld from pid file /var/run/mysqld/mysqld.pid ended

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1668