Crash with debug build and PFS | storage/perfschema/pfs.cc:4527: PSI_statement_locker* refine_statement_v1(PSI_statement_locker*, PSI_statement_key): Assertion `klass->is_mutable()' failed

[Raghavendra D Prabhu]

mysqld: /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/storage/perfschema/pfs.cc:4527: PSI_statement_locker* refine_statement_v1(PSI_statement_locker*, PSI_statement_key): Assertion `klass->is_mutable()' failed.
23:44:29 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=19
max_threads=153
thread_count=19
connection_count=19
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62061 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x49bda20
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa2581b2d20 thread_stack 0x40000
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld(my_print_stacktrace+0x35)[0xaed34c]
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld(handle_fatal_signal+0x42e)[0x7547e2]
/lib64/libpthread.so.0(+0xf710)[0x7fa285615710]
/lib64/libc.so.6(gsignal+0x35)[0x7fa283c75925]
/lib64/libc.so.6(abort+0x175)[0x7fa283c77105]
/lib64/libc.so.6(+0x2ba4e)[0x7fa283c6ea4e]
/lib64/libc.so.6(__assert_perror_fail+0x0)[0x7fa283c6eb10]
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld[0xb43618]
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld[0x80428d]
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x3b5)[0x817e4c]
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld[0x8174e0]
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x10eb)[0x80854d]
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld(_Z10do_commandP3THD+0x70c)[0x806dd7]
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x1d4)[0x7cd8a1]
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld(handle_one_connection+0x33)[0x7cd388]
/rqg/workspace/PXC-5.5-to-5.6-bench/BTYPE/release/Host/pxc-rqg/Percona-XtraDB-Cluster-5.6.15-25.2.708.Linux.x86_64/bin/mysqld(pfs_spawn_thread+0x159)[0xb3e1bb]
/lib64/libpthread.so.0(+0x79d1)[0x7fa28560d9d1]
/lib64/libc.so.6(clone+0x6d)[0x7fa283d2bb6d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7fa1b0004c30): INSERT IGNORE INTO testdb.sbtest89910 SELECT * FROM sbtest89
Connection ID (thread ID): 21
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file

[Raghavendra D Prabhu (raghavendra-prabhu)]

Full bt http://paste.wnohang.net/042012

[Raghavendra D Prabhu (raghavendra-prabhu)]

Releavant backtrace (from full bt):

#0 0x00007fa2856128ac in pthread_kill () from /lib64/libpthread.so.0
#1 0x0000000000aed3ce in my_write_core (sig=6) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/mysys/stacktrace.c:422
#2 0x0000000000754a1d in handle_fatal_signal (sig=6) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/signal_handler.cc:254
#3 <signal handler called>
#4 0x00007fa283c75925 in raise () from /lib64/libc.so.6
#5 0x00007fa283c77105 in abort () from /lib64/libc.so.6
#6 0x00007fa283c6ea4e in __assert_fail_base () from /lib64/libc.so.6
#7 0x00007fa283c6eb10 in __assert_fail () from /lib64/libc.so.6
#8 0x0000000000b43618 in refine_statement_v1 (locker=0x49bf588, key=7) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/storage/perfschema/pfs.cc:4527
#9 0x000000000080428d in inline_mysql_refine_statement (locker=0x49bf588, key=7) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/include/mysql/psi/mysql_statement.h:172
#10 0x0000000000817e4c in mysql_parse (thd=0x49bda20, rawbuf=0x7fa1b0004c30 "INSERT IGNORE INTO testdb.sbtest89910 SELECT * FROM sbtest89", length=60, parser_state=0x7fa2581b2720) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:7139
#11 0x00000000008174e0 in wsrep_mysql_parse (thd=0x49bda20, rawbuf=0x7fa1b0004c30 "INSERT IGNORE INTO testdb.sbtest89910 SELECT * FROM sbtest89", length=60, parser_state=0x7fa2581b2720) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:6940
#12 0x000000000080854d in dispatch_command (command=COM_QUERY, thd=0x49bda20, packet=0x49c20c1 "INSERT IGNORE INTO testdb.sbtest89910 SELECT * FROM sbtest89", packet_length=60) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1630
#13 0x0000000000806dd7 in do_command (thd=0x49bda20) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_parse.cc:1133
#14 0x00000000007cd8a1 in do_handle_one_connection (thd_arg=0x49bda20) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1557
#15 0x00000000007cd388 in handle_one_connection (arg=0x49bda20) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/sql/sql_connect.cc:1461
#16 0x0000000000b3e1bb in pfs_spawn_thread (arg=0x48cb130) at /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.15/storage/perfschema/pfs.cc:1858
#17 0x00007fa28560d9d1 in start_thread () from /lib64/libpthread.so.0
#18 0x00007fa283d2bb6d in clone () from /lib64/libc.so.6

[Raghavendra D Prabhu (raghavendra-prabhu)]

This is with any statement - REPLACE, DELETE etc.

(gdb) print *(PFS_statement_class *)(state->m_class)
$6 = {
  <PFS_instr_class> = {
    m_type = PFS_CLASS_STATEMENT,
    m_enabled = true,
    m_timed = true,
    m_flags = 0,
    m_event_name_index = 7,
    m_name = "statement/sql/delete", '\000' <repeats 107 times>,
    m_name_length = 20,
    m_timer = 0x1441fe0 <statement_timer>
  }, <No data fields>}

[Philip Stoev (philip-stoev-f)]

I am observing this with multiple tests from the galera MTR test suite. If you run the test suite with debug and performance schema enabled, you will get this assertion.

[Philip Stoev (philip-stoev-f)]

This issue affects both us and Percona, and both RQG and MTR, so I am upping the priority.

[Philip Stoev (philip-stoev-f)]

See https://github.com/codership/mysql-wsrep-features/issues/47

[Philip Stoev (philip-stoev-f)]

Public bug: https://github.com/codership/mysql-wsrep/issues/37

[Krunal Bauskar (krunal-bauskar)]

github issue is fixed so marking this as fixed too.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-992