Mysqld crash under BF-BF X lock conflict

[Davide Bozzelli]

Crash is due to a "create table" statement without a primary key.

Log:

BF-BF X lock conflict
RECORD LOCKS space id 1164318 page no 3 n bits 72 index `GEN_CLUST_INDEX` of table `assistenza`.`edig5590` trx id 49E6001 lock_mode X locks rec but not gap
16:19:40 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting

Version: percona-xtradb-cluster-server-5.5 5.5.37-25.10-756.squeeze

I've started seen this crash after last upgrade

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Davide,

Can you provide the create table used for this which caused this crash? Also,
any other supporting information like error log, general log should help.

[Davide Bozzelli (buzzz)]

Today happen again:

F-BF X lock conflict
RECORD LOCKS space id 1173403 page no 4 n bits 256 index `GEN_CLUST_INDEX` of table `assistenza`.`temp_mdem520` trx id 4B4C7D5 lock_mode X locks rec but not gap
08:16:24 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=33554432
read_buffer_size=2097152
max_used_connections=36
max_threads=252
thread_count=33
connection_count=33
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1068175 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f67b232d9c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...

onnection_count=33
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1068175 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f67b232d9c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f6511e56a88 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0x7f67ac3db8be]
/usr/sbin/mysqld(handle_fatal_signal+0x4a3)[0x7f67ac2b0fe3]
/lib/libpthread.so.0(+0xeff0)[0x7f67aba56ff0]
/lib/libc.so.6(gsignal+0x35)[0x7f67a9d101b5]
/lib/libc.so.6(abort+0x180)[0x7f67a9d12fc0]
/usr/sbin/mysqld(+0x68df12)[0x7f67ac511f12]
/usr/sbin/mysqld(+0x69099b)[0x7f67ac51499b]
/usr/sbin/mysqld(+0x6921e9)[0x7f67ac5161e9]
/usr/sbin/mysqld(+0x692491)[0x7f67ac516491]
/usr/sbin/mysqld(+0x5d24b7)[0x7f67ac4564b7]
/usr/sbin/mysqld(+0x5d5864)[0x7f67ac459864]
/usr/sbin/mysqld(+0x5aafe7)[0x7f67ac42efe7]
/usr/sbin/mysqld(+0x5a2e77)[0x7f67ac426e77]
/usr/sbin/mysqld(+0x5aa077)[0x7f67ac42e077]
/usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEPK14Relay_log_info+0x1d9)[0x7f67ac357389]
/usr/sbin/mysqld(_ZN21Update_rows_log_event11do_exec_rowEPK14Relay_log_info+0xa6)[0x7f67ac3578a6]
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x24b)[0x7f67ac35e7db]
/usr/sbin/mysqld(_Z14wsrep_apply_cbPvPKvmjPK14wsrep_trx_meta+0x5dd)[0x7f67ac26dbed]
/usr/lib64/libgalera_smm.so(+0x1a2909)[0x7f67a8e48909]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x273)[0x7f67a8e4afc3]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM11process_trxEPvPNS_9TrxHandleE+0x45)[0x7f67a8e4b915]
/usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x3d4)[0x7f67a8e27464]
/usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource...

[Davide Bozzelli (buzzz)]

this is the table that cause the crash:

CREATE TABLE IF NOT EXISTS `temp_mdem520` (
  `id` int(255) NOT NULL DEFAULT '0',
  `data_inizio` date DEFAULT NULL,
  `data_scadenza` date DEFAULT NULL,
  `id_cliente` int(255) DEFAULT NULL,
  `beneficiario` varchar(100) DEFAULT NULL,
  `commerciale` varchar(100) DEFAULT NULL,
  `valore` decimal(12,2) DEFAULT NULL,
  `costo` decimal(12,2) DEFAULT NULL,
  `azienda` varchar(100) DEFAULT NULL,
  `tipo` int(255) DEFAULT NULL,
  `ragsoc` varchar(255) DEFAULT NULL,
  `tipo_fatt` varchar(255) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

[Aleksey Sokolov (absokolov2010)]

Hi to all.
We also have a similar problem.

MySQL Version: Percona XtraDB Cluster (GPL), Release 25.5, Revision 759, wsrep_25.5.r4061
Log:

BF-BF X lock conflict
RECORD LOCKS space id 332 page no 492 n bits 440 index `PRIMARY` of table `billing`.`menu_operator_items` trx id 8627444 lock_mode X locks rec but not gap
22:11:37 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=10
max_threads=153
thread_count=9
connection_count=9
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 192102 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f6324000990
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f63682f1d98 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x902645]
/usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x6801e4]
/lib64/libpthread.so.0(+0xf710)[0x7f667d8d5710]
/lib64/libc.so.6(gsignal+0x35)[0x7f667bd1e925]
/lib64/libc.so.6(abort+0x175)[0x7f667bd20105]
/usr/sbin/mysqld[0x9950db]
/usr/sbin/mysqld[0x99973f]
/usr/sbin/mysqld[0x999ae3]
/usr/sbin/mysqld[0xa0b4ff]
/usr/sbin/mysqld[0xa11679]
/usr/sbin/mysqld[0x967a09]
/usr/sbin/mysqld(_ZN7handler10ha_rnd_posEPhS0_+0xac)[0x5c092c]
/usr/sbin/mysqld(_ZN14Rows_log_event24do_index_scan_and_updateEPK14Relay_log_info+0x1bf)[0x89facf]
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0xd8f)[0x8a59df]
/usr/sbin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x68)[0x8a6c38]
/usr/sbin/mysqld(_Z14wsrep_apply_cbPvPKvmjPK14wsrep_trx_meta+0x5ce)[0x5bd38e]
/usr/lib64/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmjPK14wsrep_trx_metaERS6_+0xb1)[0x7f665d69e291]
/usr/lib64/libgalera_smm.so(+0x1aaf65)[0x7f665d6d5f65]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x283)[0x7f665d6d6dd3]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM11process_trxEPvPNS_9TrxHandleE+0x45)[0x7f665d6d76c5]
/usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x2c9)[0x7f665d6b4319]
/usr/lib64/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x63)[0x7f665d6b47f3]
/usr/lib64/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x93)[0x7f665d6d33c3]
/usr/lib64/libgalera_smm.so(galera_recv+0x23)[0x7f665d6e8753]
/usr/sbin/mysqld[0x5be11f]
/usr/sbin/mysqld(start_wsrep_THD+0x3fe)[0x5ae42e]
/lib64/libpthread.so.0(+0x79d1)[0x7f667d8cd9d1]
/lib64/libc.so.6(clone+0x6d)[0x7f667bdd4b6d]

Trying to ge...

[Ville Ojamo (ville-ojamo)]

Same here, one node was updated to Percona-XtraDB-Cluster-server-55-5.5.37-25.10.756.el6.x86_64 and the node cannot join the cluster anymore.
The error mentions a table with no primary key that exists on the cluster.

[Davide Bozzelli (buzzz)]

Any news on this bug ?

[Raghavendra D Prabhu (raghavendra-prabhu)]

I have marked this a duplicate of another bug in progress relating to this.

Also, I tested with latest code and I am unable to reproduce this.

Can you test with CentOS RPMs here http://jenkins.percona.com/job/percona-xtradb-cluster-5.6-redhat-binary/label_exp=centos6-64/

or the binaries here http://jenkins.percona.com/job/percona-xtradb-cluster-5.6-binary/label_exp=centos5-64/lastSuccessfulBuild/artifact/21/Percona-XtraDB-Cluster-5.6.19-rel67.0-25.6.796.Linux.x86_64.tar.gz

[David Pappe (dpappe)]

Hi, all
I'm having the same issue with 5.6.15-56 Percona XtraDB Cluster. Has anyone resolved this problem?
Please, let me know about this.

2014-07-13 03:54:14 2397 [Note] WSREP: inited wsrep sidno 1
2014-07-13 03:54:14 2397 [Note] WSREP: SST received: 42237e6d-f1bf-11e3-8e7e-fe842680491f:90740417
2014-07-13 03:54:14 2397 [Note] /usr/sbin/mysqld: ready for connections.
Version: '5.6.15-56' socket: '/var/lib/mysql/mysql.sock' port: 3306 Percona XtraDB Cluster (GPL), Release 25.5, Revision 759, wsrep_25.5.r4061
BF-BF X lock conflict
RECORD LOCKS space id 46674 page no 367 n bits 160 index `GEN_CLUST_INDEX` of table `backoffice_prev`.`autopayMembers` trx id 5794109016 lock_mode X locks rec but not gap
09:54:14 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=83886080
read_buffer_size=1310720
max_used_connections=0
max_threads=5002
thread_count=9
connection_count=9
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 7839967 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation. less…
Bugs : Percona XtraDB Cluster bugs.launchpad.net
Percona XtraDB Cluster is a High Availability and Scalability Solution for MySQL Users and is Based on Percona Server 5.5/5.6 Percona XtraDB Cluster provides: - Synchronous replication.

[jack liu (freejack)]

Another one bites the dust. Hopefully this gets addressed soon. Let me know if you need additional info.

BF-BF X lock conflict
RECORD LOCKS space id 293 page no 5 n bits 216 index `GEN_CLUST_INDEX` of table `app`.`users` trx id 2E0784EE lock_mode X locks rec but not gap
140814 11:00:03 [ERROR] mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 5.5.38-MariaDB-wsrep
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=9
max_threads=302
thread_count=4
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 793726 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x2aabef012000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x42c0e098 thread_stack 0x48000
/usr/sbin/mysqld(my_print_stacktrace+0x2e)[0xacb8be]
/usr/sbin/mysqld(handle_fatal_signal+0x44c)[0x6f9dfc]
/lib64/libpthread.so.0[0x3edec0eb10]
/lib64/libc.so.6(gsignal+0x35)[0x3ede430265]
/lib64/libc.so.6(abort+0x110)[0x3ede431d10]
/usr/sbin/mysqld[0x9fade1]
/usr/sbin/mysqld[0x9fcb77]
/usr/sbin/mysqld[0xa005fa]
/usr/sbin/mysqld[0x938bd9]
/usr/sbin/mysqld[0x902231]
/usr/sbin/mysqld(_ZN14Rows_log_event8find_rowEPK14Relay_log_info+0x5c8)[0x7ce588]
/usr/sbin/mysqld(_ZN21Update_rows_log_event11do_exec_rowEPK14Relay_log_info+0x7e)[0x7cf86e]
/usr/sbin/mysqld(_ZN14Rows_log_event14do_apply_eventEPK14Relay_log_info+0x25b)[0x7d49db]
/usr/sbin/mysqld[0x6a155f]
/usr/sbin/mysqld(_Z14wsrep_apply_cbPvPKvmjPK14wsrep_trx_meta+0xc0)[0x6a1ad0]
/usr/lib64/galera/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmjPK14wsrep_trx_metaERS6_+0xb1)[0x2aaaab78d0d1]
/usr/lib64/galera/libgalera_smm.so[0x2aaaab7c4ab5]
/usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x283)[0x2aaaab7c5923]
/usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM11process_trxEPvPNS_9TrxHandleE+0x45)[0x2aaaab7c6215]
/usr/lib64/galera/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x2c9)[0x2aaaab7a3159]
/usr/lib64/galera/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x63)[0x2aaaab7a3633]
/usr/lib64/galera/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x93)[0x2aaaab7c1f23]
/usr/lib64/galera/libgalera_smm.so(galera_recv+0x23)[0x2aaaab7d70d3]
/usr/sbin/mysqld[0x6a24dc]
/usr/sbin/mysqld(start_wsrep_THD+0x46c)[0x4fdf0c]
/lib64/libpthread.so.0[0x3edec0673d]
/lib64/libc.so.6(clone+0x6d)[0x3ede4d44bd]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x0): is an invalid pointer
Connection ID (t...

[Przemek (pmalkowski)]

Indeed updating table without PK nor any other index was crashing PXC node in version 5.6.15, when wsrep_slave_threads>1. It is easy to reproduce - remove keys from sbtest table and run multi-threaded OLTP sysbench test. It will crash with:

BF-BF X lock conflict
RECORD LOCKS space id 485 page no 74 n bits 144 index `GEN_CLUST_INDEX` of table `db1`.`sbtest1` trx id 95033 lock_mode X locks rec but not gap
...
and very similar stack trace.

Since PXC 5.6.19, the abort was removed, so instead, we will see error log flooded with:

"2016-06-02 07:51:48 17146 [Note] WSREP: BF conflict, order: 41008 41009

BF-BF X lock conflict,mode: 1027 supremum: 0
conflicts states: my 0 locked 0
RECORD LOCKS space id 485 page no 82 n bits 144 index `GEN_CLUST_INDEX` of table `db1`.`sbtest1` trx id 93362 lock_mode X locks rec but
not gap
BF-BF lock conflict, locking: 1
RECORD LOCKS space id 485 page no 82 n bits 144 index `GEN_CLUST_INDEX` of table `db1`.`sbtest1` trx id 93363 lock_mode X locks rec but
not gap"

but nodes won't crash.

Still, using non-indexed tables in PXC/Galera is not supported and each table should have primary key.