Crash with debug build on PXC 5.6

[Raghavendra D Prabhu]

Log:

Detecting server version...
2013-09-20 13:11:30: run.sh: Starting server with id=1...
2013-09-20 13:11:30: run.sh: Creating server root directory: /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1
2013-09-20 13:11:30: run.sh: Creating server temporary directory: /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/tmp
2013-09-20 13:11:30: run.sh: Creating server data directory: /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/data
2013-09-20 13:11:30: run.sh: Calling mysql_install_db
Installing MySQL system tables...2013-09-20 13:11:30 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
OK

Filling help tables...2013-09-20 13:11:34 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system

PLEASE REMEMBER TO SET A PASSWORD FOR THE MySQL root USER !
To do so, start the server, then issue the following commands:

  /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin/mysqladmin -u root password 'new-password'
  /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin/mysqladmin -u root -h jhc-new-3-centos6-64 password 'new-password'

Alternatively you can run:

  /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin/mysql_secure_installation

which will also give you the option of removing the test
databases and anonymous user created by default. This is
strongly recommended for production servers.

See the manual for more instructions.

You can start the MySQL daemon with:

  cd . ; /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin/mysqld_safe &

You can test the MySQL daemon with mysql-test-run.pl

  cd mysql-test ; perl mysql-test-run.pl

Please report any problems with the ./bin/mysqlbug script!

The latest information about MySQL is available on the web at

  http://www.mysql.com

Support MySQL by buying support/licenses at http://shop.mysql.com

New default config file was created as /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/my.cnf and
will be used by default by the server when you start it.
You may edit this file to change server settings

WARNING: Default config file /etc/my.cnf exists on the system
This file will be read by default by the MySQL server
If you do not want to use this, either remove it, or use the
--defaults-file argument to mysqld_safe when starting the server

Starting /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld --defaults-file=/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/my.cnf
2013-09-20 13:11:38 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2013-09-20 13:11:39: run.sh: Made 1 attempts to connect to server
2013-09-20 13:11:40: run.sh: Server process PID=26382 died.
2013-09-20 13:11:40: run.sh: Can't start the server. Server log (if exists):
2013-09-20 13:11:40: run.sh: ----------------
2013-09-20 13:11:40: run.sh: Error log for server with id: 1
2013-09-20 13:11:30 26335 [Note] WSREP: Read nil XID from storage engines, skipping position init
2013-09-20 13:11:30 26335 [Note] WSREP: wsrep_load(): loading provider library 'none'
2013-09-20 13:11:30 7fe75bca07e0 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2013-09-20 13:11:30 7fe75bca07e0 InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
2013-09-20 13:11:30 26335 [Note] InnoDB: The InnoDB memory heap is disabled
2013-09-20 13:11:30 26335 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2013-09-20 13:11:30 26335 [Note] InnoDB: Compressed tables use zlib 1.2.3
2013-09-20 13:11:30 26335 [Note] InnoDB: Using Linux native AIO
2013-09-20 13:11:30 26335 [Note] InnoDB: Using CPU crc32 instructions
2013-09-20 13:11:30 26335 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2013-09-20 13:11:30 26335 [Note] InnoDB: Completed initialization of buffer pool
2013-09-20 13:11:30 26335 [Note] InnoDB: The first specified data file ./ibdata1 did not exist: a new database to be created!
2013-09-20 13:11:30 26335 [Note] InnoDB: Setting file ./ibdata1 size to 12 MB
2013-09-20 13:11:30 26335 [Note] InnoDB: Database physically writes the file full: wait...
2013-09-20 13:11:30 26335 [Note] InnoDB: Setting log file ./ib_logfile101 size to 48 MB
2013-09-20 13:11:31 26335 [Note] InnoDB: Setting log file ./ib_logfile1 size to 48 MB
2013-09-20 13:11:31 26335 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2013-09-20 13:11:31 26335 [Warning] InnoDB: New log files created, LSN=45781
2013-09-20 13:11:31 26335 [Note] InnoDB: Doublewrite buffer not found: creating new
2013-09-20 13:11:31 26335 [Note] InnoDB: Doublewrite buffer created
2013-09-20 13:11:31 26335 [Note] InnoDB: 128 rollback segment(s) are active.
2013-09-20 13:11:31 26335 [Warning] InnoDB: Creating foreign key constraint system tables.
2013-09-20 13:11:31 26335 [Note] InnoDB: Foreign key constraint system tables created
2013-09-20 13:11:31 26335 [Note] InnoDB: Creating tablespace and datafile system tables.
2013-09-20 13:11:31 26335 [Note] InnoDB: Tablespace and datafile system tables created.
2013-09-20 13:11:31 26335 [Note] InnoDB: Waiting for purge to start
2013-09-20 13:11:31 26335 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.13-rc60.5 started; log sequence number 0
2013-09-20 13:11:31 26335 [Note] RSA private key file not found: /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/data//private_key.pem. Some authentication plugins will not work.
2013-09-20 13:11:31 26335 [Note] RSA public key file not found: /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/data//public_key.pem. Some authentication plugins will not work.
2013-09-20 13:11:31 26335 [Note] WSREP: Service disconnected.
2013-09-20 13:11:32 26335 [Note] WSREP: Some threads may fail to exit.
2013-09-20 13:11:32 26335 [Note] Binlog end
2013-09-20 13:11:32 26335 [Note] InnoDB: FTS optimize thread exiting.
2013-09-20 13:11:32 26335 [Note] InnoDB: Starting shutdown...
2013-09-20 13:11:34 26335 [Note] InnoDB: Shutdown completed; log sequence number 1625977
2013-09-20 13:11:34 26358 [Note] WSREP: Read nil XID from storage engines, skipping position init
2013-09-20 13:11:34 26358 [Note] WSREP: wsrep_load(): loading provider library 'none'
2013-09-20 13:11:34 7f96cd6ce7e0 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2013-09-20 13:11:34 7f96cd6ce7e0 InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
2013-09-20 13:11:34 26358 [Note] InnoDB: The InnoDB memory heap is disabled
2013-09-20 13:11:34 26358 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2013-09-20 13:11:34 26358 [Note] InnoDB: Compressed tables use zlib 1.2.3
2013-09-20 13:11:34 26358 [Note] InnoDB: Using Linux native AIO
2013-09-20 13:11:34 26358 [Note] InnoDB: Using CPU crc32 instructions
2013-09-20 13:11:34 26358 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2013-09-20 13:11:34 26358 [Note] InnoDB: Completed initialization of buffer pool
2013-09-20 13:11:34 26358 [Note] InnoDB: Highest supported file format is Barracuda.
2013-09-20 13:11:35 26358 [Note] InnoDB: 128 rollback segment(s) are active.
2013-09-20 13:11:35 26358 [Note] InnoDB: Waiting for purge to start
2013-09-20 13:11:35 26358 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.13-rc60.5 started; log sequence number 1625977
2013-09-20 13:11:35 26358 [Note] RSA private key file not found: /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/data//private_key.pem. Some authentication plugins will not work.
2013-09-20 13:11:35 26358 [Note] RSA public key file not found: /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/data//public_key.pem. Some authentication plugins will not work.
2013-09-20 13:11:35 26358 [Note] WSREP: Service disconnected.
2013-09-20 13:11:36 26358 [Note] WSREP: Some threads may fail to exit.
2013-09-20 13:11:36 26358 [Note] Binlog end
2013-09-20 13:11:36 26358 [Note] InnoDB: FTS optimize thread exiting.
2013-09-20 13:11:36 26358 [Note] InnoDB: Starting shutdown...
2013-09-20 13:11:38 26358 [Note] InnoDB: Shutdown completed; log sequence number 1625987
2013-09-20 13:11:38 26382 [Note] Plugin 'FEDERATED' is disabled.
2013-09-20 13:11:38 7fa5fa4197e0 InnoDB: !!!!!!!! UNIV_DEBUG switched on !!!!!!!!!
2013-09-20 13:11:38 7fa5fa4197e0 InnoDB: !!!!!!!! UNIV_SYNC_DEBUG switched on !!!!!!!!!
2013-09-20 13:11:38 26382 [Note] InnoDB: The InnoDB memory heap is disabled
2013-09-20 13:11:38 26382 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2013-09-20 13:11:38 26382 [Note] InnoDB: Compressed tables use zlib 1.2.3
2013-09-20 13:11:38 26382 [Note] InnoDB: Using Linux native AIO
2013-09-20 13:11:38 26382 [Note] InnoDB: Using CPU crc32 instructions
2013-09-20 13:11:38 26382 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2013-09-20 13:11:38 26382 [Note] InnoDB: Completed initialization of buffer pool
2013-09-20 13:11:38 26382 [Note] InnoDB: Highest supported file format is Barracuda.
2013-09-20 13:11:39 26382 [Note] InnoDB: 128 rollback segment(s) are active.
2013-09-20 13:11:39 26382 [Note] InnoDB: Waiting for purge to start
2013-09-20 13:11:39 26382 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.13-rc60.5 started; log sequence number 1625987
2013-09-20 13:11:39 26382 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 2fce591b-21f6-11e3-a051-02163e085477.
2013-09-20 13:11:39 26382 [Note] RSA private key file not found: /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/data//private_key.pem. Some authentication plugins will not work.
2013-09-20 13:11:39 26382 [Note] RSA public key file not found: /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/data//public_key.pem. Some authentication plugins will not work.
2013-09-20 13:11:39 26382 [Note] Server hostname (bind-address): '*'; port: 28793
2013-09-20 13:11:39 26382 [Note] IPv6 is available.
2013-09-20 13:11:39 26382 [Note] - '::' resolves to '::';
2013-09-20 13:11:39 26382 [Note] Server socket created on IP: '::'.
2013-09-20 13:11:39 26382 [Note] Event Scheduler: Loaded 0 events
InnoDB: sync levels should be > 290 but a level is 290
-------------
RW-LATCH INFO
RW-LATCH: 0x7fa5d2cf7830
Locked: thread 140350844934112 file /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.13/storage/innobase/include/trx0sys.ic line 103 X-LOCK
Locked: thread 140350844934112 file /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.13/storage/innobase/include/trx0sys.ic line 103 X-LOCK
InnoDB: sync_thread_levels_g(array, 290) does not hold!
2013-09-20 13:11:39 7fa5fa4197e0 InnoDB: Assertion failure in thread 140350844934112 in file sync0sync.cc line 1174
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
13:11:39 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 69200 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld(my_print_stacktrace+0x35)[0x9ab5f5]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld(handle_fatal_signal+0x4b4)[0x69c704]
/lib64/libpthread.so.0[0x36d360f500]
/lib64/libc.so.6(gsignal+0x35)[0x36d2e328e5]
/lib64/libc.so.6(abort+0x175)[0x36d2e340c5]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld[0xb32a4f]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld(_Z31trx_sys_update_wsrep_checkpointPK5xid_tP5mtr_t+0x8e)[0xb4d87e]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld(_Z29trx_sys_read_wsrep_checkpointP5xid_t+0x184)[0xb4dbf4]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld[0xa105c5]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld[0x5b6a60]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_PP13st_plugin_intPvEijS4_+0x204)[0x74c244]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld(_Z10wsrep_initv+0x58)[0x5b7aa8]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld(_Z18wsrep_init_startupb+0x10)[0x5b8490]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld(_Z11mysqld_mainiPPc+0x17c4)[0x5b1b84]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x36d2e1ecdd]
/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/bin//mysqld[0x5a1505]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
2013-09-20 13:11:40: run.sh: ----------------
2013-09-20 13:11:40: run.sh: ----------------
2013-09-20 13:11:40: run.sh: Innobackupex log for server with id: 1
cat: /mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/data/innobackup*.log: No such file or directory
2013-09-20 13:11:40: run.sh: ----------------
2013-09-20 13:11:40: run.sh: ----------------
2013-09-20 13:11:40: run.sh: Configuration for server with id: 1
[mysqld]
socket=/tmp/mysql.sock.DdlBpb
port=28793
server-id=1
basedir=/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64
datadir=/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/data
tmpdir=/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/tmp
log-error=/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/var1/data/mysqld1.err
pid-file=/mnt/workspace/pxc6-binaries-sst-multi-test/Host/centos6-64/Ttype/debug/5/Percona-XtraDB-Cluster-5.6.13-24.0.433.Linux.x86_64/percona-xtradb-cluster-tests/var/mysqld1.pid

[client]
socket=/tmp/mysql.sock.DdlBpb
user=root
2013-09-20 13:11:40: run.sh: ----------------
2013-09-20 13:11:40: run.sh: Found a leftover mysqld processes with PID 26382, stopping it

[Raghavendra D Prabhu (raghavendra-prabhu)]

This crash seems to be associated with UNIV_SYNC_DEBUG.

PS added UNIV_DEBUG_SYNC back by removing the undef. For more details check: http://bugs.mysql.com/bug.php?id=69617&thanks=3&notify=195

However, the codership tree (and its binaries) still retain it, hence may not see any crash associated with it (with debug builds), for now.

[Raghavendra D Prabhu (raghavendra-prabhu)]

This affects codership tree as well, just that due to UNIV_SYNC_DEBUG being disabled, it is masked.

 === modified file 'Percona-Server/storage/innobase/trx/trx0sys.cc'
 --- Percona-Server/storage/innobase/trx/trx0sys.cc 2013-09-05 13:50:58 +0000
 +++ Percona-Server/storage/innobase/trx/trx0sys.cc 2013-09-30 12:22:41 +0000
 @@ -358,12 +358,14 @@
         mtr_start(&mtr);

         sys_header = trx_sysf_get(&mtr);
 -
 - if ((magic = mach_read_from_4(sys_header + TRX_SYS_WSREP_XID_INFO
 - + TRX_SYS_WSREP_XID_MAGIC_N_FLD))
 - != TRX_SYS_WSREP_XID_MAGIC_N) {
 + magic = mach_read_from_4(sys_header + TRX_SYS_WSREP_XID_INFO + TRX_SYS_WSREP_XID_MAGIC_N_FLD);
 +
 + mtr_commit(&mtr);
 +
 + if (magic != TRX_SYS_WSREP_XID_MAGIC_N) {
                  memset(xid, 0, sizeof(*xid));
                  xid->formatID = -1;
 + mtr_start(&mtr);
                  trx_sys_update_wsrep_checkpoint(xid, &mtr);
                  mtr_commit(&mtr);
                  return;

fixes it, but a better fix will also do.

The problem is, as the crash states, is due to X-lock.

     sys_header = trx_sysf_get(&mtr); is called twice in the code:

a) first in trx_sys_read_wsrep_checkpoint(XID* xid)

b) second in trx_sys_update_wsrep_checkpoint(xid, &mtr); while the first X-lock is still held which causes the crash.

Hence, the fix above works. Tested with SST.

>>./run.sh -c galera55 -d /pxc56 -t t/xb_galera_sst.sh
Detecting server version...
Running against Percona Server 5.6.13-24.0-debug (XtraDB 5.6.13-rc60.5)
Using 'xtrabackup_55' as xtrabackup binary
Using 8 parallel workers

==============================================================================
TEST WORKER RESULT TIME(s) or COMMENT
------------------------------------------------------------------------------
xb_galera_sst w1 [passed] 52
==============================================================================
Spent 52 of 68 seconds executing testcases

SUMMARY: 1 run, 1 successful, 0 skipped, 0 failed

See results/ for detailed output
./run.sh -c galera55 -d /pxc56 -t t/xb_galera_sst.sh 8.03s user 2.92s system 16% cpu 1:07.86 total

[Raghavendra D Prabhu (raghavendra-prabhu)]

=== modified file 'Percona-Server/storage/innobase/trx/trx0sys.cc'
--- Percona-Server/storage/innobase/trx/trx0sys.cc 2013-09-05 13:50:58 +0000
+++ Percona-Server/storage/innobase/trx/trx0sys.cc 2013-09-30 12:49:45 +0000
@@ -358,10 +358,12 @@
  mtr_start(&mtr);

  sys_header = trx_sysf_get(&mtr);
-
- if ((magic = mach_read_from_4(sys_header + TRX_SYS_WSREP_XID_INFO
- + TRX_SYS_WSREP_XID_MAGIC_N_FLD))
- != TRX_SYS_WSREP_XID_MAGIC_N) {
+ magic = mach_read_from_4(sys_header + TRX_SYS_WSREP_XID_INFO + TRX_SYS_WSREP_XID_MAGIC_N_FLD);
+
+
+ if (magic != TRX_SYS_WSREP_XID_MAGIC_N) {
+ mtr_commit(&mtr);
+ mtr_start(&mtr);
                 memset(xid, 0, sizeof(*xid));
                 xid->formatID = -1;
                 trx_sys_update_wsrep_checkpoint(xid, &mtr);

is better (since mtr_commit is avoided twice)

but it is much better to not query sys_header again in trx_sys_update_wsrep_checkpoint

[Raghavendra D Prabhu (raghavendra-prabhu)]

http://paste.wnohang.net/60e9dc?diff is much cleaner though it requires changing signature of trx_sys_update_wsrep_checkpoint

[Teemu Ollakka (teemu-ollakka)]

Duplicate of https://bugs.launchpad.net/codership-mysql/+bug/1248908.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1443