Failing assertion: table->n_ref_count > 0 || !table->can_be_evicted in file lock0lock.cc line 4083

Bug #1265070 reported by Roel Van de Paar on 2013-12-30
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
Low
Alexey Kopytov
2.1
Won't Fix
Undecided
Unassigned
2.2
Fix Released
Low
Alexey Kopytov

Bug Description

==== Prepare 1 log:
[01] Rebuilding 59 index(es).
2013-12-31 05:10:56 7f3612bfd700 InnoDB: Assertion failure in thread 139870219523840 in file lock0lock.cc line 4083
InnoDB: Failing assertion: table->n_ref_count > 0 || !table->can_be_evicted
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.

==== Command (note encrypt key may not be valid):
Prepare command: /ssd/xtrabackup-dbg-281213-1813/bin/xtrabackup --core --parallel=17 --compact --compress-threads=31 --rebuild_indexes --tables=test.* --throttle=0 --use-memory=67108863 --encryption=AES128 --encrypt-key=DA66E327C5DB8047CD8EC567984CB16C77DC6697F5C03933 --encrypt-threads=5 --encrypt-chunk-size=67108860 -
-prepare --target-dir=/sde//918447.xb/current1_1/master-data_xtrabackup_11325

==== Versions:
XB @ 2.2 4916 <email address hidden>
PS @ Percona-Server-5.6.14-rel62.0-508-debug.Linux.x86_64
randgen-xb @ 2 <email address hidden>
   with "# if ($result > 0) {" combinations.pl hack for all results

==== Stack:
(gdb) bt
#0 0x0000003168632925 in raise () from /lib64/libc.so.6
#1 0x0000003168634105 in abort () from /lib64/libc.so.6
#2 0x0000000000551c17 in lock_table_create (table=0x7f3600009698, type_mode=3, trx=0x7f3600000938) at /bzr/2.2_dbg/storage/innobase/lock/lock0lock.cc:4083
#3 0x00000000005528bb in lock_table (flags=0, table=0x7f3600009698, mode=LOCK_X, thr=0x7f3600008458) at /bzr/2.2_dbg/storage/innobase/lock/lock0lock.cc:4422
#4 0x000000000050c974 in row_merge_lock_table (trx=0x7f3600000938, table=0x7f3600009698, mode=LOCK_X) at /bzr/2.2_dbg/storage/innobase/row/row0merge.cc:2449
#5 0x000000000045a8bf in xb_rebuild_indexes_for_table (table=0x7f3600009698, trx=0x7f3600000938, thread_n=1)
    at /bzr/2.2_dbg/storage/innobase/xtrabackup/src/compact.cc:825
#6 0x000000000045ae18 in xb_rebuild_indexes_thread_func (arg=0x1dfe618) at /bzr/2.2_dbg/storage/innobase/xtrabackup/src/compact.cc:914
#7 0x0000003168e079d1 in start_thread () from /lib64/libpthread.so.0
#8 0x00000031686e8b6d in clone () from /lib64/libc.so.6

Related branches

Roel Van de Paar (roel11) wrote :
Roel Van de Paar (roel11) wrote :
Roel Van de Paar (roel11) wrote :
Roel Van de Paar (roel11) wrote :

(gdb) f 2
#2 0x0000000000551c17 in lock_table_create (table=0x7f3600009698, type_mode=3, trx=0x7f3600000938) at /bzr/2.2_dbg/storage/innobase/lock/lock0lock.cc:4083
(gdb) p table->n_ref_count
$1 = 0
(gdb) p !table->can_be_evicted
$2 = false

Roel Van de Paar (roel11) wrote :
description: updated
description: updated
Alexey Kopytov (akopytov) wrote :

A debug only issue with xtrabackup_56 in 2.1 (or the single binary in 2.2).

In xb_compact_rebuild_indexes() we should either call a higher-level function instead of dict_table_get_low(), or manipulate table->n_ref_count directly to avoid the assertion. I prefer the latter.

tags: added: low-hanging-fruit
removed: qa xb
David Bennett (dbpercona) wrote :
Download full text (5.1 KiB)

This same issue is causing intermittent PXC SST failure when the [xtrabackup] compact option is set.

==== Centos 5 x86_64

==== PXC

  jenkins build Percona-XtraDB-Cluster-5.6.20-rel68.0-25.7.886.Linux.x86_64.tar.gz

==== xtrabackup 2.2 (revno: 5022)

  -DCMAKE_CXX_FLAGS=-m64
  -DCMAKE_C_FLAGS=-m64
  -DWITH_DEBUG=1

==== MTR test

  percona-xtradb-cluster-tests/t/xb_galera_sst_advanced-v2.sh

==== percona-xtradb-cluster-tests/conf/conf2.cnf-node1 (bug occurs with any 'compat' conf)

# compress+compact + stream-compress
# v2only
[xtrabackup]
compress
compact
compress-threads=4
rebuild-threads=4

[sst]
compressor="gzip"
streamfmt=xbstream
transferfmt=socat

==== MTR log

...
2014-09-21 08:59:11 22282 [Note] WSREP: Requesting state transfer: success, donor: 0
WSREP_SST: [INFO] Proceeding with SST (20140921 08:59:12.250)
WSREP_SST: [INFO] Cleaning the existing datadir (20140921 08:59:12.252)
removed `/home/dbennett/work/1/Percona-XtraDB-Cluster-5.6.20-rel68.0-25.7.886.Linux.x86_64/percona-xtradb-cluster-tests/var/w1/var901/data/gvwstate.dat'
WSREP_SST: [INFO] Evaluating socat -u TCP-LISTEN:23038,reuseaddr stdio | gzip -dc | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20140921 08:59:12.263)
2014-09-21 08:59:13 22282 [Note] WSREP: (144e68c9, 'tcp://127.0.0.1:31049') turning message relay requesting off
2014-09-21 08:59:28 22282 [Note] WSREP: 0.0 (75fde48ae41e): State transfer to 1.0 (75fde48ae41e) complete.
2014-09-21 08:59:28 22282 [Note] WSREP: Member 0.0 (75fde48ae41e) synced with group.
WSREP_SST: [INFO] Index compaction detected (20140921 08:59:41.253)
WSREP_SST: [INFO] Rebuilding during prepare with 4 threads (20140921 08:59:41.261)
WSREP_SST: [INFO] Compressed qpress files found (20140921 08:59:41.266)
WSREP_SST: [INFO] Decompression with 12 threads (20140921 08:59:41.269)
WSREP_SST: [INFO] Evaluating find /home/dbennett/work/1/Percona-XtraDB-Cluster-5.6.20-rel68.0-25.7.886.Linux.x86_64/percona-xtradb-cluster-tests/var/w1/var901/data/ -type f -name '*.qp' -printf '%p\n%h\n' | xargs -n 2 qpress -T12d (2014
0921 08:59:41.272)
WSREP_SST: [INFO] Removing qpress files after decompression (20140921 08:59:41.315)
WSREP_SST: [INFO] Preparing the backup at /home/dbennett/work/1/Percona-XtraDB-Cluster-5.6.20-rel68.0-25.7.886.Linux.x86_64/percona-xtradb-cluster-tests/var/w1/var901/data/ (20140921 08:59:41.318)
WSREP_SST: [INFO] Evaluating innobackupex --no-version-check --apply-log $rebuildcmd ${DATA} &>/home/dbennett/logs/innobackup.prepare.14092108591411304351.log (20140921 08:59:41.320)
WSREP_SST: [ERROR] Cleanup after exit with status:1 (20140921 08:59:45.373)
WSREP_SST: [INFO] Removing the sst_in_progress file (20140921 08:59:45.375)
2014-09-21 08:59:45 22282 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '127.0.0.1:23038' --auth 'root:password' --datadir '/home/dbennett/work/1/Percona-XtraDB-Cluster-5.6.20-rel68.0-25.7.886.Linux.x86_64/percona-xtradb-cluster-tests/var/w1/var901/data/' --defaults-file '/home/dbennett/work/1/Percona-XtraDB-Cluster-5.6.20-rel68.0-25.7.886.Linux.x86_64/percona-xtradb-cluster-tests/var/w1/var901/my.cnf' --parent '22282' '' : 1 (Operation not permitted)
2014-09-21 08:59:4...

Read more...

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-904

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers