Assertion failure in DICT_TF2_TEMPORARY

[Jai Gupta]

Percona-XtraDB-Cluster-galera-3-3.7-1.3254.rhel6.x86_64
Percona-XtraDB-Cluster-server-56-5.6.20-25.7.888.el6.x86_64
Percona-XtraDB-Cluster-galera-3-debuginfo-3.7-1.3254.rhel6.x86_64
Percona-XtraDB-Cluster-client-56-5.6.20-25.7.888.el6.x86_64
Percona-XtraDB-Cluster-56-debuginfo-5.6.20-25.7.888.el6.x86_64
percona-nagios-plugins-1.1.4-1.noarch
Percona-XtraDB-Cluster-shared-56-5.6.20-25.7.888.el6.x86_64
percona-release-0.0-1.x86_64
percona-xtrabackup-2.2.3-4982.el6.x86_64
Percona-XtraDB-Cluster-test-56-5.6.20-25.7.888.el6.x86_64
Percona-XtraDB-Cluster-garbd-3-3.7-1.3254.rhel6.x86_64
Percona-XtraDB-Cluster-full-56-5.6.20-25.7.888.el6.x86_64

2014-09-22 00:15:47 7f285a4f3700 InnoDB: Assertion failure in thread 139811290560256 in file row0mysql.cc line 3383
InnoDB: Failing assertion: !DICT_TF2_FLAG_IS_SET(table, DICT_TF2_TEMPORARY)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
05:15:47 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=25165824
read_buffer_size=131072
max_used_connections=182
max_threads=202
thread_count=52
connection_count=3
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 105202 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x137f96010
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f285a4f2d38 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x8f5835]
/usr/sbin/mysqld(handle_fatal_signal+0x4b4)[0x664384]
/lib64/libpthread.so.0(+0xf710)[0x7f5fad3a8710]
/lib64/libc.so.6(gsignal+0x35)[0x7f5fab7f1635]
/lib64/libc.so.6(abort+0x175)[0x7f5fab7f2e15]
/usr/sbin/mysqld[0x9ac7bb]
/usr/sbin/mysqld[0x91fdfe]
/usr/sbin/mysqld(_ZN22Sql_cmd_truncate_table16handler_truncateEP3THDP10TABLE_LISTb+0xe0)[0x877cc0]
/usr/sbin/mysqld(_ZN22Sql_cmd_truncate_table14truncate_tableEP3THDP10TABLE_LIST+0x363)[0x8784e3]
/usr/sbin/mysqld(_ZN22Sql_cmd_truncate_table7executeEP3THD+0x5e)[0x87858e]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x138e)[0x6e990e]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x658)[0x6eedc8]
/usr/sbin/mysqld[0x6eef21]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1362)[0x6f0a92]
/usr/sbin/mysqld(_Z10do_commandP3THD+0x20a)[0x6f261a]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x17f)[0x6bb06f]
/usr/sbin/mysqld(handle_one_connection+0x47)[0x6bb247]
/usr/sbin/mysqld(pfs_spawn_thread+0x12a)[0xaee54a]
/lib64/libpthread.so.0(+0x79d1)[0x7f5fad3a09d1]
/lib64/libc.so.6(clone+0x6d)[0x7f5fab8a786d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (7f260c1fea90): is an invalid pointer
Connection ID (thread ID): 5046789
Status: NOT_KILLED

You may download the Percona XtraDB Cluster operations manual by visiting
http://www.percona.com/software/percona-xtradb-cluster/. You may find information
in the manual which will help you identify the cause of the crash.
140922 00:15:48 mysqld_safe Number of processes running now: 0
140922 00:15:48 mysqld_safe WSREP: not restarting wsrep node automatically
140922 00:15:48 mysqld_safe mysqld from pid file /var/lib/mysql/xxxx.pid ended

[Jai Gupta (jai-g)]

After starting node.

2014-09-22 00:40:51 11677 [Note] InnoDB: Using Linux native AIO
2014-09-22 00:40:51 11677 [Note] InnoDB: Using CPU crc32 instructions
2014-09-22 00:40:51 11677 [Note] InnoDB: Initializing buffer pool, size = 180.0G
2014-09-22 00:40:52 11677 [Note] WSREP: (e60766bf, 'tcp://0.0.0.0:4567') turning message relay requesting off
2014-09-22 00:41:04 11677 [Note] InnoDB: Completed initialization of buffer pool
2014-09-22 00:41:06 11677 [Note] InnoDB: Highest supported file format is Barracuda.
2014-09-22 00:41:06 11677 [Note] InnoDB: Log scan progressed past the checkpoint lsn 395166992656
2014-09-22 00:41:06 11677 [Note] InnoDB: Database was not shutdown normally!
2014-09-22 00:41:06 11677 [Note] InnoDB: Starting crash recovery.
2014-09-22 00:41:06 11677 [Note] InnoDB: Reading tablespace information from the .ibd files...
2014-09-22 01:13:14 11677 [Note] InnoDB: Restoring possible half-written data pages
2014-09-22 01:13:14 11677 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 395168714222
2014-09-22 01:13:16 11677 [Note] InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
2014-09-22 01:14:10 11677 [Note] InnoDB: 128 rollback segment(s) are active.
2014-09-22 01:14:10 11677 [Note] InnoDB: Waiting for purge to start
2014-09-22 01:14:10 11677 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.20-68.0 started; log sequence number 395168714222
2014-09-22 01:14:10 7ed4c2197700 InnoDB: Loading buffer pool(s) from .//ib_buffer_pool
2014-09-22 01:14:10 11677 [Note] RSA private key file not found: /var/lib/mysql//private_key.pem. Some authentication plugins will not work.
2014-09-22 01:14:10 11677 [Note] RSA public key file not found: /var/lib/mysql//public_key.pem. Some authentication plugins will not work.
2014-09-22 01:14:10 11677 [Note] Server hostname (bind-address): 'x.x.x.x'; port: 3306
2014-09-22 01:14:10 11677 [Note] - 'x.x.x.x' resolves to 'x.x.x.x';
2014-09-22 01:14:10 11677 [Note] Server socket created on IP: 'x.x.x.x'.
2014-09-22 01:14:10 7f0c114347e0 InnoDB: Error: table `tmp`.`#sql7529_4d0205_8` does not exist in the InnoDB internal
InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-troubleshooting.html
2014-09-22 01:14:10 7f0c114347e0 InnoDB: Error: table `tmp`.`#sql7529_4d0205_3` does not exist in the InnoDB internal
InnoDB: data dictionary though MySQL is trying to drop it.
InnoDB: Have you copied the .frm file of the table to the
InnoDB: MySQL database directory from another database?
InnoDB: You can look for further help from
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/innodb-t...

[Miguel Angel Nieto (miguelangelnieto)]

Hi.

Have you recently upgrade from 5.5 to 5.6?

There is a similar upstream bug reported but without feedback from Oracle:

http://bugs.mysql.com/bug.php?id=72080

Do you usually truncate temporary tables?

[Jai Gupta (jai-g)]

Yes, we did upgraded from 5.5 to 5.6 but that was about 4 months ago.

Yes, our application does use truncate tables sometimes during version upgrade.

[Miguel Angel Nieto (miguelangelnieto)]

I have associated the bug with the upstream one http://bugs.mysql.com/bug.php?id=72080. Confirmed.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Adding percona-server to bug since it is present in upstream.

[Valerii Kravchuk (valerii-kravchuk)]

Miguel,

As you confirmed this for PXC 5.5 and 5.6, I wonder if you have any ideas about the repeatable test case for Percona Server or upstream MySQL? Upstream bug is still Open and it seems test case is not available.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Valerii, @Laurynas,

What is the status on this issue. Has this been fixed in PS. (or Oracle MySQL - seems to be non-repeatable there).

[Laurynas Biveinis (laurynas-biveinis)]

Unable to comment on the server status as a test case is not provided at the moment

[Alexey Kopytov (akopytov)]

According to comments in the upstream bug, the problem has been fixed in MySQL in the upcoming 5.6.24, 5.7.7, 5.8.0 releases.

[Laurynas Biveinis (laurynas-biveinis)]

https://github.com/percona/percona-server/pull/60

[Raghavendra D Prabhu (raghavendra-prabhu)]

Should be fixed in 5.6.24 PXC release as well.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1112

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-820