InnoDB: Failing assertion: sym_node->table != NULL

Bug #1257266 reported by Ovais Tariq
36
This bug affects 7 people
Affects Status Importance Assigned to Milestone
MySQL Server
Unknown
Unknown
Percona Server moved to https://jira.percona.com/projects/PS
Invalid
High
Unassigned
5.6
Invalid
High
Unassigned

Bug Description

After upgrading to Percona Server 5.6.14 from 5.5.24, Percona Server keeps on crashing and fails to start.

Relevant lines from the error log:
2013-12-03 04:03:36 7f3a711b8720 InnoDB: Assertion failure in thread 139888982460192 in file pars0pars.cc line 865
InnoDB: Failing assertion: sym_node->table != NULL
...
12:03:36 UTC - mysqld got signal 6 ;
...
usr/sbin/mysqld(my_print_stacktrace+0x35)[0x91c605]
/usr/sbin/mysqld(handle_fatal_signal+0x4c4)[0x683134]
/lib/libpthread.so.0(+0xf8f0)[0x7f3a70d9d8f0]
/lib/libc.so.6(gsignal+0x35)[0x7f3a6f460b65]
/lib/libc.so.6(abort+0x180)[0x7f3a6f4646b0]
/usr/sbin/mysqld[0x99f599]
/usr/sbin/mysqld[0x9a1b7a]
/usr/sbin/mysqld(_Z7yyparsev+0x994)[0xac5dc4]
/usr/sbin/mysqld[0x99ff82]
/usr/sbin/mysqld[0x9a5ba6]
/usr/sbin/mysqld[0x9cdd90]
/usr/sbin/mysqld[0x9ceffe]
/usr/sbin/mysqld[0x97596d]
/usr/sbin/mysqld[0x9f743d]
/usr/sbin/mysqld[0x933cd4]
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x5c1e18]
/usr/sbin/mysqld[0x718061]
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0xb0e)[0x71c7ce]
/usr/sbin/mysqld[0x5b4ab2]
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x455)[0x5b9b15]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7f3a6f44bc8d]
/usr/sbin/mysqld[0x5aaf49]

The crash is initiated after a call to row_mysql_drop_temp_tables() that tries to drop probably a temporary table whose tablespace does not exist "mysql/#sql2144_47cae_53". However, in exactly the same case Percona Server 5.5 does not raise any assertion.

Percona Server 5.6 should be more lenient in such cases.

I will be attaching the GDB backtraces separately

Revision history for this message
Ovais Tariq (ovais-tariq) wrote :
Revision history for this message
Ovais Tariq (ovais-tariq) wrote :
Revision history for this message
George Ormond Lorch III (gl-az) wrote :

I also see this no when testing XB 2.2 single binary against Percona Server 5.1 during prepare phase.

Revision history for this message
George Ormond Lorch III (gl-az) wrote :

Here is the same/similar thread apply all bt from the single binary 2.2 xtrabackup. This is preparing a backup taken from upstream 5.5.

Revision history for this message
Ovais Tariq (ovais-tariq) wrote :

@George,
So this affects 5.5 as well or was the backup being prepared for 5.6 server?

Revision history for this message
George Ormond Lorch III (gl-az) wrote :

Ovais, the source/target server really isn't of consequence here. This occurred during the XB 2.2 prepare phase which uses the mini-embedded InnoDB code compiled into xtrabackup which is now supposed to be version agnostic. I _believe_ this mini-InnoDB code is based on the PS 5.6 XtraDB code but I am not certain at all about that. The only real useful data point that I had to offer here is that I am able to easily reproduce the same assertion with a very similar call stack by simply preparing a backup taken from a server under RQG load. From the call stacks, a quick guess is that InnoDB crash recovery now has some issue if a crash occurs at some sensitive point during a DDL (DROP TABLE specifically) operation.

Revision history for this message
Valerii Kravchuk (valerii-kravchuk) wrote :

See also http://bugs.mysql.com/bug.php?id=68987 (does not mention xtrabackup or Precona Server at all)

Other reports of the same assertion are related to InnoDB table with fulltext index (see http://bugs.mysql.com/bug.php?id=71649)

tags: added: upstream
tags: added: i43717
tags: added: i47197
Revision history for this message
markus_albe (markus-albe) wrote :

For 47197 it seems it could be related to open temporary tables on the slave at the time of backup (was running without --safe-slave-backup).

Revision history for this message
Peiran Song (peiran-song) wrote :

Another customer (#48390) hit the same assertion failure during the prepare step of xtrabackup restore. It is reproducible on a dev server while backups from production server of the same server version haven't encountered this problem.

Source server : PS 5.5.36-34.2-648.precise-log (Ubuntu)
Xtrabackup on the source server: version 2.2.5 based on MySQL server 5.6.21 Linux (x86_64)
Xtrabackup on the destination server: version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64)

I didn't reproduce the error when tested with backups taken while DROP TABLE was running on small tables using the same server and xtrabackup version.

tags: added: i48390
Revision history for this message
Roel Van de Paar (roel11) wrote :

As far as "InnoDB: Failing assertion: sym_node->table != NULL" is concerned, see also bug 1383062 and bug 1368846

Revision history for this message
Alexey Kopytov (akopytov) wrote :

See also PXB bug #1399471 (which is likely a duplicate of bug #1383062).

Kenny Gryp (gryp)
tags: added: i45163
Revision history for this message
George Ormond Lorch III (gl-az) wrote :

Can't reproduce on any current release or not enough info to reproduce.

Revision history for this message
Steve Wechsler (steven-wechsler) wrote :
Download full text (9.6 KiB)

Just received this error with:

xtrabackup version 2.2.9 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
MySQL version 5.5.27-percona-log (both source and target)

Backup command was:

$ innobackupex --socket=/opt/mysql/instance/live-arccms-01/admin/var/mysql.sock --user=dba_admin --password=mypass --defaults-file=/opt/mysql/instance/live-arccms-01/admin/conf/my.cnf --no-timestamp --stream=tar /opt/dba/local

Apply log command was:

$ innobackupex --apply-log /opt/mysql/instance/rep01-arccms-01/db/data

Log follows:

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150423 20:40:51 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".

150423 20:40:51 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/opt/mysql/instance/rep01-arccms-01/db/data/backup-my.cnf"
-defaults-group="mysqld" --prepare --target-dir=/opt/mysql/instance/rep01-arccms-01/db/data

xtrabackup version 2.2.9 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: cd to /opt/mysql/instance/rep01-arccms-01/db/data
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=45572571136, start_lsn=(188057949968766)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata_01.idb:1G:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 45572571136
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata_01.idb:1G:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 45572571136
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 100.0M
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 188057949968766
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages
InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 188057955211264 (0%)

[...]

In...

Read more...

Revision history for this message
mig5 (mig5) wrote :

I reproduce this in 5.6.38-83.0-1.jessie by simply altering a table (adding a column)

ALTER TABLE `sales_order_grid` ADD COLUMN `bold_order_comment` text NULL COMMENT 'Bold Order Comment'...

25 seconds into altering this table (it has lots of data) suddenly MySQL crashes with:

2018-01-09 20:28:26 7f1b97fff700 InnoDB: Assertion failure in thread 139756490979072 in file pars0pars.cc line 865
InnoDB: Failing assertion: sym_node->table != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
04:28:26 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=16777216
read_buffer_size=131072
max_used_connections=7
max_threads=153
thread_count=6
connection_count=6
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 77253 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x80000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8c5c6c]
/usr/sbin/mysqld(handle_fatal_signal+0x469)[0x64c599]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f1bfe0a9890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f1bfc02f067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f1bfc030448]
/usr/sbin/mysqld[0x9ef737]
/usr/sbin/mysqld(_Z7yyparsev+0x131b)[0xb2cecb]
/usr/sbin/mysqld[0x9f0dff]
/usr/sbin/mysqld[0xb29653]
/usr/sbin/mysqld[0xb154a2]
/usr/sbin/mysqld[0xb16106]
/usr/sbin/mysqld(_Z23fts_optimize_sync_tablem+0x45)[0xb21765]
/usr/sbin/mysqld[0xb21ba8]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f1bfe0a2064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f1bfc0e262d]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-745

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.