Innodb: Failing assertion: slot != NULL from os0file.cc line 5126

[Ramesh Sivaraman]

========================= Error log:
2014-06-07 15:38:06 7fa66c3fe700 InnoDB: Assertion failure in thread 140352757425920 in file os0file.cc line 5126
InnoDB: Failing assertion: slot != NULL
InnoDB: We intentionally generate a memory trap.
---
--
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
Writing a core file

========================= gdb :
(gdb) +thread apply all bt

Thread 116 (Thread 15420):
+bt
#0 0x00007fa69693843c in ?? ()
Cannot access memory at address 0x7fa5cdf36d00
--
--
+bt
#0 0x00007fa69693969c in ?? ()
Cannot access memory at address 0x7fa66c3fd638
(gdb) +set logging off

========================= Run details:
ramesh@ramesh-550P5C-550P7C:~/bug/BUNDLE_327$ cat cmd327
if [ -r /usr/lib64/libjemalloc.so.1 ]; then export LD_PRELOAD=/usr/lib64/libjemalloc.so.1
elif [ -r /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 ]; then export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
else echo 'Error: jemalloc not found, please install it first'; exit 1; fi
ps -ef | grep 'rundir1_327' | grep -v grep | awk '{print $2}' | xargs sudo kill -9 > /dev/null 2>&1
rm -Rf /data/bench/qa56dbg/33/rundir1_327
rm -Rf /data/bench/qa56dbg/33/rundir1_327_epoch
mkdir /data/bench/qa56dbg/33/rundir1_327
mkdir /data/bench/qa56dbg/33/rundir1_327_epoch
cd /data/bench/qa56dbg/randgen
bash -c "set -o pipefail; perl /data/bench/qa56dbg/randgen/runall.pl --queries=100000000 --seed=32219 --duration=300 --querytimeout=60 --short_column_names --reporter=Shutdown,Backtrace,QueryTimeout,ErrorLog,ErrorLogAlarm --mysqld=--log-output=none --mysqld=--sql_mode=ONLY_FULL_GROUP_BY --mysqld=--plugin-load=tokudb=ha_tokudb.so --mysqld=--init-file=/data/bench/qa56dbg/randgen/conf/percona_qa/5.6/TokuDB.sql --mysqld=--utility-user-password=test --grammar=conf/percona_qa/5.6/5.6.yy --gendata=conf/percona_qa/5.6/5.6.zz1 --threads=15 --no-mask --basedir=/ssd/qa56dbg/Percona-Server-5.6.17-rel66.0-608.Linux.x86_64-debug --notnull --mysqld=--innodb_change_buffering=none --mysqld=--innodb_changed_pages=ON --mysqld=--innodb_fast_shutdown=2 --mysqld=--innodb_file_format=barracuda --mysqld=--innodb_file_per_table=1 --mysqld=--innodb_flush_method=O_DSYNC --mysqld=--innodb_log_arch_dir=/data/bench/qa56dbg/33/rundir1_327_epoch --mysqld=--innodb_log_arch_expire_sec=120 --mysqld=--innodb_log_archive=1 --mysqld=--innodb_log_buffer_size=1048577 --mysqld=--innodb_log_file_size=1048576 --mysqld=--innodb_log_files_in_group=2 --mysqld=--innodb_log_group_home_dir=/data/bench/qa56dbg/33/rundir1_327_epoch --mysqld=--innodb_max_bitmap_file_size=4097 --mysqld=--innodb_max_changed_pages=100 --mysqld=--innodb_track_changed_pages=1 --mysqld=--innodb_use_global_flush_log_at_trx_commit=0 --mysqld=--innodb-buffer-pool-populate --mysqld=--loose-readonly-key-cache-division-limit=1 --mysqld=--minimum-join-buffer-size=128 --mysqld=--query_cache_size=1048576 --mysqld=--query_cache_type=1 --mysqld=--readonly-key-cache-block-size=1 --mysqld=--readonly-loose-max-connect-errors=1 --mysqld=--slow_query_log --mysqld=--thread_handling=pool-of-threads --mysqld=--userstat --mysqld=--utility-user-password=test --mysqld=--hidden-key-buffer-size=0 --mysqld=--innodb_purge_threads=32 --mysqld=--innodb_adaptive_hash_index_partitions=16 --mysqld=--innodb_buffer_pool_instances=4 --mtr-build-thread=788 --mask=1571 --vardir1=/data/bench/qa56dbg/33/rundir1_327 > /data/bench/qa56dbg/33/cmd327.log 2>&1"
tail -n1 /data/bench/qa56dbg/33/cmd327.log

========================= version:
Server: 608 revid:<email address hidden>
RQG: 973 <email address hidden>

This bug seen several times in last QA run.

[Ramesh Sivaraman (rameshvs02)]

This issue is likely due to almost-out-of-disk-space and as such it is very low priority

[Laurynas Biveinis (laurynas-biveinis)]

The stacktraces are broken.

[Raghavendra D Prabhu (raghavendra-prabhu)]

@Ramesh,

Is this certainly always due to almost ENOSPC or are there any other conditions? It has also been reported here https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1325469 and I have asked for more details there.

[Ramesh Sivaraman (rameshvs02)]

@Raghu,

Yes, it is mostly due to lack of space in the system. I have initiated a fresh QA run after clearing the space. I will let you know if we get the same type of crashes again.

[Akshay Suryawanshi (akshay-suryawanshi)]

Similar crash with same Assertion Failure message was encountered a few days ago.

Server version: 5.6.22-71.0-log Percona Server (GPL), Release 71.0, Revision 726

2015-09-24 11:13:07 7f612f8bc700 InnoDB: Assertion failure in thread 140055386244864 in file os0file.cc line 5126
InnoDB: Failing assertion: slot != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
05:43:07 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=1073741824
read_buffer_size=131072
max_used_connections=126
max_threads=1202
thread_count=7
connection_count=5
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1527811 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x9091dc]
/usr/sbin/mysqld(handle_fatal_signal+0x352)[0x662482]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f909a1cb340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7f9099821bb9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f9099824fc8]
/usr/sbin/mysqld[0x98a2fb]
/usr/sbin/mysqld[0xa9dfb5]
/usr/sbin/mysqld[0x9f2d15]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f909a1c3182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f90998e5efd]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.
150924 11:13:07 mysqld_safe Number of processes running now: 0
150924 11:13:07 mysqld_safe mysqld restarted

However as specified in the reproducible case disk space wasnt a constraint. There was more than 100G space remaining on the mysql datadir volume.

Below is the my.cnf
[mysql]

port = 3306
socket = /dbvol/mysql/var/run/mysqld/mysqld.sock

[mysqld]

user = mysql
default-storage-engine = InnoDB
socket = /dbvol/mysql/var/run/mysqld/mysqld.sock
pid-file = /dbvol/mysql/var/run/mysqld/mysqld.pid

key-buffer-size = 1G
myisam-recover ...

[Launchpad Janitor (janitor)]

[Expired for Percona Server 5.5 because there has been no activity for 60 days.]

[Zach Moazeni (zmoazeni)]

We ran into a very similar issue on a production server. The line numbers are different, but here's some information from our crash.

We have plenty of disk space (for both our data and tmp) and plenty of memory. So that doesn't seem to be the cause for our crash.

Percona Server 5.6.33-79.0 on Ubuntu 14.04.5 LTS using Kernel 3.13.0-24-generic

2016-11-25 23:53:21 7f012806a700 InnoDB: Assertion failure in thread 139642943219456 in file os0file.cc line 5241
InnoDB: Failing assertion: slot != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
23:53:21 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=268435456
read_buffer_size=131072
max_used_connections=10429
max_threads=40002
thread_count=4851
connection_count=4849
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 87862134 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8d777c]
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x659df1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10330)[0x7f4d6b6ef330]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f4d6ab30c37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f4d6ab34028]
/usr/sbin/mysqld[0x98d67e]
/usr/sbin/mysqld[0xab0611]
/usr/sbin/mysqld[0x9fa6f0]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8184)[0x7f4d6b6e7184]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f4d6abf437d]
You may download the Percona Server operations manual by visiting
http://www.percona.com/software/percona-server/. You may find information
in the manual which will help you identify the cause of the crash.

[Sveta Smirnova (svetasmirnova)]

Akshay, Zach,

original issue was due to lack of free space. Please check if this could be the case in your environment too.

If this is not the case, please, enable core files and send us generated core together with full error log file and configuration file next time crash happens.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-2074