InnoDB: Failing assertion: event in file os0sync.cc line 475 | abort (sig=6) in os_event_reset on shutdown

[Roel Van de Paar]

(gdb) bt
#0 0x00007fee09631771 in pthread_kill () from /lib64/libpthread.so.0
#1 0x0000000000ab09a2 in my_write_core (sig=6) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.20-68.0/mysys/stacktrace.c:422
#2 0x000000000072ca03 in handle_fatal_signal (sig=6) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.20-68.0/sql/signal_handler.cc:236
#3 <signal handler called>
#4 0x00007fee08235989 in raise () from /lib64/libc.so.6
#5 0x00007fee08237098 in abort () from /lib64/libc.so.6
#6 0x0000000000b5d216 in os_event_reset (event=0x0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.20-68.0/storage/innobase/os/os0sync.cc:475
#7 0x0000000000b38038 in log_io_complete_checkpoint () at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.20-68.0/storage/innobase/log/log0log.cc:1924
#8 0x0000000000b365dc in log_io_complete (group=0x7fee077b3578) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.20-68.0/storage/innobase/log/log0log.cc:1270
#9 0x0000000000d2552f in fil_aio_wait (segment=1) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.20-68.0/storage/innobase/fil/fil0fil.cc:5861
#10 0x0000000000c27b45 in io_handler_thread (arg=0x18b1a28 <n+8>) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.20-68.0/storage/innobase/srv/srv0start.cc:498
#11 0x00007fee0962cdf3 in start_thread () from /lib64/libpthread.so.0
#12 0x00007fee082f63dd in clone () from /lib64/libc.so.6

2014-09-22 20:11:45 15979 [Note] InnoDB: Starting shutdown...
2014-09-22 20:11:46 7fed587fe700 InnoDB: Assertion failure in thread 140657368753920 in file os0sync.cc line 475
InnoDB: Failing assertion: event
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:11:46 UTC - mysqld got signal 6 ;

[Roel Van de Paar (roel11)]

============= Testcase:
1. Start mysqld:

[roel@localhost Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug]$ cat start
if [ -r /usr/lib64/libjemalloc.so.1 ]; then export LD_PRELOAD=/usr/lib64/libjemalloc.so.1
elif [ -r /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 ]; then export LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1
elif [ -r /sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/lib/mysql/libjemalloc.so.1 ]; then export LD_PRELOAD=/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/lib/mysql/libjemalloc.so.1
else echo 'Error: jemalloc not found, please install it first'; exit 1; fi
/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/bin/mysqld --innodb_buffer_pool_size=2147483648 --basedir=/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug --tmpdir=/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/data --datadir=/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/data --plugin-load=tokudb=ha_tokudb.so --core --socket=/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/socket.sock --port=10480 --log-error=/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/log/master.err 2>&1 &
echo 'Server socket: /sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/socket.sock with datadir: /sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/data'

2. Start client:

[roel@localhost Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug]$ cat cl
/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/bin/mysql -A -uroot -S/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/socket.sock test

3. Execute:

mysql> source bug1372210.sql;

4. Shutdown: (important!)

[roel@localhost Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug]$ cat stop
/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/bin/mysqladmin -uroot -S/sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/socket.sock shutdown
echo 'Server on socket /sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/socket.sock with datadir /sda/Percona-Server-5.6.20-rel68.0-657.Linux.x86_64-debug/data halted'

5. Check error log.

[Roel Van de Paar (roel11)]

Not present in upstream 5.6.20

[Roel Van de Paar (roel11)]

Shorter testcase

DROP DATABASE test;CREATE DATABASE test;USE test;
SET @@GLOBAL.innodb_track_changed_pages=1;
SELECT sleep(4);

+ Shutdown mysqld. Wait. Observe crash.