innodb_auto_lru_dump crashes if ib_lru_dump doesn't exist

[Baron Schwartz]

It is impossible to set innodb_auto_lru_dump and start the server unless you have previously saved a dump file. XtraDB should check for the existence of the file at startup and only load it if the file exists.

101219 9:24:43 [Note] Flashcache bypass: disabled
101219 9:24:43 [Note] Flashcache setup error is : open flash device failed

101219 9:24:43 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use InnoDB's own implementation
InnoDB: Compressed tables use zlib 1.2.3
101219 9:24:44 InnoDB: highest supported file format is Barracuda.
101219 9:24:44 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
 InnoDB: cannot open ib_lru_dump
101219 9:24:44 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=0
max_threads=151
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 337790 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x30000
./libexec/mysqld(my_print_stacktrace+0x21)[0x84bea81]
./libexec/mysqld(handle_segfault+0x37f)[0x81d96bf]
[0xefa400]
./libexec/mysqld[0x8422ecc]
101219 9:24:44 Percona XtraDB (http://www.percona.com) 1.0.11-12.1 started; log sequence number 16046380
./libexec/mysqld[0x83cd95f]
/lib/libpthread.so.0[0x847919]
/lib/libc.so.6(clone+0x5e)[0x790cce]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

[Alexey Kopytov (akopytov)]

Baron,

I cannot repeat the crash. This is what I get after starting with --innodb_auto_lru_dump=5 and no ib_lru_dump in the data dir:

$ ./mysqld --innodb_auto_lru_dump=5
101221 0:58:56 [Warning] Setting lower_case_table_names=2 because file system for /usr/local/mysql/var/ is case insensitive
101221 0:58:56 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use InnoDB's own implementation
InnoDB: Compressed tables use zlib 1.2.5
101221 0:58:56 InnoDB: highest supported file format is Barracuda.
101221 0:58:56 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
InnoDB: If you are installing InnoDB, remember that you must create
InnoDB: directories yourself, InnoDB does not create them.
 InnoDB: cannot open ib_lru_dump
101221 0:58:56 Percona XtraDB (http://www.percona.com) 1.0.13-12.4 started; log sequence number 527792833
101221 0:58:56 [Note] Event Scheduler: Loaded 0 events
101221 0:58:56 [Note] ./mysqld: ready for connections.
Version: '5.1.53-debug' socket: '/tmp/mysql.sock' port: 3306 Source distribution

Unfortunately, stack traces in the error log are useless for plugins, as addresses are not resolved. Can you run mysqld under gdb and get a gdb stack trace for the crash?

[Baron Schwartz (baron-xaprb)]

Maybe my architecture, etc is also important. Here are more details:

[baron@ginger 5.1.50-rel12.1-127]$ uname -a
Linux ginger 2.6.34.7-56.fc13.i686.PAE #1 SMP Wed Sep 15 03:27:15 UTC 2010 i686 i686 i386 GNU/Linux

My command-line here is:

./libexec/mysqld --defaults-file=my.cnf

and here is my configuration file:

[baron@ginger 5.1.50-rel12.1-127]$ cat my.cnf
[client]
user = root
pass = msandbox
port = 15150
protocol = tcp
socket = /home/baron/etc/mysql/server/5.1.50-rel12.1-127/data/mysql.sock
[mysql]
prompt = "15150> "
[mysqld]
datadir = /home/baron/etc/mysql/server/5.1.50-rel12.1-127/data/
port = 15150
socket = /home/baron/etc/mysql/server/5.1.50-rel12.1-127/data/mysql.sock
language = ./share/mysql/english
basedir = /home/baron/etc/mysql/server/5.1.50-rel12.1-127
innodb_auto_lru_dump=5

And here is the result under GDB:

[baron@ginger 5.1.50-rel12.1-127]$ gdb libexec/mysqld
GNU gdb (GDB) Fedora (7.1-34.fc13)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/baron/etc/mysql/server/5.1.50-rel12.1-127/libexec/mysqld...(no debugging symbols found)...done.
(gdb) run --defaults-file=my.cnf
Starting program: /home/baron/etc/mysql/server/5.1.50-rel12.1-127/libexec/mysqld --defaults-file=my.cnf
[Thread debugging using libthread_db enabled]
[New Thread 0xb7fe7b70 (LWP 20675)]
[Thread 0xb7fe7b70 (LWP 20675) exited]
101221 14:40:53 [Note] Flashcache bypass: disabled
101221 14:40:53 [Note] Flashcache setup error is : open flash device failed

101221 14:40:53 [Note] Plugin 'FEDERATED' is disabled.
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use InnoDB's own implementation
InnoDB: Compressed tables use zlib 1.2.3
[New Thread 0xb7fe7b70 (LWP 20676)]
[New Thread 0xade41b70 (LWP 20677)]
[New Thread 0xad440b70 (LWP 20678)]
[New Thread 0xaca3fb70 (LWP 20679)]
[New Thread 0xac03eb70 (LWP 20680)]
[New Thread 0xab63db70 (LWP 20681)]
[New Thread 0xaac3cb70 (LWP 20682)]
[New Thread 0xaa23bb70 (LWP 20683)]
[New Thread 0xa983ab70 (LWP 20684)]
[New Thread 0xa8e39b70 (LWP 20685)]
101221 14:40:54 InnoDB: highest supported file format is Barracuda.
[New Thread 0xa7e9ab70 (LWP 20686)]
[New Thread 0xa7499b70 (LWP 20687)]
[Thread 0xa7e9ab70 (LWP 20686) exited]
[New Thread 0xa7e9ab70 (LWP 20688)]
[New Thread 0xa6a98b70 (LWP 20689)]
[New Thread 0xa6097b70 (LWP 20690)]
101221 14:40:54 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
 InnoDB: cannot open ib_lru_dump

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xa6097b70 (LWP 20690)]
0x00726a48 in free () from /lib/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.12.1-2.i686 libgcc-4.4.4-10.fc13.i686 nss-softokn-freebl-3.12.7-6.fc1...

[Baron Schwartz (baron-xaprb)]

Can you describe the fix quickly for the records?

[Alexey Kopytov (akopytov)]

The fix is described in the revision comments and the merge proposal for this bug (I wish Launchpad would make them visible somehow in the bug). I'll quote them here:

"
    Bug #692211: innodb_auto_lru_dump crashes if ib_lru_dump doesn't exist

    Starting the server with a non-zero innodb_auto_lru_dump value could
    crash the server if the dump file does not exist.

    The problem was that the 'records' pointer was not initialized at the
    start of buf_LRU_file_restore(). This could lead to calling ut_free()
    with uninitialized value when the dump file is not found and further
    execution of buf_LRU_file_restore() is therefore aborted.

    Fixed by initializing 'records' with NULL.
"

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-453