| 2013-03-15 06:58:44 |
Roel Van de Paar |
bug |
|
|
added bug |
| 2013-03-15 06:58:44 |
Roel Van de Paar |
attachment added |
|
thread apply all bt on core https://bugs.launchpad.net/bugs/1155475/+attachment/3575094/+files/gdb_9633_150313-1747_STD.txt |
|
| 2013-03-15 07:02:40 |
Roel Van de Paar |
description |
========================= Error log:
2013-03-15 06:38:57 5363 [Note] /ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64/bin/mysqld-debug: ready for connections.
Version: '5.6.10-alpha60.2-debug-log' socket: '/ssd/198649/current1_3/tmp/master.sock' port: 13040 Percona Server with XtraDB (GPL), Release alpha60.2, Revision 313-debug
2013-03-15 06:39:19 7f7611c86700 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2013-03-15 06:39:19 7f7611c86700 InnoDB: File name /ssd/tmp/ib_log_archive_0000000000045568
2013-03-15 06:39:19 7f7611c86700 InnoDB: File operation call: 'open' returned OS error 71.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Cannot continue operation.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Assertion failure in thread 140145081214720 in file os0file.cc line 662
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
========================= gdb :
Thread 1 (LWP 5366):
+bt
#0 0x0000003da180c60c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000006ab85f in handle_fatal_signal (sig=6) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/sql/signal_handler.cc:249
#2 <signal handler called>
#3 0x0000003da1435935 in raise () from /lib64/libc.so.6
#4 0x0000003da14370e8 in abort () from /lib64/libc.so.6
#5 0x00000000009f625c in os_file_handle_error_cond_exit (name=name@entry=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", operation=
0xdd84db "open", should_exit=should_exit@entry=1, on_error_silent=on_error_silent@entry=0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/stora
ge/innobase/os/os0file.cc:662
#6 0x00000000009f7c0a in os_file_handle_error (operation=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568") at /
ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:681
#7 os_file_create_func (name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", create_mode=51, purpose=<optimized out>, type=100, suc
cess=0x7f7611c858f0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:1741
#8 0x00000000009de72e in pfs_os_file_create_func (purpose=61, type=100, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage
/innobase/log/log0log.cc", src_line=2403, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", succ
ess=0x7f7611c858f0, type=100, purpose=61, create_mode=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", key=<opt
imized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/include/os0file.ic:151
#9 log_group_archive (group=0x3b6e068) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2403
#10 0x00000000009df809 in log_archive_groups () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2513
#11 log_archive_check_completion_low () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2612
#12 log_io_complete_archive () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2649
#13 log_io_complete (group=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:1060
#14 0x0000000000b7610a in fil_aio_wait (segment=1) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/fil/fil0fil.cc:5440
#15 0x0000000000a9f5e8 in io_handler_thread (arg=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/srv/srv0start
.cc:477
#16 0x0000003da1807d14 in start_thread () from /lib64/libpthread.so.0
#17 0x0000003da14f168d in clone () from /lib64/libc.so.6
========================= Run details:
========================= Notes: |
========================= Error log:
2013-03-15 06:38:57 5363 [Note] /ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64/bin/mysqld-debug: ready for connections.
Version: '5.6.10-alpha60.2-debug-log' socket: '/ssd/198649/current1_3/tmp/master.sock' port: 13040 Percona Server with XtraDB (GPL), Release alpha60.2, Revision 313-debug
2013-03-15 06:39:19 7f7611c86700 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2013-03-15 06:39:19 7f7611c86700 InnoDB: File name /ssd/tmp/ib_log_archive_0000000000045568
2013-03-15 06:39:19 7f7611c86700 InnoDB: File operation call: 'open' returned OS error 71.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Cannot continue operation.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Assertion failure in thread 140145081214720 in file os0file.cc line 662
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
========================= gdb :
Thread 1 (LWP 5366):
+bt
#0 0x0000003da180c60c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000006ab85f in handle_fatal_signal (sig=6) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/sql/signal_handler.cc:249
#2 <signal handler called>
#3 0x0000003da1435935 in raise () from /lib64/libc.so.6
#4 0x0000003da14370e8 in abort () from /lib64/libc.so.6
#5 0x00000000009f625c in os_file_handle_error_cond_exit (name=name@entry=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", operation=0xdd84db "open", should_exit=should_exit@entry=1, on_error_silent=on_error_silent@entry=0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:662
#6 0x00000000009f7c0a in os_file_handle_error (operation=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568") at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:681
#7 os_file_create_func (name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", create_mode=51, purpose=<optimized out>, type=100, success=0x7f7611c858f0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:1741
#8 0x00000000009de72e in pfs_os_file_create_func (purpose=61, type=100, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", src_line=2403, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", success=0x7f7611c858f0, type=100, purpose=61, create_mode=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", key=<opt
imized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/include/os0file.ic:151
#9 log_group_archive (group=0x3b6e068) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2403
#10 0x00000000009df809 in log_archive_groups () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2513
#11 log_archive_check_completion_low () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2612
#12 log_io_complete_archive () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2649
#13 log_io_complete (group=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:1060
#14 0x0000000000b7610a in fil_aio_wait (segment=1) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/fil/fil0fil.cc:5440
#15 0x0000000000a9f5e8 in io_handler_thread (arg=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/srv/srv0start.cc:477
#16 0x0000003da1807d14 in start_thread () from /lib64/libpthread.so.0
#17 0x0000003da14f168d in clone () from /lib64/libc.so.6
========================= Run details:
[Roel@qaserver 198649]$ cat ./cmd9633
ps -ef | grep 'cmdrun_9633' | grep -v grep | awk '{print $2}' | xargs sudo kill -9
rm -Rf /ssd/198649/cmdrun_9633
mkdir /ssd/198649/cmdrun_9633
cd /ssd/randgen
bash -c "set -o pipefail; perl runall.pl --queries=100000000 --seed=10879 --duration=120 --querytimeout=60 --short_column_names --reporter=Shutdown,Backtrace,QueryTimeout,ErrorLog,ErrorLogAlarm --mysqld=--log-output=none --mysqld=--sql_mode=ONLY_FULL_GROUP_BY --grammar=conf/percona_qa/percona_qa.yy --gendata=conf/percona_qa/percona_qa.zz --basedir=/ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64 --threads=25 --validator=Transformer --mysqld=--innodb_log_archive=1 --mysqld=--innodb_log_arch_dir=/ssd/tmp --mysqld=--innodb_log_group_home_dir=/ssd/tmp --mysqld=--innodb_log_arch_expire_sec=0 --mysqld=--innodb_flush_log_at_trx_commit=0 --mysqld=--innodb_flush_method=O_DIRECT --mtr-build-thread=788 --mask=56686 --vardir1=/ssd/198649/cmdrun_9633 > /ssd/198649/cmdrun9633.log 2>&1"
========================= Notes:
Seen in many runs. |
|
| 2013-03-15 07:03:15 |
Roel Van de Paar |
summary |
InnoDB: Failing assertion: 0 in file os0file.cc line 662 | File operation call: 'open' returned OS error 71. | InnoDB: Operating system error number 2 in a file operation. | handle_fatal_signal (sig=6) in os_file_handle_error_cond_exit |
InnoDB: Failing assertion: 0 in file os0file.cc line 662 | handle_fatal_signal (sig=6) in os_file_handle_error_cond_exit | File operation call: 'open' returned OS error 71. | InnoDB: Operating system error number 2 in a file operation. |
|
| 2013-03-15 07:03:40 |
Roel Van de Paar |
description |
========================= Error log:
2013-03-15 06:38:57 5363 [Note] /ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64/bin/mysqld-debug: ready for connections.
Version: '5.6.10-alpha60.2-debug-log' socket: '/ssd/198649/current1_3/tmp/master.sock' port: 13040 Percona Server with XtraDB (GPL), Release alpha60.2, Revision 313-debug
2013-03-15 06:39:19 7f7611c86700 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2013-03-15 06:39:19 7f7611c86700 InnoDB: File name /ssd/tmp/ib_log_archive_0000000000045568
2013-03-15 06:39:19 7f7611c86700 InnoDB: File operation call: 'open' returned OS error 71.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Cannot continue operation.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Assertion failure in thread 140145081214720 in file os0file.cc line 662
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
========================= gdb :
Thread 1 (LWP 5366):
+bt
#0 0x0000003da180c60c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000006ab85f in handle_fatal_signal (sig=6) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/sql/signal_handler.cc:249
#2 <signal handler called>
#3 0x0000003da1435935 in raise () from /lib64/libc.so.6
#4 0x0000003da14370e8 in abort () from /lib64/libc.so.6
#5 0x00000000009f625c in os_file_handle_error_cond_exit (name=name@entry=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", operation=0xdd84db "open", should_exit=should_exit@entry=1, on_error_silent=on_error_silent@entry=0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:662
#6 0x00000000009f7c0a in os_file_handle_error (operation=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568") at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:681
#7 os_file_create_func (name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", create_mode=51, purpose=<optimized out>, type=100, success=0x7f7611c858f0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:1741
#8 0x00000000009de72e in pfs_os_file_create_func (purpose=61, type=100, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", src_line=2403, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", success=0x7f7611c858f0, type=100, purpose=61, create_mode=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", key=<opt
imized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/include/os0file.ic:151
#9 log_group_archive (group=0x3b6e068) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2403
#10 0x00000000009df809 in log_archive_groups () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2513
#11 log_archive_check_completion_low () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2612
#12 log_io_complete_archive () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2649
#13 log_io_complete (group=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:1060
#14 0x0000000000b7610a in fil_aio_wait (segment=1) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/fil/fil0fil.cc:5440
#15 0x0000000000a9f5e8 in io_handler_thread (arg=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/srv/srv0start.cc:477
#16 0x0000003da1807d14 in start_thread () from /lib64/libpthread.so.0
#17 0x0000003da14f168d in clone () from /lib64/libc.so.6
========================= Run details:
[Roel@qaserver 198649]$ cat ./cmd9633
ps -ef | grep 'cmdrun_9633' | grep -v grep | awk '{print $2}' | xargs sudo kill -9
rm -Rf /ssd/198649/cmdrun_9633
mkdir /ssd/198649/cmdrun_9633
cd /ssd/randgen
bash -c "set -o pipefail; perl runall.pl --queries=100000000 --seed=10879 --duration=120 --querytimeout=60 --short_column_names --reporter=Shutdown,Backtrace,QueryTimeout,ErrorLog,ErrorLogAlarm --mysqld=--log-output=none --mysqld=--sql_mode=ONLY_FULL_GROUP_BY --grammar=conf/percona_qa/percona_qa.yy --gendata=conf/percona_qa/percona_qa.zz --basedir=/ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64 --threads=25 --validator=Transformer --mysqld=--innodb_log_archive=1 --mysqld=--innodb_log_arch_dir=/ssd/tmp --mysqld=--innodb_log_group_home_dir=/ssd/tmp --mysqld=--innodb_log_arch_expire_sec=0 --mysqld=--innodb_flush_log_at_trx_commit=0 --mysqld=--innodb_flush_method=O_DIRECT --mtr-build-thread=788 --mask=56686 --vardir1=/ssd/198649/cmdrun_9633 > /ssd/198649/cmdrun9633.log 2>&1"
========================= Notes:
Seen in many runs. |
========================= Error log:
2013-03-15 06:38:57 5363 [Note] /ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64/bin/mysqld-debug: ready for connections.
Version: '5.6.10-alpha60.2-debug-log' socket: '/ssd/198649/current1_3/tmp/master.sock' port: 13040 Percona Server with XtraDB (GPL), Release alpha60.2, Revision 313-debug
2013-03-15 06:39:19 7f7611c86700 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2013-03-15 06:39:19 7f7611c86700 InnoDB: File name /ssd/tmp/ib_log_archive_0000000000045568
2013-03-15 06:39:19 7f7611c86700 InnoDB: File operation call: 'open' returned OS error 71.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Cannot continue operation.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Assertion failure in thread 140145081214720 in file os0file.cc line 662
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
========================= gdb :
Thread 1 (LWP 5366):
+bt
#0 0x0000003da180c60c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000006ab85f in handle_fatal_signal (sig=6) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/sql/signal_handler.cc:249
#2 <signal handler called>
#3 0x0000003da1435935 in raise () from /lib64/libc.so.6
#4 0x0000003da14370e8 in abort () from /lib64/libc.so.6
#5 0x00000000009f625c in os_file_handle_error_cond_exit (name=name@entry=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", operation=0xdd84db "open", should_exit=should_exit@entry=1, on_error_silent=on_error_silent@entry=0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:662
#6 0x00000000009f7c0a in os_file_handle_error (operation=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568") at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:681
#7 os_file_create_func (name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", create_mode=51, purpose=<optimized out>, type=100, success=0x7f7611c858f0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:1741
#8 0x00000000009de72e in pfs_os_file_create_func (purpose=61, type=100, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", src_line=2403, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", success=0x7f7611c858f0, type=100, purpose=61, create_mode=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", key=<opt
imized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/include/os0file.ic:151
#9 log_group_archive (group=0x3b6e068) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2403
#10 0x00000000009df809 in log_archive_groups () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2513
#11 log_archive_check_completion_low () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2612
#12 log_io_complete_archive () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2649
#13 log_io_complete (group=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:1060
#14 0x0000000000b7610a in fil_aio_wait (segment=1) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/fil/fil0fil.cc:5440
#15 0x0000000000a9f5e8 in io_handler_thread (arg=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/srv/srv0start.cc:477
#16 0x0000003da1807d14 in start_thread () from /lib64/libpthread.so.0
#17 0x0000003da14f168d in clone () from /lib64/libc.so.6
========================= Run details:
[Roel@qaserver 198649]$ cat ./cmd9633
ps -ef | grep 'cmdrun_9633' | grep -v grep | awk '{print $2}' | xargs sudo kill -9
rm -Rf /ssd/198649/cmdrun_9633
mkdir /ssd/198649/cmdrun_9633
cd /ssd/randgen
bash -c "set -o pipefail; perl runall.pl --queries=100000000 --seed=10879 --duration=120 --querytimeout=60 --short_column_names --reporter=Shutdown,Backtrace,QueryTimeout,ErrorLog,ErrorLogAlarm --mysqld=--log-output=none --mysqld=--sql_mode=ONLY_FULL_GROUP_BY --grammar=conf/percona_qa/percona_qa.yy --gendata=conf/percona_qa/percona_qa.zz --basedir=/ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64 --threads=25 --validator=Transformer --mysqld=--innodb_log_archive=1 --mysqld=--innodb_log_arch_dir=/ssd/tmp --mysqld=--innodb_log_group_home_dir=/ssd/tmp --mysqld=--innodb_log_arch_expire_sec=0 --mysqld=--innodb_flush_log_at_trx_commit=0 --mysqld=--innodb_flush_method=O_DIRECT --mtr-build-thread=788 --mask=56686 --vardir1=/ssd/198649/cmdrun_9633 > /ssd/198649/cmdrun9633.log 2>&1"
========================= Notes:
- Seen in many runs.
- Long title is for SEO purposes only |
|
| 2013-03-15 07:04:12 |
Roel Van de Paar |
description |
========================= Error log:
2013-03-15 06:38:57 5363 [Note] /ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64/bin/mysqld-debug: ready for connections.
Version: '5.6.10-alpha60.2-debug-log' socket: '/ssd/198649/current1_3/tmp/master.sock' port: 13040 Percona Server with XtraDB (GPL), Release alpha60.2, Revision 313-debug
2013-03-15 06:39:19 7f7611c86700 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2013-03-15 06:39:19 7f7611c86700 InnoDB: File name /ssd/tmp/ib_log_archive_0000000000045568
2013-03-15 06:39:19 7f7611c86700 InnoDB: File operation call: 'open' returned OS error 71.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Cannot continue operation.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Assertion failure in thread 140145081214720 in file os0file.cc line 662
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
========================= gdb :
Thread 1 (LWP 5366):
+bt
#0 0x0000003da180c60c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000006ab85f in handle_fatal_signal (sig=6) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/sql/signal_handler.cc:249
#2 <signal handler called>
#3 0x0000003da1435935 in raise () from /lib64/libc.so.6
#4 0x0000003da14370e8 in abort () from /lib64/libc.so.6
#5 0x00000000009f625c in os_file_handle_error_cond_exit (name=name@entry=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", operation=0xdd84db "open", should_exit=should_exit@entry=1, on_error_silent=on_error_silent@entry=0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:662
#6 0x00000000009f7c0a in os_file_handle_error (operation=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568") at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:681
#7 os_file_create_func (name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", create_mode=51, purpose=<optimized out>, type=100, success=0x7f7611c858f0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:1741
#8 0x00000000009de72e in pfs_os_file_create_func (purpose=61, type=100, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", src_line=2403, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", success=0x7f7611c858f0, type=100, purpose=61, create_mode=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", key=<opt
imized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/include/os0file.ic:151
#9 log_group_archive (group=0x3b6e068) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2403
#10 0x00000000009df809 in log_archive_groups () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2513
#11 log_archive_check_completion_low () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2612
#12 log_io_complete_archive () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2649
#13 log_io_complete (group=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:1060
#14 0x0000000000b7610a in fil_aio_wait (segment=1) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/fil/fil0fil.cc:5440
#15 0x0000000000a9f5e8 in io_handler_thread (arg=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/srv/srv0start.cc:477
#16 0x0000003da1807d14 in start_thread () from /lib64/libpthread.so.0
#17 0x0000003da14f168d in clone () from /lib64/libc.so.6
========================= Run details:
[Roel@qaserver 198649]$ cat ./cmd9633
ps -ef | grep 'cmdrun_9633' | grep -v grep | awk '{print $2}' | xargs sudo kill -9
rm -Rf /ssd/198649/cmdrun_9633
mkdir /ssd/198649/cmdrun_9633
cd /ssd/randgen
bash -c "set -o pipefail; perl runall.pl --queries=100000000 --seed=10879 --duration=120 --querytimeout=60 --short_column_names --reporter=Shutdown,Backtrace,QueryTimeout,ErrorLog,ErrorLogAlarm --mysqld=--log-output=none --mysqld=--sql_mode=ONLY_FULL_GROUP_BY --grammar=conf/percona_qa/percona_qa.yy --gendata=conf/percona_qa/percona_qa.zz --basedir=/ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64 --threads=25 --validator=Transformer --mysqld=--innodb_log_archive=1 --mysqld=--innodb_log_arch_dir=/ssd/tmp --mysqld=--innodb_log_group_home_dir=/ssd/tmp --mysqld=--innodb_log_arch_expire_sec=0 --mysqld=--innodb_flush_log_at_trx_commit=0 --mysqld=--innodb_flush_method=O_DIRECT --mtr-build-thread=788 --mask=56686 --vardir1=/ssd/198649/cmdrun_9633 > /ssd/198649/cmdrun9633.log 2>&1"
========================= Notes:
- Seen in many runs.
- Long title is for SEO purposes only |
========================= Error log:
2013-03-15 06:38:57 5363 [Note] /ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64/bin/mysqld-debug: ready for connections.
Version: '5.6.10-alpha60.2-debug-log' socket: '/ssd/198649/current1_3/tmp/master.sock' port: 13040 Percona Server with XtraDB (GPL), Release alpha60.2, Revision 313-debug
2013-03-15 06:39:19 7f7611c86700 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
2013-03-15 06:39:19 7f7611c86700 InnoDB: File name /ssd/tmp/ib_log_archive_0000000000045568
2013-03-15 06:39:19 7f7611c86700 InnoDB: File operation call: 'open' returned OS error 71.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Cannot continue operation.
2013-03-15 06:39:19 7f7611c86700 InnoDB: Assertion failure in thread 140145081214720 in file os0file.cc line 662
InnoDB: Failing assertion: 0
InnoDB: We intentionally generate a memory trap.
========================= gdb :
Thread 1 (LWP 5366):
+bt
#0 0x0000003da180c60c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000006ab85f in handle_fatal_signal (sig=6) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/sql/signal_handler.cc:249
#2 <signal handler called>
#3 0x0000003da1435935 in raise () from /lib64/libc.so.6
#4 0x0000003da14370e8 in abort () from /lib64/libc.so.6
#5 0x00000000009f625c in os_file_handle_error_cond_exit (name=name@entry=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", operation=0xdd84db "open", should_exit=should_exit@entry=1, on_error_silent=on_error_silent@entry=0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:662
#6 0x00000000009f7c0a in os_file_handle_error (operation=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568") at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:681
#7 os_file_create_func (name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", create_mode=51, purpose=<optimized out>, type=100, success=0x7f7611c858f0) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/os/os0file.cc:1741
#8 0x00000000009de72e in pfs_os_file_create_func (purpose=61, type=100, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", src_line=2403, src_file=0xdf04f8 "/ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc", success=0x7f7611c858f0, type=100, purpose=61, create_mode=<optimized out>, name=0x7f7611c85950 "/ssd/tmp/ib_log_archive_", '0' <repeats 11 times>, "45568", key=<opt
imized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/include/os0file.ic:151
#9 log_group_archive (group=0x3b6e068) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2403
#10 0x00000000009df809 in log_archive_groups () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2513
#11 log_archive_check_completion_low () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2612
#12 log_io_complete_archive () at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:2649
#13 log_io_complete (group=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/log/log0log.cc:1060
#14 0x0000000000b7610a in fil_aio_wait (segment=1) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/fil/fil0fil.cc:5440
#15 0x0000000000a9f5e8 in io_handler_thread (arg=<optimized out>) at /ssd/ps56-univ-log-archive-qa/Percona-Server-5.6.8-alpha60.2/storage/innobase/srv/srv0start.cc:477
#16 0x0000003da1807d14 in start_thread () from /lib64/libpthread.so.0
#17 0x0000003da14f168d in clone () from /lib64/libc.so.6
========================= Run details:
[Roel@qaserver 198649]$ cat ./cmd9633
ps -ef | grep 'cmdrun_9633' | grep -v grep | awk '{print $2}' | xargs sudo kill -9
rm -Rf /ssd/198649/cmdrun_9633
mkdir /ssd/198649/cmdrun_9633
cd /ssd/randgen
bash -c "set -o pipefail; perl runall.pl --queries=100000000 --seed=10879 --duration=120 --querytimeout=60 --short_column_names --reporter=Shutdown,Backtrace,QueryTimeout,ErrorLog,ErrorLogAlarm --mysqld=--log-output=none --mysqld=--sql_mode=ONLY_FULL_GROUP_BY --grammar=conf/percona_qa/percona_qa.yy --gendata=conf/percona_qa/percona_qa.zz --basedir=/ssd/Percona-Server-5.6.8-alpha60.2-313-debug.Linux.x86_64 --threads=25 --validator=Transformer --mysqld=--innodb_log_archive=1 --mysqld=--innodb_log_arch_dir=/ssd/tmp --mysqld=--innodb_log_group_home_dir=/ssd/tmp --mysqld=--innodb_log_arch_expire_sec=0 --mysqld=--innodb_flush_log_at_trx_commit=0 --mysqld=--innodb_flush_method=O_DIRECT --mtr-build-thread=788 --mask=56686 --vardir1=/ssd/198649/cmdrun_9633 > /ssd/198649/cmdrun9633.log 2>&1"
========================= Notes:
- Seen in many runs.
- Long title is for SE search purposes only |
|
| 2013-03-15 07:04:19 |
Roel Van de Paar |
nominated for series |
|
percona-server/5.6 |
|
| 2013-03-15 07:04:19 |
Roel Van de Paar |
bug task added |
|
percona-server/5.6 |
|
| 2013-03-15 07:06:02 |
Roel Van de Paar |
attachment added |
|
thread apply all bt full (with local vars) on core https://bugs.launchpad.net/percona-server/+bug/1155475/+attachment/3575115/+files/gdb_9633_150313-1747_FULL.txt |
|
| 2013-03-15 07:06:28 |
Roel Van de Paar |
attachment added |
|
mysqld error log https://bugs.launchpad.net/percona-server/+bug/1155475/+attachment/3575116/+files/master_150313-1747.err |
|
| 2013-03-15 07:37:09 |
Sergei Glushchenko |
percona-server/5.6: assignee |
Vlad Lesin (vlad-lesin) |
Sergei Glushchenko (sergei.glushchenko) |
|
| 2013-03-15 07:57:34 |
Roel Van de Paar |
percona-server/5.6: importance |
Undecided |
High |
|
| 2013-03-16 12:24:35 |
Sergei Glushchenko |
percona-server/5.6: status |
Confirmed |
Triaged |
|
| 2013-03-16 12:24:42 |
Sergei Glushchenko |
percona-server/5.6: status |
Triaged |
In Progress |
|
| 2013-03-16 12:24:46 |
Sergei Glushchenko |
percona-server/5.6: status |
In Progress |
Fix Committed |
|
| 2013-03-16 12:25:34 |
Sergei Glushchenko |
branch linked |
|
lp:~sergei.glushchenko/percona-server/ps56-univ-log-archive |
|
| 2013-03-21 06:33:26 |
Launchpad Janitor |
branch linked |
|
lp:~sergei.glushchenko/percona-server/56-bug1157078-bug1157075 |
|
| 2013-03-21 07:33:15 |
Sergei Glushchenko |
branch unlinked |
lp:~sergei.glushchenko/percona-server/56-bug1157078-bug1157075 |
|
|
| 2013-04-03 06:40:26 |
Stewart Smith |
percona-server/5.6: milestone |
|
5.6.10-60.3 |
|
| 2013-04-03 06:40:29 |
Stewart Smith |
percona-server/5.6: status |
Fix Committed |
Fix Released |
|
| 2013-04-18 19:50:07 |
Roel Van de Paar |
bug watch added |
|
http://bugs.mysql.com/bug.php?id=69000 |
|
