server won't start when flush_method=ALL_O_DIRECT and track_changed_pages=TRUE
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| Percona Server moved to https://jira.percona.com/projects/PS |
Fix Released
|
High
|
George Ormond Lorch III | |
| 5.1 |
Fix Released
|
High
|
Laurynas Biveinis | |
| 5.5 |
Fix Released
|
High
|
Laurynas Biveinis | |
| 5.6 |
Fix Released
|
High
|
George Ormond Lorch III |
Bug Description
Server fails to start with:
InnoDB: Error: tried to read 65536 bytes at offset 0 2048.
InnoDB: Was only able to read 0.
my.cnf:
[client]
port=10031
socket=
[mysqld]
basedir=
datadir=
tmpdir=
port=10031
socket=
pid-file=
console
user=root
server-id=1
innodb_
innodb_
Using branch:
parent branch: bzr+ssh:
Related branches
- Stewart Smith (community): Approve on 2013-05-31
-
Diff: 1153 lines (+400/-142) (has conflicts)16 files modifiedPercona-Server/mysql-test/suite/innodb_plugin/r/percona_changed_page_bmp.result (+7/-0)
Percona-Server/mysql-test/suite/innodb_plugin/r/percona_changed_pages.result (+58/-33)
Percona-Server/mysql-test/suite/innodb_plugin/r/percona_changed_pages_empty.result (+1/-0)
Percona-Server/mysql-test/suite/innodb_plugin/t/percona_changed_page_bmp.test (+33/-0)
Percona-Server/mysql-test/suite/innodb_plugin/t/percona_changed_page_bmp_flush.test (+2/-2)
Percona-Server/mysql-test/suite/innodb_plugin/t/percona_changed_pages.test (+99/-28)
Percona-Server/mysql-test/suite/innodb_plugin/t/percona_changed_pages_empty.test (+4/-2)
Percona-Server/storage/innodb_plugin/handler/i_s.cc (+8/-6)
Percona-Server/storage/innodb_plugin/include/log0log.h (+3/-1)
Percona-Server/storage/innodb_plugin/include/log0online.h (+2/-0)
Percona-Server/storage/innodb_plugin/include/os0file.h (+4/-2)
Percona-Server/storage/innodb_plugin/log/log0log.c (+13/-3)
Percona-Server/storage/innodb_plugin/log/log0online.c (+162/-54)
Percona-Server/storage/innodb_plugin/log/log0recv.c (+4/-3)
Percona-Server/storage/innodb_plugin/os/os0file.c (+0/-2)
Percona-Server/storage/innodb_plugin/srv/srv0start.c (+0/-6)
- Stewart Smith (community): Approve on 2013-05-31
-
Diff: 1312 lines (+537/-143) (has conflicts)19 files modifiedPercona-Server/mysql-test/suite/innodb/r/percona_changed_page_bmp.result (+7/-0)
Percona-Server/mysql-test/suite/innodb/r/percona_changed_pages.result (+58/-33)
Percona-Server/mysql-test/suite/innodb/r/percona_changed_pages_empty.result (+1/-0)
Percona-Server/mysql-test/suite/innodb/t/percona_changed_page_bmp.test (+33/-0)
Percona-Server/mysql-test/suite/innodb/t/percona_changed_page_bmp_flush.test (+2/-2)
Percona-Server/mysql-test/suite/innodb/t/percona_changed_pages.test (+99/-28)
Percona-Server/mysql-test/suite/innodb/t/percona_changed_pages_empty.test (+4/-2)
Percona-Server/mysql-test/suite/sys_vars/r/all_vars.result (+0/-2)
Percona-Server/mysql-test/suite/sys_vars/r/innodb_max_changed_pages_basic.result (+78/-0)
Percona-Server/mysql-test/suite/sys_vars/t/innodb_max_changed_pages_basic.test (+60/-0)
Percona-Server/storage/innobase/handler/i_s.cc (+8/-6)
Percona-Server/storage/innobase/include/log0log.h (+3/-1)
Percona-Server/storage/innobase/include/log0online.h (+2/-0)
Percona-Server/storage/innobase/include/os0file.h (+4/-2)
Percona-Server/storage/innobase/log/log0log.c (+13/-3)
Percona-Server/storage/innobase/log/log0online.c (+161/-53)
Percona-Server/storage/innobase/log/log0recv.c (+4/-3)
Percona-Server/storage/innobase/os/os0file.c (+0/-2)
Percona-Server/storage/innobase/srv/srv0start.c (+0/-6)
- Laurynas Biveinis: Approve on 2013-09-27
-
Diff: 176 lines (+49/-12)7 files modifiedPercona-Server/mysql-test/suite/innodb/r/percona_changed_page_bmp_flush.result (+6/-0)
Percona-Server/mysql-test/suite/innodb/t/percona_changed_page_bmp_flush.test (+16/-1)
Percona-Server/storage/innobase/fil/fil0fil.cc (+6/-4)
Percona-Server/storage/innobase/include/srv0srv.h (+5/-1)
Percona-Server/storage/innobase/log/log0log.cc (+9/-4)
Percona-Server/storage/innobase/os/os0file.cc (+4/-2)
Percona-Server/storage/innobase/srv/srv0start.cc (+3/-0)
George Ormond Lorch III (gl-az) wrote : | #1 |
tags: | added: bitmap xtradb |
tags: | added: qa |
George Ormond Lorch III (gl-az) wrote : Re: [Bug 1131949] Re: server won't start when flush_method=ALL_O_DIRECT and track_changed_pages=TRUE | #3 |
Sure Laurynas, in fact, I'll even go a little further and see if I can
debug/fix it for you, should be pretty straightforward.
On 5/27/2013 10:50 AM, Laurynas Biveinis wrote:
> George -
>
> I cannot reproduce this locally. Could you please
> 1) attach the full error log for the failed run;
> 2) set a breakpoint at these InnoDB diagnostics you get and provide a stacktrace leading either to the diagnostics or the failing call itself?
>
> ** Changed in: percona-server/5.1
> Status: In Progress => Triaged
>
> ** Changed in: percona-server/5.5
> Assignee: Laurynas Biveinis (laurynas-biveinis) => George Ormond Lorch III (gl-az)
>
> ** Changed in: percona-server/5.1
> Assignee: Laurynas Biveinis (laurynas-biveinis) => George Ormond Lorch III (gl-az)
>
> ** Changed in: percona-server/5.6
> Assignee: Laurynas Biveinis (laurynas-biveinis) => George Ormond Lorch III (gl-az)
>
--
George O. Lorch III
Software Engineer, Percona
+1-888-401-3401 x542 US/Arizona (GMT -7)
skype: george.
George Ormond Lorch III (gl-az) wrote : | #4 |
Looks like it is unaligned i/o on log file
Backtrace:
(gdb) bt
#0 _os_file_read (file=12, buf=0x7fffa8966010, offset=2048, offset_high=0, n=65536, trx=0x0) at os/os0file.c:2525
#1 0x0000000000865fc0 in os_aio (type=10, mode=24, name=0x3a948b8 "./ib_logfile0", file=12, buf=0x7fffa8966010, offset=2048, offset_high=0, n=65536, message1=0x3a947b8, message2=0x0, space_id=
#2 0x0000000000935d58 in _fil_io (type=10, sync=1, space_id=
#3 0x0000000000853f7b in log_group_
#4 0x0000000000856504 in log_online_
#5 0x0000000000856697 in log_online_
#6 0x0000000000856ba9 in log_online_
#7 0x00000000008b514d in srv_redo_
#8 0x000000391b007851 in start_thread () from /lib64/
#9 0x000000391a8e811d in clone () from /lib64/libc.so.6
5.6 does not have the option to set O_DIRECT for log files. Will merge the fix of aligning the read buffer at OS_FILE_
Same happens with Percona Server 5.7.10-3 GA as soon as I have:
innodb_
in my.cnf:
[root@centos ~]# service mysql stop
Stopping mysqld: [ OK ]
[root@centos ~]# vi /etc/my.cnf
[root@centos ~]# service mysql start
Percona MySQL Daemon failed to start.
Starting mysqld: [FAILED]
[root@centos ~]# tail -100 /var/log/mysqld.log
...
2016-02-
160226 17:49:07 mysqld_safe mysqld from pid file /var/run/
160226 17:50:41 mysqld_safe Transparent huge pages are already set to: never.
160226 17:50:41 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
2016-02-
This has the same root cause as bug 1529885, and the fix will be tracked there.
tags: | added: i64171 |
Shahriyar Rzayev (rzayev-sehriyar) wrote : | #8 |
Percona now uses JIRA for bug reports so this bug report is migrated to: https:/
George -
I cannot reproduce this locally. Could you please
1) attach the full error log for the failed run;
2) set a breakpoint at these InnoDB diagnostics you get and provide a stacktrace leading either to the diagnostics or the failing call itself?