ALL_O_DIRECT causes unaligned AIO/DIO

Bug #1033051 reported by Raghavendra D Prabhu
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Fix Released
Medium
Hrvoje Matijakovic
5.1
Fix Released
Medium
Hrvoje Matijakovic
5.5
Fix Released
Medium
Hrvoje Matijakovic

Bug Description

Setting innodb_flush_method to ALL_O_DIRECT seems to be causing unaligned AIO/DIO.

It causes "EXT4-fs (dm-0): Unaligned AIO/DIO on inode 1439824 by mysqld; performance will be poor." in dmesg where inode 1439824 is none other than ib_logfile0.

To reproduce:

1. Set innodb_flush_method=ALL_O_DIRECT in my.cnf and restart the server.

2. Check for something along the lines of

"EXT4-fs (dm-0): Unaligned AIO/DIO on inode 1442673 by mysqld; performance will be poor."

in dmesg.

Now, this dmesg message is from http://lxr.linux.no/#linux+v3.5/fs/ext4/file.c#L129 and is printed when unaligned AIO (with direct-io) take place; it is serialized to avoid the corruption, and it is this serialization which causes performance loss.

Since this message appears only on ALL_O_DIRECT and not on O_DIRECT, it looks like somewhere in log flushing/write code there must be unaligned access (aligned I/O is a must for direct-io).

Tags: doc i25299

Related branches

tags: added: i25299
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Wasn't that hard to diagnose after checking log_group_write_buf that writes are aligned wrt. OS_FILE_LOG_BLOCK_SIZE. This block size is tunable only in Percona server with innodb-log-block-size.

So, the 'fix' is quite simply setting innodb-block-size to 4096.

4096 because it is the default log-block-size in ext4 and xfs.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

I guess there is no code fix required here (since there is no bug), however, a note can be added in innodb-log-block-size's documentation about this dmesg warning and/or about the crash https://bugs.launchpad.net/percona-server/+bug/1040803 and the workaround.

tags: added: doc
Revision history for this message
Stewart Smith (stewart) wrote :

Was innodb-log-block-size set to get this warning? if we get this warning by default then we may want to look at fixing something.

Revision history for this message
Jay Janssen (jay-janssen) wrote :

I don't believe innodb_log_block_size was being set, but this was so long ago that I have forgotten even which customer it was.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

@Stewart,

From what I have seen, innodb-log-block-size was not set during
this, and that is what caused this, since default
innodb-log-block-size is 512bytes whereas the filesystem block-size
is 4k and ext* filesystems print this warning to dmesg (xfs
doesn't print but may not be affected as much as well) due to
non-alignment.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-1266

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.