Built-in compression does inefficient I/O

Reported by Alexey Kopytov on 2013-01-02
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup
High
Alexey Kopytov
2.0
High
Alexey Kopytov
2.1
High
Alexey Kopytov

Bug Description

--compress does 4 unbuffered I/O operations per every 64K block of uncompressed data:

- it first writes an 8-byte compressed block header
- then it writes an 8-byte file offset
- then it writes a 4-byte checksum
- finally the compressed contents of the original 64K block is written

Each write is followed by a posix_fadvise(FADV_DONTNEED) call, which is of course very inefficient.

The fix is to buffer the output compressed stream.

There is also one here:

stream_one_file(File file, xb_wstream_file_t *xbfile)
{
 uchar buf[XBSTREAM_BUFFER_SIZE];
 size_t bytes;

 posix_fadvise(file, 0, 0, POSIX_FADV_SEQUENTIAL);

 while ((bytes = my_read(file, buf, XBSTREAM_BUFFER_SIZE,
    MYF(MY_WME))) > 0) {
  if (xb_stream_write_data(xbfile, buf, bytes)) {
   msg("%s: xb_stream_write_data() failed.\n",
       my_progname);
   return 1;
  }
  posix_fadvise(file, 0, 0, POSIX_FADV_DONTNEED);

 }
......
.....

here, it is doing fadvise POSIX_FADV_SEQ which doubles the
readahead but in the loop it discards it everytime. It should
either do POSIX_FADV_DONTNEED at the end (outside the loop) or
use ranges with POSIX_FADV_DONTNEED in the loop.

Hi Raghavendra,

On Thu, 03 Jan 2013 14:50:45 -0000, Raghavendra D Prabhu wrote:
> There is also one here:
>
>
> stream_one_file(File file, xb_wstream_file_t *xbfile)
> {
> uchar buf[XBSTREAM_BUFFER_SIZE];
> size_t bytes;
>
> posix_fadvise(file, 0, 0, POSIX_FADV_SEQUENTIAL);
>
> while ((bytes = my_read(file, buf, XBSTREAM_BUFFER_SIZE,
> MYF(MY_WME))) > 0) {
> if (xb_stream_write_data(xbfile, buf, bytes)) {
> msg("%s: xb_stream_write_data() failed.\n",
> my_progname);
> return 1;
> }
> posix_fadvise(file, 0, 0, POSIX_FADV_DONTNEED);
>
> }
> ......
> .....
>
> here, it is doing fadvise POSIX_FADV_SEQ which doubles the
> readahead but in the loop it discards it everytime. It should
> either do POSIX_FADV_DONTNEED at the end (outside the loop) or
> use ranges with POSIX_FADV_DONTNEED in the loop.
>

Agree, please report it as a separate bug, as it's not directly related
to this one. And the same is done by xtrabackup when reading the file IIRC.

I will report it as a separate bug; as you mentioned we may need to audit other locations of its usage.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers