Incremental restore in XB 2.4 requires 1+GB of RAM

Bug #1582456 reported by Andrey Galkin
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Status tracked in 2.4
2.4
Fix Released
Medium
Sergei Glushchenko

Bug Description

Steps to reproduce:
-------------------
S1. Create a low memory VM e.g. <1GB (alternatively, limit with ulimit/cgroup)
S2. Make sure swap is off
S3. Create full backup + incremental backup
S4. Try to restore
S5. See a failure on incremental:
...
Applying /db/mysql_mysrv1/data.incremental/ibdata1.delta to ./ibdata1...
InnoDB: Retry attempts for reading partial data failed.
InnoDB: Tried to read 16384 bytes at offset 0 was only able to read0
InnoDB: Operating system error number 14 in a file operation.
InnoDB: Error number 14 means 'Bad address'
...

What is expected:
-----------------
W1. Make sure out-of-memory error is properly reported (check malloc() return)
W2. Use smaller buffer of 64MB as before or even less, do not use UNIV_PAGE_SIZE_MAX in calculations
W3. Obey --use-memory as much as possible

Debug results:
--------------
D1. xtrabackup.cc:5844 (https://github.com/percona/percona-xtrabackup/blob/2.4/storage/innobase/xtrabackup/src/xtrabackup.cc#L5844)

...
 incremental_buffer_base = static_cast<byte *>
  (ut_malloc_nokey((UNIV_PAGE_SIZE_MAX / 4 + 1) *
      UNIV_PAGE_SIZE_MAX));
 incremental_buffer = static_cast<byte *>
  (ut_align(incremental_buffer_base,
     UNIV_PAGE_SIZE_MAX));
...

D2. incremental_buffer_base & incremental_buffer are NOT checked other than for ut_free()

D3. UNIV_PAGE_SIZE_MAX has changed due to change of UNIV_PAGE_SIZE_SHIFT_MAX from 14 to 16 in rev 717b4b2e

#define UNIV_PAGE_SIZE_MAX (1 << UNIV_PAGE_SIZE_SHIFT_MAX)

D4. Consequences memory usage for the buffer raised from ~64MB to ~1GB

Revision history for this message
Andrey Galkin (andvgal) wrote :

It may not look like important for production. However, it has implications for test automation activities involving XB.

Please consider returning memory usage back to fixed 64MB for incremental buffer at least.

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :

Hello Andrey,

MySQL starting with 5.7.6 supports page size up to 64K, so
`UNIV_PAGE_SIZE_MAX' is the upstream change. It would be not trivial to
change amount of memory needed for `incremental_buffer' without changing
`.delta' file format.

So, for the larger fix I started to draft this blueprint:

https://blueprints.launchpad.net/percona-xtrabackup/+spec/incremental-memory-limit

What I am going to do now is to replace `UNIV_PAGE_SIZE_MAX' with
`page_size', so for default page size we will still use 64M buffer.

Revision history for this message
Andrey Galkin (andvgal) wrote :

Hi Sergei,

Yes, I understood it was upstream change. Perhaps, you may want to use mmap'ed file to workaround the case. If there is enough memory then OS would operate in plain RAM with minimal overhead as before. Low memory systems would slow down, but still continue to operate instead of hard failing.

Thanks for looking into that.

Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-753

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.