Incremental backup extremly slow

[Klausenbusk]

Copy-paste https://jira.mariadb.org/browse/MDEV-14077 :
(The code is the same and the performance is the same, all research was done with mariabackup although)

Hello

Taking incremental backup like: `mariabackup --backup --incremental-lsn=xx` is way slower than taking a full backup, something like 20x+ as slow.

I did a bit of debugging, and noticed that it spent way longer in xtrabackup_copy_datafile, when doing inc backup (~ 0.02 vs ~ 0.001 (full backup)).

After a bit more trail-and-error, I found that it spent most of the time in a `memset` line. This line: https://github.com/MariaDB/server/blob/da05d0276a0569341c8bb41365dc7b05f9c4ddb7/extra/mariabackup/write_filt.cc#L80

I found that by adding the following print statement to the file: https://github.com/klausenbusk/server/commit/afcdb9128927e714ee0e3bcf5c14fea2f56855e0
Which gave me:
inc: 0.000001
inc: 0.000004
inc: 0.027387
inc: 0.027435

Ideally the code should only write delta file, when something has changed. At the moment it write a lot of "dummy" delta files, which only contain the header.

Regards Kristian Klausen

----

See also: https://www.percona.com/forums/questions-discussions/percona-xtrabackup/49859-incremental-backup-extremly-slow-mariadb-10-1-28-galera-cluster and https://jira.mariadb.org/browse/MDEV-14077

[Sergei Glushchenko (sergei.glushchenko)]

In order to understand whats going on we need some more information from you. What is exact command you were running. What is your my.cnf contents, more specifically what is innodb_page_size and how many parallel threads you are using. If page size is higher than default one, then you might be low on memory and go swapping. This line

https://github.com/percona/percona-xtrabackup/blob/80cb69baf7964ddb94e568d7dbcef173b16db054/storage/innobase/xtrabackup/src/write_filt.cc#L78

allocates ~1G when page size is 64k. Also, how many tables do you have?

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1481