incremental's xtrabackup_logfile is not reusable after applying log

[Valentine Gostev]

After we apply log from incremental changes - xtrabackup_logfile's content changes and data from incremental dir cannot be used for prepare again.

It could be a nice feature if the contents is not changed - it will allow to use incremental-dir for applying logs t full backup several times.

[Stewart Smith (stewart)]

I'm not entirely sure what the implications of this are.... could you expand?

[Vadim Tkachenko (vadim-tk)]

The impact is that the same incremental can't be applied twice to the same full backup.

[Stewart Smith (stewart)]

So this is just a feature request to be able to apply the incremental backups to the same full backup repeatedly (essentially making it a no-op as they're already applied)?.

[Vadim Tkachenko (vadim-tk)]

I am not sure about no-op, but just being able to repeat the same procedure.

On Fri, May 20, 2011 at 6:14 PM, Stewart Smith <email address hidden> wrote:
> So this is just a feature request to be able to apply the incremental
> backups to the same full backup repeatedly (essentially making it a no-
> op as they're already applied)?.
>
> --
> You received this bug notification because you are a member of Percona
> developers, which is the registrant for Percona XtraBackup.
> https://bugs.launchpad.net/bugs/782850
>
> Title:
>  incremental's xtrabackup_logfile is not reusable after applying log
>
> Status in Percona XtraBackup:
>  Triaged
>
> Bug description:
>  After we apply log from incremental changes - xtrabackup_logfile's
>  content changes and data from incremental dir cannot be used for
>  prepare again.
>
>  It could be a nice feature if the contents is not changed - it will
>  allow to use incremental-dir for applying logs t full backup several
>  times.
>

--
Vadim Tkachenko, CTO, Percona Inc.
Phone [phone],  Skype: vadimtk153
Schedule meeting: http://tungle.me/VadimTkachenko

Flat-rate 24x7 support for MySQL <http://percona.com/mysql-support>

[Peter Zaitsev (pz-percona)]

Stewart,

I think the bug description is consuming making it as non issue, While I
think it is really serious design issue.

The problem is the incremental backup itself is modified and can't be reused
again for other restore. This requires
extra copy operation which is not efficient.

We do need to do the copy for non-incremental backup to put it back in place
but it would be good we could essentially
apply incremental backup from read only NFS volume, where they can be
re-used to restore data to multiple slaves for example

Vadim is my understanding correct ?

On Fri, May 20, 2011 at 6:14 PM, Stewart Smith <email address hidden>wrote:

> So this is just a feature request to be able to apply the incremental
> backups to the same full backup repeatedly (essentially making it a no-
> op as they're already applied)?.
>
> --
> You received this bug notification because you are a member of Percona
> developers, which is the registrant for Percona XtraBackup.
> https://bugs.launchpad.net/bugs/782850
>
> Title:
> incremental's xtrabackup_logfile is not reusable after applying log
>
> Status in Percona XtraBackup:
> Triaged
>
> Bug description:
> After we apply log from incremental changes - xtrabackup_logfile's
> content changes and data from incremental dir cannot be used for
> prepare again.
>
> It could be a nice feature if the contents is not changed - it will
> allow to use incremental-dir for applying logs t full backup several
> times.
>

--
Peter Zaitsev, CEO, Percona Inc.
Tel: [phone] ext 501 Skype: peter_zaitsev
24/7 Emergency Line [phone] ext 911

Percona Live MySQL Conference comes to NYC
http://www.percona.com/live/nyc-2011/

[Vadim Tkachenko (vadim-tk)]

Peter,

Your understanding is correct,
however one may ask, why it is needed to apply incremental backup twice ?

In regular workflow - you just apply incremental to full and may delete and
forget about it.

From that point of view this issue does not appear as high priority for me,
but it would be "nice to have"

On Fri, May 20, 2011 at 6:50 PM, Peter Zaitsev <email address hidden> wrote:
> Stewart,
>
> I think the bug description is consuming making it as non issue, While I
> think it is really serious design issue.
>
> The problem is the incremental backup itself is modified and can't be reused
> again for other restore.  This requires
> extra copy operation which is not efficient.
>
> We do need to do the copy for non-incremental backup to put it back in place
> but it would be good we could essentially
> apply incremental backup from read only NFS volume, where they can be
> re-used  to restore data to multiple slaves for example
>
> Vadim is my understanding correct ?
>
> On Fri, May 20, 2011 at 6:14 PM, Stewart Smith
> <email address hidden>wrote:
>
>> So this is just a feature request to be able to apply the incremental
>> backups to the same full backup repeatedly (essentially making it a no-
>> op as they're already applied)?.
>>
>> --
>> You received this bug notification because you are a member of Percona
>> developers, which is the registrant for Percona XtraBackup.
>> https://bugs.launchpad.net/bugs/782850
>>
>> Title:
>>  incremental's xtrabackup_logfile is not reusable after applying log
>>
>> Status in Percona XtraBackup:
>>   Triaged
>>
>> Bug description:
>>  After we apply log from incremental changes - xtrabackup_logfile's
>>  content changes and data from incremental dir cannot be used for
>>  prepare again.
>>
>>  It could be a nice feature if the contents is not changed - it will
>>  allow to use incremental-dir for applying logs t full backup several
>>  times.
>>
>
>
> --
> Peter Zaitsev, CEO, Percona Inc.
> Tel: [phone] ext 501   Skype:  peter_zaitsev
> 24/7 Emergency Line [phone] ext 911
>
> Percona Live MySQL Conference comes to NYC
> http://www.percona.com/live/nyc-2011/
>
> --
> You received this bug notification because you are a member of Percona
> developers, which is the registrant for Percona XtraBackup.
> https://bugs.launchpad.net/bugs/782850
>
> Title:
>  incremental's xtrabackup_logfile is not reusable after applying log
>
> Status in Percona XtraBackup:
>  Triaged
>
> Bug description:
>  After we apply log from incremental changes - xtrabackup_logfile's
>  content changes and data from incremental dir cannot be used for
>  prepare again.
>
>  It could be a nice feature if the contents is not changed - it will
>  allow to use incremental-dir for applying logs t full backup several
>  times.
>

--
Vadim Tkachenko, CTO, Percona Inc.
Phone [phone],  Skype: vadimtk153
Schedule meeting: http://tungle.me/VadimTkachenko

Flat-rate 24x7 support for MySQL <http://percona.com/mysql-support>

[Vadim Tkachenko (vadim-tk)]

Also,

If I remember correctly,
xtrabackup will modify xtrabackup_checkpoints file in full backup,
so it will not allow to apply the same incremental second time, so it will look like Stewart said - "no-op" operation.

[Stewart Smith (stewart)]

Wishlist seems accurate Importance then.

[Alexey Kopytov (akopytov)]

See also bug #908767.

[Imri Zvik (imrizvik)]

This should be at least clearly documented.
A use case where it might interfere is what XBM (https://code.google.com/p/xtrabackup-manager/) calls "continuous Incremental backup" - It creates a seed backup (i.e. full backup) and then keep backup incremental ones.
When the chain reaches a certain size, it will apply the oldest snapshot to the seed, and delete it.

The problem is that XBM has another feature called "materialized copy" where after *each* backup, it applies that *same* snapshot to *another* copy of the seed, therefore always maintaining a prepared backup copy, ready to restore.

The process goes something like this:

1. Create a full backup under /backups/1
2. Create a symlink called /backups/m1 which points to /backups/1
3. Create an incremental backup under /backups/2
4. Copy /backups/1 to /backups/m2, and remove the /backups/m1 symlink
5. Apply /backups/2 to /backups/m2
6. Create an incremental backup under /backups/3
7. Apply /backups/3 to /backups/m2
8. Repeat steps 6 and 7 until you reach the maximum allowed snapshots:
8.1 Apply the oldest snapshot (/backups/2) to the seed ( /backups/1) and delete the snapshot ( /backups/2).

Step 8.1 will produce bad results as /backups/2 was already used in step 5

In my case, it produced an ibdata file of 1.2TB in size, while the actual ibdata file is 180MB ~

[Jervin R (revin)]

Here's one reason why we need this - https://bugs.launchpad.net/percona-xtrabackup/+bug/1264432

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-102