Incremental SST

[Raghavendra D Prabhu]

A full SST is hard on the systems everytime there is an issue or
when needed.

On the other hand, if every node has a base backup (present when
the node was initialized), it can use (and should be able to) the
LSN of the base backup and fetch only the pages after that. This
should help even more now that XtraDB/Xtrabackup support changed
pages tracking for faster incremental backups.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Note, that this requires that innobackupex on donor be aware of
the LSN of the joiner. So, the script needs to listen on donor
as well (for a short while) to get the LSN from joiner.

[Alex Yurchenko (ayurchen)]

Raghu, note that joiner can pass required info as part of the address string. Look for

echo "ready ${ADDR}/${MODULE}"

string (currently it is in wait_for_nc() function, which is not right). Anyway, you can append anything there as long as the donor part of the script can parse it.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Ack, thanks. That should make it easier to setup.

[Raghavendra D Prabhu (raghavendra-prabhu)]

One more thing, this needs to distinguish between when full SST
is needed (there is nothing in datadir and no base backups either) and when incremental SST
is needed (recovery from last lsn).

[Raghavendra D Prabhu (raghavendra-prabhu)]

This will be applicable in
a) When there is an existing datadir and node has been down for
sometime (so out of date, requiring a SST).

b) When there is a prepared backup lying on the host (used to
create the node) and that be reused.

The joiner sends LSN to donor the way uuid:seqno are sent. To obtain
LSN, currently, mysqld needs to be run with wsrep-recover or so. Note,
that this applies only when using pre-existing datadir, otherwise
xtrabackup_checkpoints file can be reused. The donor then creates the
incremental stream and sends to joiner. The joiner needs to apply this
delta to the pre-existing backup and use it. Also, note that, the
pre-existing backup on joiner needs to be apply-log-only prepared.

[Raghavendra D Prabhu (raghavendra-prabhu)]

There are a couple of issues here:

a) LSN needs to be recovered by the SST script. This is possible
, since SST script and InnoDB are not up at same time, another
instance can be spawned with --wsrep-recover. However, it would
be better if xtrabackup can made to print this (like it does with
--stats). This has another advantage that it won't require
wsrep-recover stage during normal operations as well.

b) Currently, Xtrabackup requires xtrabackup_checkpoints to be
present in datadir. This is fine, however, if and when it becomes
out of date during normal operation, the incremental apply fails.
It would be better if Xtrabackup could take --incremental-lsn (as
to_lsn in xtrabackup_checkpoints) during prepare as well.

c) Rsync SST: This has to be incremental by its nature, assuming -W is
not used, since rsync does delta transfer. Sure, it will transfer a
bit more (since it doesn't know innodb file structure), however, for
subsequent SSTs it should only transfer only the deltas.

For Xtrabackup SST, a) and b) need to be satisfied. a) can be
worked around. However, b) requires some manual adjustment or
xtrabackup should support this.

Note, this is only in the context of existing data dir. If there
is a separate prepared backup (with xtrabackup_checkpoints), it can be used for incremental SST
just fine.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Raised https://bugs.launchpad.net/percona-xtrabackup/+bug/1200282 for the LSN with Xtrabackup.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Need to check whether full backup donor and incremental's donor match.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Currently any hostname information is not passed by xtrabackup, hence, it may not be possible to currently ensure #8, it is upto the admin who is running this to ensure this (and enforce with wsrep_sst_donor).

[Raghavendra D Prabhu (raghavendra-prabhu)]

To reiterate #9, unless xtrabackup supports incremental backups
with gtid (than LSN), automation of incremental SST is not possible since it
is possible that donor of full SST and that of incremental SST
are not the same. (It is possible to circumvent with hostnames in
SST metadata, but this will fail later on).

Also, even if lsn can be obtained during wsrep-recover, the
problem remains that of prepare wherein xtrabackup_checkpoints is
required since xtrabackup cannot directly get it from datadir.

[Raghavendra D Prabhu (raghavendra-prabhu)]

So, as discussed, this is how it will be done:

a) Run --wsrep-recover and get LSN (like how we get uuid:seqno)

b) Provide that LSN to mysqld (like how we provide
--wsrep-start-position but with a different parameter)

c) wsrep_sst_xtrabackup makes use of that LSN and if incremental
is enabled and SST is asked for, it will send that LSN in 'ready'
(It does this already except using the LSN of previous steps)
along with hostname (obtained from previous full SST, details
below)

d) Donor gets the LSN co-ordinates

    i) In the first stage of SST, it sends the hostname if this
    is a non-incremental backup.

    ii) It checks the hostname obtained from joiner, if it
    matches its own, then it does xtrabackup incremental backup.

    iii) If not, it does a full backup but also sends 'full' in
    the magic file sent in i).

e) Joiner knows if it is receiving full or incremental from
    the file it received earlier.

f) In case of full, it is as before.

g) In case of incremental, it is streamed to a temporary
directory wherein it is prepared (while at the same time,
creating a fake xtrabackup_checkpoints file in $datadir).

h) Hostname is maintained in xtrabackup_ist file in $datadir.

All this will run automatically without user intervention, incremental=1 under [sst] will imply that do incremental if feasible.

Feasible here implies, if the hostnames of donors who provided
full and incremental don't match, it will be a full SST.

wsrep_sst_donor can be used to prefer donors in this case..

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1165