kolla-ansible mariadb_backup fails due to xtrabackup being incompatible with mariadb 10.3

[Mark Goddard]

The kolla-ansible mariadb_backup command uses the xtrabackup image to take a full or incremental backup of the MariaDB database. However, xtrabackup is not compatible with MariaDB 10.3, which is used on CentOS in Stein and Ubuntu in Train. We should instead switch to mariabackup.

[Mark Goddard (mgoddard)]

Here is the documentation for the feature: https://docs.openstack.org/kolla-ansible/latest/admin/mariadb-backup-and-restore.html.

[Mark Goddard (mgoddard)]

Mariabackup docs: https://mariadb.com/kb/en/library/mariabackup-overview/

[Benjamin (tumbl3w33d)]

Something that can be used as a workaround for those affected - on the db master host you execute:

docker exec mariadb bash -c 'mariabackup --backup --target-dir /var/lib/mysql/backup/$(date +"%Y-%m-%d-%H-%M-%S") --user backup --password=<mariadb_backup_database_password>'

The full backup requires an empty directory as destination and creates it if it doesn't exist. The backup appears on your docker host in /var/lib/docker/volumes/mariadb/_data/backup.

You might consider that as base for rewriting the current backup logic. The old backup mechanism used a separate container and a dedicated named volume. This snippet, however, creates a backup folder in the mariadb volume itself. You might opt for adding a dedicated backup volume to the existing mariadb container.

Unfortunately, the kolla_docker.py doesn't allow a simple exec-like execution yet. That would make the change quite easy. Unless I miss the point of having it run in a separate container (which I tried and which caused too much trouble to follow that path).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681694
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=016c0f577c535def40dd32ddf41e6cb145a2f9b8
Submitter: Zuul
Branch: master

commit 016c0f577c535def40dd32ddf41e6cb145a2f9b8
Author: Marcin Juszkiewicz <email address hidden>
Date: Thu Sep 12 11:48:07 2019 +0200

    Drop Percona XtraBackup in favour of MariaDB Backup

    XtraBackup started failing to install on Ubuntu and does not support
    MariaDB 10.3 version.

    Related-Bug: #1843043

    Change-Id: I9cf62c8e03a02574d85e5349df1d23a51d4c99be

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/690589

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/690598

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690589
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=65b27d772be32738a8508ea8e6c07f74f229cdb7
Submitter: Zuul
Branch: master

commit 65b27d772be32738a8508ea8e6c07f74f229cdb7
Author: Mark Goddard <email address hidden>
Date: Wed Oct 23 14:08:35 2019 +0100

    Fix mariabackup arguments

    After switching from innobackupex to mariabackup [1] for database
    backups, the kolla_mariadb_backup.sh script was not modified to account
    for the difference in arguments between innobackupex and mariabackup.
    There is a compatibility mode, the documentation [2] for which covers
    some of the differences.

    The following have been changed:

    - Add explicit --backup argument, now required
    - Remove './' positional argument - not required with --stream
    - Remove --no-timestamp argument - only supported in innobackupex
      compatibility mode
    - Remove --incremental argument - implied by --incremental-history-name
    - Remove deprecated --compress argument, which requires qpress to be
      installed (it is not). The stream is now passed through gzip instead
      [3]

    [1] https://mariadb.com/kb/en/library/mariabackup-overview/
    [2] https://mariadb.com/kb/en/library/mariabackup-options/#-innobackupex
    [3] https://mariadb.com/kb/en/library/using-encryption-and-compression-tools-with-mariabackup/

    Change-Id: I67cff47cbf56570b8eaeb66092dd87c2769fc8a6
    Closes-Bug: #1843043

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/691290

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691290
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=88f374c914c4d6709a1edf407a912c8cf566038b
Submitter: Zuul
Branch: master

commit 88f374c914c4d6709a1edf407a912c8cf566038b
Author: Mark Goddard <email address hidden>
Date: Fri Oct 25 12:33:01 2019 +0100

    Set pipefail in kolla_mariadb_backup.sh

    Without this, a failure in mariabackup would be ignored due to the pipe
    to gzip.

    Change-Id: I84536b302991e8fb04d5fa67ea266bc428437ce1
    Related-Bug: #1843043

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/690598
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=7f47ddf7f4933357769a715a2f8b3fb2157a7dd0
Submitter: Zuul
Branch: master

commit 7f47ddf7f4933357769a715a2f8b3fb2157a7dd0
Author: Mark Goddard <email address hidden>
Date: Wed Oct 23 14:47:21 2019 +0000

    Use mariabackup for database backups

    Currently, Xtrabackup is used for database backups. However, Xtrabackup
    is not compatible with MariaDB 10.3. This change switches to use
    mariabackup [1], which is available in the mariadb image.

    The documented full and incremental restore procedures have been
    modified to use mariabackup, following [2] and [3].

    [1] https://mariadb.com/kb/en/library/mariabackup-overview/
    [2] https://mariadb.com/kb/en/library/full-backup-and-restore-with-mariabackup/
    [3] https://mariadb.com/kb/en/library/incremental-backup-and-restore-with-mariabackup/

    Change-Id: Id52b9b1f7b013277e401b1f6b8aed34473d2b2c4
    Closes-Bug: #1843043
    Depends-On: https://review.opendev.org/691290

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/692996

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/693023

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 9.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692996
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=25890a485b20f9530f769c23af841d6a50593b37
Submitter: Zuul
Branch: stable/stein

commit 25890a485b20f9530f769c23af841d6a50593b37
Author: Marcin Juszkiewicz <email address hidden>
Date: Thu Sep 12 11:48:07 2019 +0200

    Stein: support mariabackup for backups

    This is a combination of 3 commits from Train. Rather than dropping
    xtrabackup, we are keeping it in addition to adding support for
    backups using mariabackup.

    1. Drop Percona XtraBackup in favour of MariaDB Backup

    XtraBackup started failing to install on Ubuntu and does not support
    MariaDB 10.3 version.

    Related-Bug: #1843043

    I9cf62c8e03a02574d85e5349df1d23a51d4c99be

    2. Fix mariabackup arguments

    After switching from innobackupex to mariabackup [1] for database
    backups, the kolla_mariadb_backup.sh script was not modified to account
    for the difference in arguments between innobackupex and mariabackup.
    There is a compatibility mode, the documentation [2] for which covers
    some of the differences.

    The following have been changed:

    - Add explicit --backup argument, now required
    - Remove './' positional argument - not required with --stream
    - Remove --no-timestamp argument - only supported in innobackupex
      compatibility mode
    - Remove --incremental argument - implied by --incremental-history-name
    - Remove deprecated --compress argument, which requires qpress to be
      installed (it is not). The stream is now passed through gzip instead
      [3]

    [1] https://mariadb.com/kb/en/library/mariabackup-overview/
    [2] https://mariadb.com/kb/en/library/mariabackup-options/#-innobackupex
    [3] https://mariadb.com/kb/en/library/using-encryption-and-compression-tools-with-mariabackup/

    I67cff47cbf56570b8eaeb66092dd87c2769fc8a6
    Closes-Bug: #1843043
    (cherry picked from commit 65b27d772be32738a8508ea8e6c07f74f229cdb7)

    3. Set pipefail in kolla_mariadb_backup.sh

    Without this, a failure in mariabackup would be ignored due to the pipe
    to gzip.

    I84536b302991e8fb04d5fa67ea266bc428437ce1
    Related-Bug: #1843043

    Change-Id: Idf1aed905a7d9ad461dd6fe17ed199c8d60370e6

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/693023
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=de730a4cb4397d9914403578c832199fa4aa4367
Submitter: Zuul
Branch: stable/stein

commit de730a4cb4397d9914403578c832199fa4aa4367
Author: Mark Goddard <email address hidden>
Date: Wed Oct 23 14:47:21 2019 +0000

    Use mariabackup for database backups

    Currently, Xtrabackup is used for database backups. However, Xtrabackup
    is not compatible with MariaDB 10.3. This change switches to use
    mariabackup [1], which is available in the mariadb image.

    The documented full and incremental restore procedures have been
    modified to use mariabackup, following [2] and [3].

    [1] https://mariadb.com/kb/en/library/mariabackup-overview/
    [2] https://mariadb.com/kb/en/library/full-backup-and-restore-with-mariabackup/
    [3] https://mariadb.com/kb/en/library/incremental-backup-and-restore-with-mariabackup/

    Change-Id: Id52b9b1f7b013277e401b1f6b8aed34473d2b2c4
    Closes-Bug: #1843043
    Depends-On: https://review.opendev.org/692996
    (cherry picked from commit 7f47ddf7f4933357769a715a2f8b3fb2157a7dd0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 9.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/696129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696129
Committed: https://git.openstack.org/cgit/openstack/kayobe/commit/?id=e93bbd9ed568e98507c792244df58672ab3d84ef
Submitter: Zuul
Branch: master

commit e93bbd9ed568e98507c792244df58672ab3d84ef
Author: Mark Goddard <email address hidden>
Date: Tue Nov 26 16:24:46 2019 +0000

    Use mariabackup for database backups

    Kolla ansible switched its database backup support to mariabackup from
    xtrabackup due to incompatibilities. See
    https://bugs.launchpad.net/kolla/+bug/1843043 for details.

    Change-Id: Ib95771f09fd6d5e71a2af471de47f811e1cab88b
    Related-Bug: #1843043
    Story: 2006952
    Task: 37634

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/704624

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 8.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 8.0.2 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/704624
Committed: https://git.openstack.org/cgit/openstack/kayobe/commit/?id=93f1aaab4c08ca8b303bc03500a8b1d4f276dbba
Submitter: Zuul
Branch: stable/stein

commit 93f1aaab4c08ca8b303bc03500a8b1d4f276dbba
Author: Mark Goddard <email address hidden>
Date: Tue Nov 26 16:24:46 2019 +0000

    Use mariabackup for database backups

    Kolla ansible switched its database backup support to mariabackup from
    xtrabackup due to incompatibilities. See
    https://bugs.launchpad.net/kolla/+bug/1843043 for details.

    Change-Id: Ib95771f09fd6d5e71a2af471de47f811e1cab88b
    Related-Bug: #1843043
    Story: 2006952
    Task: 37634
    (cherry picked from commit e93bbd9ed568e98507c792244df58672ab3d84ef)