Unable to perform a full backup

[Sam Schmitt]

Attempting to run a full backup of mariadb as described in the docs fails with the following

        "[00] FATAL ERROR: 2024-04-16 18:20:35 failed to execute query CREATE TABLE IF NOT EXISTS mysql.mariadb_backup_history(uuid VARCHAR(40) NOT NULL PRIMARY KEY,name VARCHAR(255) DEFAULT NULL,tool_name VARCHAR(255) DEFAULT NULL,tool_command TEXT DEFAULT NULL,tool_version VARCHAR(255) DEFAULT NULL,ibbackup_version VARCHAR(255) DEFAULT NULL,server_version VARCHAR(255) DEFAULT NULL,start_time TIMESTAMP NULL DEFAULT NULL,end_time TIMESTAMP NULL DEFAULT NULL,lock_time BIGINT UNSIGNED DEFAULT NULL,binlog_pos VARCHAR(128) DEFAULT NULL,innodb_from_lsn BIGINT UNSIGNED DEFAULT NULL,innodb_to_lsn BIGINT UNSIGNED DEFAULT NULL,partial ENUM('Y', 'N') DEFAULT NULL,incremental ENUM('Y', 'N') DEFAULT NULL,format ENUM('file', 'tar', 'xbstream') DEFAULT NULL,compressed ENUM('Y', 'N') DEFAULT NULL) CHARACTER SET utf8 ENGINE=innodb: CREATE command denied to user 'backup'@'infra-prod-controller-03' for table `mysql`.`mariadb_backup_history`"
    ],
    "stdout": "Taking a full backup\n",
    "stdout_lines": [
        "Taking a full backup"
    ]
}

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
infra-prod-controller-01 : ok=3 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
infra-prod-controller-02 : ok=2 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
infra-prod-controller-03 : ok=2 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0

This is right after enabling `enable_mariabackup: "yes"` in globals and running a `reconfigure -t mariadb`

It does seem that the backup file was put into the docker volume, but the backup task failes

[Jay Jahns (jayjahns)]

We are also observing this issue immediately after a 2023.2 upgrade.

[Grzegorz Koper (koperg)]

Please provide more information about environment.
kolla-ansible version, OS version , Docker version. If possible attach mariadb logs.

[Jay Jahns (jayjahns)]

I think the problem with this is that when upgrading from 2023.1 to 2023.2 or later, the mariadb version changes from 10.6 to 10.11.

The mariabackup command that gets executed uses the --history option. In 10.6 this information gets written to PERCONA_SCHEMA.xtrabackup_history. In 10.11 that has changed to mysql.mariadb_backup_history.

After our own investigation, it was determined that the variable mariadb_backup_database_schema needs to be changed from "PERCONA_SCHEMA" to "mysql" in order to ensure the correct database/tables have the correct permissions.

[Jay Jahns (jayjahns)]

This issue is documented at:
https://mariadb.com/kb/en/mariabackup-options/#-history

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/922729