Incorrect setting for --incremental-history-name in kolla/docker/mariadb/backup.sh

[Albert Braden]

We are running Train. I wrote a cron job to do mariadb daily full backups with "kolla-ansible -i <inventory> mariadb_backup" and hourly incrementals with "kolla-ansible -i <inventory> mariadb_backup --incremental"

What happened: Incrementals work correctly until UTC 0 and then fail until the next full runs

What you expected to happen: Incrementals should continue to work correctly after UTC 0.

Analysis: The problem is on line 24 of /opt/openstack/kolla/docker/mariadb/backup.sh:
        --incremental-history-name=$(date +%d-%m-%Y) \

After UTC 0 (date +%d-%m-%Y) refers to a full backup that does not exist yet.

How to reproduce it (minimal and precise): Setup daily full and hourly incremental cron jobs as described above.

OS: Centos 8.1.1911

Kernel: 4.18.0-147.8.1.el8_1.x86_64

Kolla-Ansible version: stable/train

[Radosław Piliszek (yoctozepto)]

So the issue is incrementals require the same-day full to have been run instead of using the latest full?

[Albert Braden (ozzzo)]

Yes. For now I'm working around it by running the full between 0 and 1 AM.

[Radosław Piliszek (yoctozepto)]

Ok, thanks for confirming.

[Albert Braden (ozzzo)]

The wishlisting of this bug makes me think that I'm doing something wrong. How can I setup my cron jobs so that incrementals do not fail from midnight until the next full?

[Radosław Piliszek (yoctozepto)]

Ah, sorry. It just means it's not strictly a bug but lack of an expected (by you and I guess some others) feature to allow an arbitrary previous full. The issue is there, the expectation is sane. Such bugs end up triaged-wishlist which is not a bad state, albeit not the highest priority obviously. We are currently at the end of Victoria cycle so pushing hard existing changes/patches which means it may take some time before it gets proper attention.

[Mark Goddard (mgoddard)]

What should the behaviour be here? Find the most recent history file, based on <something>? Timestamps are mutable, and the history filename does not allow for easy sorting (although it could be done).

[Albert Braden (ozzzo)]

I was thinking we would find the latest with -t: replace "$(date +%d-%m-%Y)” with “$HISTORY_NAME" where HISTORY_NAME=`ls -t $BACKUP_DIR/mysqlbackup*|head -1|cut -d- -f2-4`

If we are worried about timestamps changing then we would need a different solution but it's not obvious how we would find the latest if we can't depend on the timestamp. Maybe save it in a file in the /backup dir?

[Radosław Piliszek (yoctozepto)]

Saving extra metadata makes sense to me.
1) metadata exists -> use it
2) metadata does not exist -> legacy behaviour

Are you up to writing a patch for that?

[Albert Braden (ozzzo)]

I think so. I haven't written a patch before but I did follow the instructions to get setup as a contributor. Do I need to change both of these files?

https://github.com/openstack/kolla/blob/master/docker/mariadb/mariadb/backup.sh
https://github.com/openstack/kolla/blob/master/docker/mariadb/mariadb-server/backup.sh

[Radosław Piliszek (yoctozepto)]

Yes, for stable backports best to have both modified already. On the path forward we will only have mariadb-server image.

[Albert Braden (ozzzo)]

What's the convention on testing? Should I create a modified docker image and test it locally before submitting for review?

[Radosław Piliszek (yoctozepto)]

We have some CI scenario that exercises mariadb ops so we should be generally fine. Mark can confirm the scope (if it does not include incremental, it can be added easily).

[Mark Goddard (mgoddard)]

Sadly I was unable to get the backup working in CI. Here is the attempted patch: https://review.opendev.org/#/c/693332/

[Albert Braden (ozzzo)]

I guess I need to test it locally first. We don't have a dev environment yet but we will be building it "any day now."

[Radosław Piliszek (yoctozepto)]

You might also propose right away and someone else might be able to test it. Better upstream early.

[Albert Braden (ozzzo)]

https://review.opendev.org/#/c/758486/

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 11.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 10.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 9.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 12.0.0.0rc1 release candidate.