Incorrect setting for --incremental-history-name in kolla/docker/mariadb/backup.sh

Bug #1897948 reported by Albert Braden on 2020-09-30
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla-ansible
Wishlist
Unassigned

Bug Description

We are running Train. I wrote a cron job to do mariadb daily full backups with "kolla-ansible -i <inventory> mariadb_backup" and hourly incrementals with "kolla-ansible -i <inventory> mariadb_backup --incremental"

What happened: Incrementals work correctly until UTC 0 and then fail until the next full runs

What you expected to happen: Incrementals should continue to work correctly after UTC 0.

Analysis: The problem is on line 24 of /opt/openstack/kolla/docker/mariadb/backup.sh:
        --incremental-history-name=$(date +%d-%m-%Y) \

After UTC 0 (date +%d-%m-%Y) refers to a full backup that does not exist yet.

How to reproduce it (minimal and precise): Setup daily full and hourly incremental cron jobs as described above.

OS: Centos 8.1.1911

Kernel: 4.18.0-147.8.1.el8_1.x86_64

Kolla-Ansible version: stable/train

Radosław Piliszek (yoctozepto) wrote :

So the issue is incrementals require the same-day full to have been run instead of using the latest full?

Albert Braden (ozzzo) wrote :

Yes. For now I'm working around it by running the full between 0 and 1 AM.

Radosław Piliszek (yoctozepto) wrote :

Ok, thanks for confirming.

Changed in kolla-ansible:
importance: Undecided → Wishlist
status: New → Triaged
Albert Braden (ozzzo) wrote :

The wishlisting of this bug makes me think that I'm doing something wrong. How can I setup my cron jobs so that incrementals do not fail from midnight until the next full?

Radosław Piliszek (yoctozepto) wrote :

Ah, sorry. It just means it's not strictly a bug but lack of an expected (by you and I guess some others) feature to allow an arbitrary previous full. The issue is there, the expectation is sane. Such bugs end up triaged-wishlist which is not a bad state, albeit not the highest priority obviously. We are currently at the end of Victoria cycle so pushing hard existing changes/patches which means it may take some time before it gets proper attention.

Mark Goddard (mgoddard) wrote :

What should the behaviour be here? Find the most recent history file, based on <something>? Timestamps are mutable, and the history filename does not allow for easy sorting (although it could be done).

Albert Braden (ozzzo) wrote :

I was thinking we would find the latest with -t: replace "$(date +%d-%m-%Y)” with “$HISTORY_NAME" where HISTORY_NAME=`ls -t $BACKUP_DIR/mysqlbackup*|head -1|cut -d- -f2-4`

If we are worried about timestamps changing then we would need a different solution but it's not obvious how we would find the latest if we can't depend on the timestamp. Maybe save it in a file in the /backup dir?

Radosław Piliszek (yoctozepto) wrote :

Saving extra metadata makes sense to me.
1) metadata exists -> use it
2) metadata does not exist -> legacy behaviour

Are you up to writing a patch for that?

Albert Braden (ozzzo) wrote :

I think so. I haven't written a patch before but I did follow the instructions to get setup as a contributor. Do I need to change both of these files?

https://github.com/openstack/kolla/blob/master/docker/mariadb/mariadb/backup.sh
https://github.com/openstack/kolla/blob/master/docker/mariadb/mariadb-server/backup.sh

Yes, for stable backports best to have both modified already. On the path forward we will only have mariadb-server image.

Albert Braden (ozzzo) wrote :

What's the convention on testing? Should I create a modified docker image and test it locally before submitting for review?

We have some CI scenario that exercises mariadb ops so we should be generally fine. Mark can confirm the scope (if it does not include incremental, it can be added easily).

Mark Goddard (mgoddard) wrote :

Sadly I was unable to get the backup working in CI. Here is the attempted patch: https://review.opendev.org/#/c/693332/

Albert Braden (ozzzo) wrote :

I guess I need to test it locally first. We don't have a dev environment yet but we will be building it "any day now."

You might also propose right away and someone else might be able to test it. Better upstream early.

This issue was fixed in the openstack/kolla 11.0.0.0rc2 release candidate.

This issue was fixed in the openstack/kolla 10.2.0 release.

This issue was fixed in the openstack/kolla 9.3.0 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers