cinder-backup tries to take incremental backups of failed previous backups

[Andrew Bogott]

When instructed to make an incremental backup of volume X, cinder-backup tries to base that incremental backup on the most recent backup for volume X. If that most recent backup failed, the new incremental backup fails with an error about the base backup not being in the 'available' state.

This means that for scripted daily backups, one failed backup will break all future incremental backups until a full backup is requested.

A better behavior would be for cinder-backup to base an incremental backup on the most recent available backup, and simply ignore any more recent jobs with invalid states.

This seems like a safe change -- I strongly doubt that anyone is relying on cascading backup failure as part of their workflow.

[Sofia Enriquez (lsofia-enriquez)]

Greetings Andrew Bogott,

Thank you for reporting this issue. However, I need more information from you so we can reproduce the issue.
- Which backend are you using for the volumes and which one for the backups?
- How do you create the script for the daily backups? Could you share some steps with us so that the team can reproduce the issue?
- Do you have a c-bak LOGS, which shows the errors you see?

Thanks in advance
Sofia

[Andrew Bogott (andrewbogott)]

This is with the PosixBackupDriver backend. It has several issues which I'm chipping away at.

The backup job is invoked using the cinder python client:

                backupjob_rec = self.cinderclient.backups.create(
                    self.volume.id, name=new_backup_name, incremental=incremental
                )

You can see the error message that results from this issue in the attached patch; it's thrown here:

            if latest_backup['status'] != fields.BackupStatus.AVAILABLE:
                QUOTAS.rollback(context, reservations)
                msg = _('The parent backup must be available for '
                        'incremental backup.')

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/838009
Committed: https://opendev.org/openstack/cinder/commit/7c9d114e0130e82703418586d332560f70fc78de
Submitter: "Zuul (22348)"
Branch: master

commit 7c9d114e0130e82703418586d332560f70fc78de
Author: Andrew Bogott <email address hidden>
Date: Thu Apr 14 14:15:35 2022 -0500

    backups: Use the most recent available backup for incrementals

    Previously we chose the most recent backup and then crashed out
    if it wasn't available. That behavior meant that once
    any backup has an error state incremental backups fail
    forever after.

    Closes-Bug: #1974092
    Change-Id: Ie5001c5ac6ccd7d033e7a0b97ef254570a9f75a7

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/cinder/+/863273

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 22.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/cinder/+/863273
Reason: stable/zed branch of openstack/cinder is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/zed if you want to further work on this patch.