[posix] cinder schedules incremental backups on the wrong node

Bug #1952805 reported by Andrew Bogott
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Unassigned

Bug Description

I'm running 17.2 with the posix backend and two cinder-backup nodes.

When I run an incremental backup, the increment has a 50/50 chance of working: if it's scheduled on the same node that got the original full backup then it works, and if it gets scheduled on the other node with no full backup it fails:

2021-11-30 20:50:42.159 11905 ERROR oslo_messaging.rpc.server FileNotFoundError: [Errno 2] No such file or directory: '/srv/cinder-backups/28/77/2877b51f-6b0e-47ab-9582-bd5d12f6b7a3/volume_74bf4553-c92e-4fd5-88ef-33fb789ab07a_20211130204954_backup_2877b51f-6b0e-47ab-9582-bd5d12f6b7a3_sha256file

That file isn't found because it's on the other node, not this one.

summary: - cinder schedules incremental backups on the wrong node
+ [posix] cinder schedules incremental backups on the wrong node
tags: added: backup-service incremental posix schedules
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/861566

Changed in cinder:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/861566
Committed: https://opendev.org/openstack/cinder/commit/73c0d73aba1855edb774d69ae940a1b53dc975e0
Submitter: "Zuul (22348)"
Branch: master

commit 73c0d73aba1855edb774d69ae940a1b53dc975e0
Author: Andrew Bogott <email address hidden>
Date: Sun Oct 16 22:37:36 2022 -0500

    cinder-backup: use the same backup backend host for incremental backups

    Incremental backups only work if there's a previous backup to base
    it on. With the posix driver, this means there needs to be a previous
    backup on the same host where the incremental backup is created.

    This patch ensures that an incremental backup is scheduled on the
    host that that contains the base backup being used for the increment.
    Previously we were relying on luck for this and much of the time an
    incremental backup would fail due for want of a base backup.

    Closes-bug: 1952805
    Change-Id: Id239b4150b1c8e9f4bf32f2ef867fdffbe84f96d

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 22.0.0.0rc1

This issue was fixed in the openstack/cinder 22.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/cinder/+/916044

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (stable/zed)

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/cinder/+/916044
Reason: stable/zed branch of openstack/cinder is about to be deleted. To be able to do that, all open patches need to be abandoned. Please cherry pick the patch to unmaintained/zed if you want to further work on this patch.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.