cinder backup-create with ceph fails for incremental backup if latest incremental backup is deleted

[Chaynika Saikia]

If the latest incremental backup for a volume is deleted, then the next time an incremental backup is done for that volume, it fails and instead does a full backup saying that the snapshot does not exist.

9203 Jul 07 15:49:26 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] returning [{'timestamp': u'1499456874.44', 'backup_id': u'3c4d0b03-1aee-4a3b-8386-0050befd0ffc', 'na me': u'backup.3c4d0b03-1aee-4a3b-8386-0050befd0ffc.snap.1499456874.44'}] {{(pid=23880) get_backup_snaps /opt/stack/cinder/cind er/backup/drivers/ceph.py:794}}
9204 Jul 07 15:49:26 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Using --from-snap 'backup.3c4d0b03-1aee-4a3b-8386-0050befd0ffc.snap.1499456874.44' for incremental b ackup of volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/ceph.p y:636}}
9205 Jul 07 15:49:26 localhost.localdomain cinder-backup[23880]: INFO cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77- 78d2280621ed admin None] Snapshot='backup.3c4d0b03-1aee-4a3b-8386-0050befd0ffc.snap.1499456874.44' does not exist in base imag e='volume-ebf16d3c-deb8-4168-9dfd-50d386860d42.backup.base' - aborting incremental backup
9206 Jul 07 15:49:26 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Forcing full backup of volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) backup /opt/stack/ cinder/cinder/backup/drivers/ceph.py:905}}
9207 Jul 07 15:49:26 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Creating backup base image='volume-ebf16d3c-deb8-4168-9dfd-50d386860d42.backup.7f27f402-31cb-49fc-ad 75-a52d6a5252aa' for volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) _full_backup /opt/stack/cinder/cinder/backup/d rivers/ceph.py:736}}
9208 Jul 07 15:49:27 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Copying data from volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) _full_backup /opt/stack /cinder/cinder/backup/drivers/ceph.py:745}}

How reproducible:
Everytime

Steps to Reproduce:
1. cinder backup-create 7f27f402-31cb-49fc-ad75-a52d6a5252aa (should create full backup)
2. cinder backup-create 7f27f402-31cb-49fc-ad75-a52d6a5252aa --force --incremental (should create incremental backup)
3. cinder backup-delete 3c4d0b03-1aee-4a3b-8386-0050befd0ffc (the backup ID of the backup created in step 2)
3. cinder backup-create 7f27f402-31cb-49fc-ad75-a52d6a5252aa --force --incremental (should create incremental because it still should have other snapshots to create an incremental backup from but instead creates a full backup without any warning to the user)

[Gorka Eguileor (gorka)]

Sounds about right, the fix could go like this:

- We need to keep all source volume's backup snapshots (I believe we are currently only keeping the last one)
- Whenever we delete a backup we delete its corresponding source volume snapshot.

I believe that would not only solve this problem but also other potential problems that would appear when you start deleting backups that are not the last one.

[Chaynika Saikia (csaikia)]

Thank you! I have the same opinion as well.

[zheng yin (yin-zheng)]

I also try to resolve this issue, I keep all source volume's backup snapshots, but I don't know how to delete source volume snapshot when I delete backup.

[Chaynika Saikia (csaikia)]

I am already making changes to deal with the deletion part. Thank you!

[zheng yin (yin-zheng)]

hi, csaikia

    I have resolved this issue and deleted source volume snap successfully. thanks ;)

[Chaynika Saikia (csaikia)]

Thanks for letting me know!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/486678

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/486678
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=bc9ab142da919087c71525978f960115ff0259b9
Submitter: Jenkins
Branch: master

commit bc9ab142da919087c71525978f960115ff0259b9
Author: Chaynika Saikia <email address hidden>
Date: Mon Jul 24 11:48:00 2017 -0400

    cinder incremental backup with ceph fails if last one deleted

    If the last incremental backup is deleted then the backup snapshot
    is deleted but the source volume snapshot remains. Due to mismatch
    between the source volume snapshot and backup snapshot, there is an
    error when "rbd export-diff" and "rbd import-diff" is done. Hence,
    full backup happens.

    To fix this issue, all the previous source volume snapshots are
    preserved. When the last incremental backup is deleted, then its
    backup snapshot is deleted. But because all the source volume snaps
    exist, thus it can now update --from-snap for "rbd export-diff" to
    be the most recent source volume snapshot for which there exists a
    backup snapshot as well. This is how the incremental backup operation
    will go successful.

    For a user who upgrades to the new code, if the last backup snapshot
    and source volume snapshot are in sync, i.e. both of them exist, then
    the next create backup operation will continue as an incremental one.
    However, if there is no source volume snapshot for the last existing
    incremental backup (before the code was upgraded), then a full backup
    happens. After this full backup, all the next backups are incremental
    while all their corresponding source volume snapshots are also preserved.
    In this way, after a user upgrades, if he deletes the last incremental
    backup, the next create backup operation will be successful and be
    incremental.

    Since all the backup snapshots exist for all the backups, if the user
    wants to restore a backup, which was created before the code was upgraded,
    the user will not lose any data since the same base backup is used.

    Change-Id: Ia9c29bb720152d42bec273202fa49ca4b6a41ce2
    Closes-Bug: #1703011

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/538298

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/538298
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=0e4e1483e1e1e90ea8c9c45d967f6e772e15df97
Submitter: Zuul
Branch: stable/pike

commit 0e4e1483e1e1e90ea8c9c45d967f6e772e15df97
Author: Chaynika Saikia <email address hidden>
Date: Mon Jul 24 11:48:00 2017 -0400

    cinder incremental backup with ceph fails if last one deleted

    If the last incremental backup is deleted then the backup snapshot
    is deleted but the source volume snapshot remains. Due to mismatch
    between the source volume snapshot and backup snapshot, there is an
    error when "rbd export-diff" and "rbd import-diff" is done. Hence,
    full backup happens.

    To fix this issue, all the previous source volume snapshots are
    preserved. When the last incremental backup is deleted, then its
    backup snapshot is deleted. But because all the source volume snaps
    exist, thus it can now update --from-snap for "rbd export-diff" to
    be the most recent source volume snapshot for which there exists a
    backup snapshot as well. This is how the incremental backup operation
    will go successful.

    For a user who upgrades to the new code, if the last backup snapshot
    and source volume snapshot are in sync, i.e. both of them exist, then
    the next create backup operation will continue as an incremental one.
    However, if there is no source volume snapshot for the last existing
    incremental backup (before the code was upgraded), then a full backup
    happens. After this full backup, all the next backups are incremental
    while all their corresponding source volume snapshots are also preserved.
    In this way, after a user upgrades, if he deletes the last incremental
    backup, the next create backup operation will be successful and be
    incremental.

    Since all the backup snapshots exist for all the backups, if the user
    wants to restore a backup, which was created before the code was upgraded,
    the user will not lose any data since the same base backup is used.

    Change-Id: Ia9c29bb720152d42bec273202fa49ca4b6a41ce2
    Closes-Bug: #1703011
    (cherry picked from commit bc9ab142da919087c71525978f960115ff0259b9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brian Rosmaita (<email address hidden>) on branch: stable/ocata
Review: https://review.opendev.org/649652
Reason: This has sat for 7 months with no activity and ocata is set to go EOL next week.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/912379

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Liron Kuchlani <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/912379