cinder backup is_incremental flag is True even if incremental backup fails

[Chaynika Saikia]

Description of problem:

cinder incremental backup sets is_incremental as True even if it does a forceful full backup

[csaikia@localhost ~]$ cinder backup-create ebf16d3c-deb8-4168-9dfd-50d386860d42 --force --incremental
+-----------+--------------------------------------+
| Property | Value |
+-----------+--------------------------------------+
| id | 7f27f402-31cb-49fc-ad75-a52d6a5252aa |
| name | None |
| volume_id | ebf16d3c-deb8-4168-9dfd-50d386860d42 |
+-----------+--------------------------------------+

$ cinder backup-show 7f27f402-31cb-49fc-ad75-a52d6a5252aa
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| availability_zone | nova |
| container | backups |
| created_at | 2017-07-07T19:49:26.000000 |
| data_timestamp | 2017-07-07T19:49:26.000000 |
| description | None |
| fail_reason | None |
| has_dependent_backups | False |
| id | 7f27f402-31cb-49fc-ad75-a52d6a5252aa |
| is_incremental | True |
| name | None |
| object_count | 0 |
| size | 1 |
| snapshot_id | None |
| status | available |
| updated_at | 2017-07-07T19:50:06.000000 |
| volume_id | ebf16d3c-deb8-4168-9dfd-50d386860d42 |
+-----------------------+--------------------------------------+

While in the logs, one can see that there is a full backup done for that backup ID.

9206 Jul 07 15:49:26 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Forcing full backup of volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) backup /opt/stack/ cinder/cinder/backup/drivers/ceph.py:905}}
9207 Jul 07 15:49:26 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Creating backup base image='volume-ebf16d3c-deb8-4168-9dfd-50d386860d42.backup.7f27f402-31cb-49fc-ad 75-a52d6a5252aa' for volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) _full_backup /opt/stack/cinder/cinder/backup/d rivers/ceph.py:736}}
9208 Jul 07 15:49:27 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Copying data from volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) _full_backup /opt/stack /cinder/cinder/backup/drivers/ceph.py:745}}
9209 Jul 07 15:49:27 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Transferring data between 'volume-ebf16d3c-deb8-4168-9dfd-50d386860d42' and 'volume-ebf16d3c-deb8-41 68-9dfd-50d386860d42.backup.7f27f402-31cb-49fc-ad75-a52d6a5252aa' {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup /drivers/ceph.py:338}}
9210 Jul 07 15:49:27 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] 8 chunks of 134217728 bytes to be transferred {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/ backup/drivers/ceph.py:342}}
9211 Jul 07 15:49:27 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Transferred chunk 1 of 8 (286576K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dr ivers/ceph.py:364}}
9212 Jul 07 15:49:32 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Transferred chunk 2 of 8 (24954K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri vers/ceph.py:364}}
9213 Jul 07 15:49:37 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Transferred chunk 3 of 8 (25152K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri vers/ceph.py:364}}
9214 Jul 07 15:49:43 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Transferred chunk 4 of 8 (25191K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri vers/ceph.py:364}}
9215 Jul 07 15:49:48 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Transferred chunk 5 of 8 (25293K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri vers/ceph.py:364}}
9216 Jul 07 15:49:53 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Transferred chunk 6 of 8 (24709K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri vers/ceph.py:364}}
9217 Jul 07 15:49:58 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Transferred chunk 7 of 8 (25041K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri vers/ceph.py:364}}
9218 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Transferred chunk 8 of 8 (24935K/s) {{(pid=23880) _transfer_data /opt/stack/cinder/cinder/backup/dri vers/ceph.py:364}}
9219 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22 80621ed admin None] Getting metadata type 'volume-base-metadata' {{(pid=23880) _save_vol_base_meta /opt/stack/cinder/cinder/ba ckup/driver.py:80}}
9220 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22 80621ed admin None] Completed fetching metadata type 'volume-base-metadata' {{(pid=23880) _save_vol_base_meta /opt/stack/cinde r/cinder/backup/driver.py:97}}
9221 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22 80621ed admin None] Getting metadata type 'volume-metadata' {{(pid=23880) _save_vol_meta /opt/stack/cinder/cinder/backup/drive r.py:108}}
9222 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22 80621ed admin None] No metadata type 'volume-metadata' available {{(pid=23880) _save_vol_meta /opt/stack/cinder/cinder/backup/ driver.py:122}}
9223 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22 80621ed admin None] Getting metadata type 'volume-glance-metadata' {{(pid=23880) _save_vol_glance_meta /opt/stack/cinder/cinde r/backup/driver.py:131}}
9224 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.driver [None req-e8adeca3-4dc7-427f-8a77-78d22 80621ed admin None] No metadata type 'volume-glance-metadata' available {{(pid=23880) _save_vol_glance_meta /opt/stack/cinder/ cinder/backup/driver.py:146}}
9225 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Backing up metadata for volume ebf16d3c-deb8-4168-9dfd-50d386860d42. {{(pid=23880) _backup_metadata /opt/stack/cinder/cinder/backup/drivers/ceph.py:871}}
9226 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG cinder.backup.drivers.ceph [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Backup '7f27f402-31cb-49fc-ad75-a52d6a5252aa' of volume ebf16d3c-deb8-4168-9dfd-50d386860d42 finishe d. {{(pid=23880) backup /opt/stack/cinder/cinder/backup/drivers/ceph.py:927}}
9227 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG os_brick.initiator.connectors.rbd [None req-e8adeca3-4dc7-42 7f-8a77-78d2280621ed admin None] ==> disconnect_volume: call u"{'connection_properties': {u'secret_type': u'ceph', u'name': u' volumes/volume-ebf16d3c-deb8-4168-9dfd-50d386860d42', u'encrypted': False, u'keyring': None, u'cluster_name': u'ceph', u'secre t_uuid': u'***', u'qos_specs': None, u'auth_enabled': True, u'hosts': [u'10.13.57.200'], u'volume_id': u'ebf16d3c-deb8-4168-9d fd-50d386860d42', u'discard': True, u'access_mode': u'rw', u'auth_username': u'cinder', u'ports': [u'6789']}, 'self': <os_bric k.initiator.connectors.rbd.RBDConnector object at 0x7fe02831ff50>, 'force': False, 'device_info': {'path': <os_brick.initiator .linuxrbd.RBDVolumeIOWrapper object at 0x7fdff059c090>}, 'ignore_errors': False}" {{(pid=23880) trace_logging_wrapper /opt/sta ck/os-brick/os_brick/utils.py:146}}
9228 Jul 07 15:50:04 localhost.localdomain cinder-backup[23880]: DEBUG os_brick.initiator.connectors.rbd [None req-e8adeca3-4dc7-42 7f-8a77-78d2280621ed admin None] <== disconnect_volume: return (2ms) None {{(pid=23880) trace_logging_wrapper /opt/stack/os-br ick/os_brick/utils.py:170}}
9229 Jul 07 15:50:06 localhost.localdomain cinder-backup[23880]: WARNING oslo.service.loopingcall [None req-e8adeca3-4dc7-427f-8a77 -78d2280621ed admin None] Function 'cinder.service.Service.report_state' run outlasted interval by 18.95 sec
9230 Jul 07 15:50:07 localhost.localdomain cinder-backup[23880]: INFO cinder.backup.manager [None req-e8adeca3-4dc7-427f-8a77-78d22 80621ed admin None] Create backup finished. backup: 7f27f402-31cb-49fc-ad75-a52d6a5252aa.

Version-Release number of selected component (if applicable):
[csaikia@localhost cinder]$ cinder --version
2.2.1
[csaikia@localhost cinder]$ openstack --version
openstack 3.11.0

How reproducible:
Every time

Expected results:
is_incremental flag should be set to False if the full backup is done when the incremental backup fails

[Gorka Eguileor (gorka)]

The --force optional parameter is not meant to force full backups, but force backups of in-use volumes. So a --force --incremental will do an incremental backup of an in-use volume.

[Chaynika Saikia (csaikia)]

In cinder backup-create, whenever an attempt to do an incremental backup fails, it goes to doing a full backup. However, in this case, if the person had set the --incremental flag and then the incremental backup failed, then also the is_incremental flag never gets updated.

Secondly, as of now, if there is an existing full backup for a volume, the next attempt for a backup create is always an incremental backup. However, cinder never actually updates the is_incremental flag unless the user had set it in the CLI command. An example is:

[csaikia@localhost ~]$ cinder backup-create f24bae67-9342-42de-b01b-b63c55c84239
+-----------+--------------------------------------+
| Property | Value |
+-----------+--------------------------------------+
| id | 59c17b88-a2c8-4d4d-b16c-c0a5a8312100 |
| name | None |
| volume_id | f24bae67-9342-42de-b01b-b63c55c84239 |
+-----------+--------------------------------------+
[csaikia@localhost ~]$ cinder backup-show 59c17b88-a2c8-4d4d-b16c-c0a5a8312100
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| availability_zone | nova |
| container | backups |
| created_at | 2017-07-10T14:50:41.000000 |
| data_timestamp | 2017-07-10T14:50:41.000000 |
| description | None |
| fail_reason | None |
| has_dependent_backups | False |
| id | 59c17b88-a2c8-4d4d-b16c-c0a5a8312100 |
| is_incremental | False |
| name | None |
| object_count | 0 |
| size | 1 |
| snapshot_id | None |
| status | available |
| updated_at | 2017-07-10T14:50:45.000000 |
| volume_id | f24bae67-9342-42de-b01b-b63c55c84239 |
+-----------------------+--------------------------------------+

However from the logs, one can see that an incremental backup was performed:

10685 Jul 10 10:50:42 localhost.localdomain cinder-backup[11683]: DEBUG cinder.backup.drivers.ceph [None req-fedf4549-fda2-4745-add f-2c1e1c7a8da7 admin None] Volume file is RBD: attempting incremental backup. {{(pid=11683) backup /opt/stack/cinder/cinder/b ackup/drivers/ceph.py:894}}
10686 Jul 10 10:50:42 localhost.localdomain cinder-backup[11683]: DEBUG cinder.backup.drivers.ceph [None req-fedf4549-fda2-4745-add f-2c1e1c7a8da7 admin None] Using --from-snap 'backup.6990dcb4-2161-4979-a937-30c87f717a41.snap.1499698148.33' for incremental backup of volume f24bae67-9342-42de-b01b-b63c55c84239. {{(pid=11683) _backup_rbd /opt/stack/cinder/cinder/backup/drivers/cep h.py:636}}
10687 Jul 10 10:50:42 localhost.localdoma...

[Gorka Eguileor (gorka)]

OK, so we are saying that the problem is that the reported bug is that the Ceph driver does not set the incremental field properly.

Then it is correct, and what we need to do is to set the right values after creating the backup to overwrite whatever the API will have written, since the api could be saying it's incremental and Cinder saying it's not, and the other way around.

We need to properly set the parent_id when Ceph driver does an incremental backup and set it to None when it does a full backup (even if the API has set it to something).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/483089

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/483089
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f6cbcf46361c49dc75c3fca2a265de734bfd2e61
Submitter: Jenkins
Branch: master

commit f6cbcf46361c49dc75c3fca2a265de734bfd2e61
Author: Chaynika Saikia <email address hidden>
Date: Wed Jul 12 16:09:47 2017 -0400

    cinder backup sets incorrect parent_id

    If --incremental flag is used and an incremental backup
    fails, it should update the parent_id of the backup to
    None.

    Also for ceph driver, any backup after the first one
    is an incremental backup automatically. Hence, the
    is_incremental field for cinder backup-show <backup-id>
    is misleading since it is set to False currently.

    The patch correctly updates the parent_id for a
    backup in case the incremental backup is successful.
    If not, then it retains the parent_id as None. This
    will also update the parent_id to None if incremental
    backup fails even when --incremental flag is used.

    Change-Id: I22deff68d9376c2ecc88bd2d9d0e1b445d793123
    Closes-Bug: #1703013

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 11.0.0.0rc1 release candidate.