cinder backup incremental when volume change size

[Sang Tran]

Hello team,

I've just caught a new bug with cinder-backup, the step to reproduce as below:

1. Create a full backup with a volume size of 50 GB
2. Extend the original volume to 100 GB
3. Create an incremental backup of the same volume id --> Fail created

Workaround solution:
- Create the new full backup and then we can create incremental backup work normally

I think we should check the size changed before creating incremental or force create a new full backup with a new volume size instead.

[Sofia Enriquez (lsofia-enriquez)]

Hi Sang Tran,
Please share the next information:
- Backup driver
- Logs with the errors from c-bak and c-api.
Thanks

[Sang Tran (sangtq8)]

Hi Sofia,

I used Backup S3 driver and the logs error from c-bak is shown below:

2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server [req-934aa717-c161-43c7-8db6-3d8ddaee046f 25cc962ecb444e0d8e28c843782958f5 ca53347fad024242bc91e7a11b87db16 - - -] Exception during message handling: cinder.exception.InvalidBackup: Invalid backup: Volume size increased since the last backup. Do a full backup.
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/utils.py", line 890, in wrapper
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 410, in create_backup
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err))
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server raise self.value
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 399, in create_backup
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume)
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 482, in _run_backup
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server updates = backup_service.backup(
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 587, in backup
2022-10-13 19:48:00.605 22 ERROR oslo_messaging.rpc.server raise exception.InvalidBackup(reason=err)
2022-10-13 19:4...

[Nguyen Luu Hoang Minh (minhnlh2)]

Hi Sofia and Sang Tran,

I think this is not a bug because the code explicitly raise the error if there are any changes in size of the volume:
https://opendev.org/openstack/cinder/src/branch/master/cinder/backup/chunkeddriver.py#:~:text=if%20backup.size,reason%3Derr)

However, I agree with Sang that this can be improved. I think it would be better if we can specify an options to force create a full backup if user try to create an incremental backup on the volume whose size has just been changed.

For example:
openstack volume backup create --incremental --force-on-extended-volume <<volume_id>>

This will create a full backup first then comes the incremental, instead of raising any errors. For the force created full backup, there would be some information stored in the metadata for further tracking or monitoring. Something like "create_reason": "Force created as a checkpoint backup on extended volume"

I think having an option to automatically create a full backup can be helpful in some cases.
- An automation external scheduling backup tool can send backup creation requests seamlessly to cinder and will not stuck on this exception, otherwise we have to manually run a full backup.

If you agree with my opinion, we can discuss further and I think I can start on this bug.

Thanks,