Cinder

Backup fails with VolumeNotFound but not set to error

Bug #1996049 reported by Tobias Urdin on 2022-11-09

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	Fix Released	Low	Unassigned
	Xena	New	Low	Unassigned

Bug Description

The below is for the Xena release, the actual race should be semi-fixed by the patch to the bug https://bugs.launchpad.net/cinder/+bug/1970768 in Zed

When the backup fails we try to set the volume status but if the volume does not exist we fail on that so the backup is never set to "error" and will be in "creating" state forever until cloud admin fixes the status.

2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server [req-69f9f287-8f25-498e-87ab-e7f17211bc5b 9a0267bb0ce845328fe0dac79c0a505b a3860748038847e9b55ba9a935a0d5ca - - -] Exception during message handling: cinder.exception.VolumeNotFound: Volume e613a98c-0ed3-4656-8fed-fef938cc1167 could not be found.
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 411, in create_backup
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 528, in _run_backup
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server message_field.Detail.BACKUP_CREATE_DRIVER_ERROR)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server raise self.value
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 520, in _run_backup
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server tpool.Proxy(device_path))
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 664, in backup
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server self.delete_backup(backup)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server raise self.value
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 659, in backup
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server self._backup_metadata(backup, object_meta)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 453, in _backup_metadata
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server json_meta = self.get_metadata(backup['volume_id'])
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/driver.py", line 360, in get_metadata
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server return self.backup_meta_api.get(volume_id)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/driver.py", line 315, in get
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server self._save_vol_base_meta(container, volume_id)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/driver.py", line 80, in _save_vol_base_meta
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server meta = self.db.volume_get(self.context, volume_id)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 273, in volume_get
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server return IMPL.volume_get(context, volume_id)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 2130, in volume_get
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server return _volume_get(context, volume_id)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 1889, in _volume_get
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server raise exception.VolumeNotFound(volume_id=volume_id)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server cinder.exception.VolumeNotFound: Volume e613a98c-0ed3-4656-8fed-fef938cc1167 could not be found.
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 950, in wrapper
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 421, in create_backup
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server 'previous_status': 'error_backing-up'})
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 326, in volume_update
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server return IMPL.volume_update(context, volume_id, values)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 234, in wrapper
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 2598, in volume_update
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server raise exception.VolumeNotFound(volume_id=volume_id)
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server cinder.exception.VolumeNotFound: Volume e613a98c-0ed3-4656-8fed-fef938cc1167 could not be found.
2022-11-09 10:42:55.172 1047974 ERROR oslo_messaging.rpc.server

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-09: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/864100

Changed in cinder:
status:	New → In Progress

Revision history for this message

Sofia Enriquez (lsofia-enriquez) wrote on 2022-11-09:

Hi Tobias Urdin, thanks for reporting this bug.
What backend are you using?
Thanks in advance
Sofia

tags:	added: backup-service
Changed in cinder:
importance:	Undecided → Low
summary:	- Backup fails with VolumeNotFound but is not set to error + Backup fails with VolumeNotFound but not set to error

Revision history for this message

Tobias Urdin (tobias-urdin) wrote on 2022-11-09:

Hello, thanks for triaging. This is using the Swift backend

Sofia Enriquez (lsofia-enriquez) on 2022-11-09

tags:

added: swift

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-16: Fix merged to cinder (master)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/864100
Committed: https://opendev.org/openstack/cinder/commit/813df9b99f23bdc54787a0c918e47f9d61df3232
Submitter: "Zuul (22348)"
Branch: master

commit 813df9b99f23bdc54787a0c918e47f9d61df3232
Author: Tobias Urdin <email address hidden>
Date: Wed Nov 9 11:14:03 2022 +0100

Set backup status to error on VolumeNotFound

    When creating a backup and it fails we will try
    to update the status on the volume but if it does
    not exist we will instead fail on that and the
    backup will be stuck in creating status which means
    cloud admin need to help resetting the status.

    This catches the VolumeNotFound exception and ignores
    updating the volume status and thus backup will be
    set to error and can be deleted by user instead.

Closes-Bug: #1996049
Change-Id: Iafa92ece7f83323af257c1702df5029469b11739

Changed in cinder:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-11-17: Fix proposed to cinder (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/cinder/+/864701

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-03-08: Fix included in openstack/cinder 22.0.0.0rc1

This issue was fixed in the openstack/cinder 22.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-08-14: Fix merged to cinder (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/cinder/+/864701
Committed: https://opendev.org/openstack/cinder/commit/5f357504ff95ea4731ba309f285b937066d8908a
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 5f357504ff95ea4731ba309f285b937066d8908a
Author: Tobias Urdin <email address hidden>
Date: Wed Nov 9 11:14:03 2022 +0100

Set backup status to error on VolumeNotFound

    This catches the VolumeNotFound exception and ignores
    updating the volume status and thus backup will be
    set to error and can be deleted by user instead.

    Closes-Bug: #1996049
    Change-Id: Iafa92ece7f83323af257c1702df5029469b11739
    (cherry picked from commit 813df9b99f23bdc54787a0c918e47f9d61df3232)

tags:

added: in-stable-zed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-09-14: Fix included in openstack/cinder 21.3.1

This issue was fixed in the openstack/cinder 21.3.1 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.