At RDO, we are hitting the issue below in one of our scenario when Ceph is used as backup driver.
Full log at
https://review.rdoproject.org/zuul/build/a7feaabb6e144094a3ec314345cbd323
https://logserver.rdoproject.org/87/48487/1/check/weirdo-bobcat-promote-puppet-scenario001-centos-stream-9/1bbba21/logs/weirdo-project/logs/testr_results.html
https://logserver.rdoproject.org/openstack-periodic-daily/review.rdoproject.org/rdoinfo/master/weirdo-bobcat-promote-puppet-scenario001-centos-stream-9/a7feaab/logs/weirdo-project/logs/cinder/backup.txt.gz
The issue was spotted since the merge of https://review.opendev.org/c/openstack/tempest/+/890798
2023-09-08 02:40:04.602 94882 DEBUG os_brick.initiator.connectors.rbd [None req-56eb13eb-d701-4a5c-852b-a28856a19a65 ee2b8d39be9d490ba070b1431a3784ee ae025cfb04954e62a85ba4bf06c1e31f - - - -] <== disconnect_volume: return (28ms) None trace_logging_wrapper /usr/lib/python3.9/site-packages/os_brick/utils.py:203
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server [None req-56eb13eb-d701-4a5c-852b-a28856a19a65 ee2b8d39be9d490ba070b1431a3784ee ae025cfb04954e62a85ba4bf06c1e31f - - - -] Exception during message handling: rados.ObjectNotFound: [errno 2] RADOS object not found (error opening pool 'tempest-VolumesBackupsAdminTest-backup-container-335842613')
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 531, in continue_backup
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err))
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server raise self.value
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 503, in continue_backup
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context(
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server raise self.value
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 497, in continue_backup
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server updates = backup_service.backup(backup,
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/ceph.py", line 1047, in backup
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server updates = self._backup_rbd(backup, volume_file, volume.name,
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/ceph.py", line 799, in _backup_rbd
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server from_snap, image_created = self._full_rbd_backup(backup.container,
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/ceph.py", line 736, in _full_rbd_backup
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server with eventlet.tpool.Proxy(rbd_driver.RADOSClient(self,
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 266, in __init__
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server self.cluster, self.ioctx = driver._connect_to_rados(pool)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/ceph.py", line 325, in _connect_to_rados
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server ioctx = client.open_ioctx(pool_to_open)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server raise value
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "rados.pyx", line 988, in rados.Rados.open_ioctx
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server rados.ObjectNotFound: [errno 2] RADOS object not found (error opening pool 'tempest-VolumesBackupsAdminTest-backup-container-335842613')
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server
2023-09-08 02:40:05.002 94882 INFO cinder.backup.manager [None req-dda3b8d3-8be1-4ee6-bf46-b4b408dcfc3b ee2b8d39be9d490ba070b1431a3784ee ae025cfb04954e62a85ba4bf06c1e31f - - - -] Delete backup started, backup: 6c17823c-239a-4e81-9de7-ce0145671fe8.
for the record, a duplicate of this one: https:/ /bugs.launchpad .net/tempest/ +bug/2034933