Creating volume backup fails when Ceph is used as backup driver

Bug #2034913 reported by Joel Capitao
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
tempest
Fix Released
Undecided
Unassigned

Bug Description

At RDO, we are hitting the issue below in one of our scenario when Ceph is used as backup driver.
Full log at
https://review.rdoproject.org/zuul/build/a7feaabb6e144094a3ec314345cbd323
https://logserver.rdoproject.org/87/48487/1/check/weirdo-bobcat-promote-puppet-scenario001-centos-stream-9/1bbba21/logs/weirdo-project/logs/testr_results.html
https://logserver.rdoproject.org/openstack-periodic-daily/review.rdoproject.org/rdoinfo/master/weirdo-bobcat-promote-puppet-scenario001-centos-stream-9/a7feaab/logs/weirdo-project/logs/cinder/backup.txt.gz

The issue was spotted since the merge of https://review.opendev.org/c/openstack/tempest/+/890798

2023-09-08 02:40:04.602 94882 DEBUG os_brick.initiator.connectors.rbd [None req-56eb13eb-d701-4a5c-852b-a28856a19a65 ee2b8d39be9d490ba070b1431a3784ee ae025cfb04954e62a85ba4bf06c1e31f - - - -] <== disconnect_volume: return (28ms) None trace_logging_wrapper /usr/lib/python3.9/site-packages/os_brick/utils.py:203
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server [None req-56eb13eb-d701-4a5c-852b-a28856a19a65 ee2b8d39be9d490ba070b1431a3784ee ae025cfb04954e62a85ba4bf06c1e31f - - - -] Exception during message handling: rados.ObjectNotFound: [errno 2] RADOS object not found (error opening pool 'tempest-VolumesBackupsAdminTest-backup-container-335842613')
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 531, in continue_backup
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err))
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server raise self.value
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 503, in continue_backup
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context(
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server raise self.value
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/manager.py", line 497, in continue_backup
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server updates = backup_service.backup(backup,
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/ceph.py", line 1047, in backup
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server updates = self._backup_rbd(backup, volume_file, volume.name,
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/ceph.py", line 799, in _backup_rbd
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server from_snap, image_created = self._full_rbd_backup(backup.container,
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/ceph.py", line 736, in _full_rbd_backup
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server with eventlet.tpool.Proxy(rbd_driver.RADOSClient(self,
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 266, in __init__
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server self.cluster, self.ioctx = driver._connect_to_rados(pool)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/cinder/backup/drivers/ceph.py", line 325, in _connect_to_rados
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server ioctx = client.open_ioctx(pool_to_open)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server raise value
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server File "rados.pyx", line 988, in rados.Rados.open_ioctx
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server rados.ObjectNotFound: [errno 2] RADOS object not found (error opening pool 'tempest-VolumesBackupsAdminTest-backup-container-335842613')
2023-09-08 02:40:04.650 94882 ERROR oslo_messaging.rpc.server
2023-09-08 02:40:05.002 94882 INFO cinder.backup.manager [None req-dda3b8d3-8be1-4ee6-bf46-b4b408dcfc3b ee2b8d39be9d490ba070b1431a3784ee ae025cfb04954e62a85ba4bf06c1e31f - - - -] Delete backup started, backup: 6c17823c-239a-4e81-9de7-ce0145671fe8.

Joel Capitao (jcapitao)
description: updated
Revision history for this message
Martin Kopec (mkopec) wrote :

for the record, a duplicate of this one: https://bugs.launchpad.net/tempest/+bug/2034933

Revision history for this message
Martin Kopec (mkopec) wrote :

we've merged the revert: https://review.opendev.org/c/openstack/tempest/+/894269

Next action is to revert the revert + some modifications so that the mentioned jobs don't fail now.

Changed in tempest:
status: New → Confirmed
Revision history for this message
Martin Kopec (mkopec) wrote :

This should fix this (it reverts the revert + makes sure that this LP is resolved):
https://review.opendev.org/c/openstack/tempest/+/896011

Changed in tempest:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tempest (master)

Reviewed: https://review.opendev.org/c/openstack/tempest/+/896011
Committed: https://opendev.org/openstack/tempest/commit/553d91b51e12be77e5d6d38d666cd0b23b12e626
Submitter: "Zuul (22348)"
Branch: master

commit 553d91b51e12be77e5d6d38d666cd0b23b12e626
Author: Lukáš Piwowarski <email address hidden>
Date: Thu Sep 21 11:14:24 2023 +0200

    Fix cleanup for object_storage tests

    This reverts change I98a75cbf119ba8126253a681c046f4cf44b1607e.

    Volume backup tests create a container when Swift is used as a backup
    driver. This causes a failure of an object storage test [1] as it
    expects no containers being present before the testing.

    This patch fixes the cleanup by deleting properly the containers
    created during the volume backup tests. Before the deletion we check
    using a new tempest.conf option whether swift is used as a backup
    driver.

    This patche also un-skips the
    test_volume_backup_create_get_detailed_list_restore_delete test as the
    patch ensures that we do not use the container parameter in the API
    call when Swift is not used as a backup driver.

    [1] https://opendev.org/openstack/tempest/src/tempest/api/object_storage/test_account_services.py#L67

    Closes-Bug: #2034913
    Change-Id: I33ba1838bf0bfcf94424e7288249dae3feeeb2a2

Changed in tempest:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tempest 37.0.0

This issue was fixed in the openstack/tempest 37.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.