OpenStack Cinder Backup Swift Proxy Charm

[Focal][ceph-octopus] os-brick bug causes volume backup creation to fail

Bug #1890821 reported by Alex Kavanagh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Cinder Backup Swift Proxy Charm
New
Undecided
Unassigned

Bug Description

The bug in os-brick (https://bugs.launchpad.net/tripleo/+bug/1865754) means that the volumes cannot be backed up to a ceph octopus cluster. The charm will install correctly with no errors, but attempting to create the backup will result in an errored volume:

$ openstack volume backup list
+--------------------------------------+------+-------------+--------+------+
| ID | Name | Description | Status | Size |
+--------------------------------------+------+-------------+--------+------+
| 76b2b2a1-4281-4b5b-9fcf-fb42915135cc | None | None | error | 1 |
+--------------------------------------+------+-------------+--------+------+

The trace in the cinder-backup.log file will look like:

2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server [req-a1e10fa7-e9eb-4a43-90cb-70253dabee2f 14da5b67011842a68fd2d333e447b3d0 1a97b414a3314f2fa71c61fe23f4140a - 0de5b9a2bc474d069126a8604a8f6335 0de5b9a2bc474d069126a8604a8f6335] Exception during mess
age handling: rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 409, in create_backup
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, six.text_type(err))
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server raise value
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 398, in create_backup
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 466, in _run_backup
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server attach_info = self._attach_device(context,
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 1006, in _attach_device
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server return self._attach_volume(ctxt, backup_device, properties)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 1025, in _attach_volume
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server LOG.warning("Failed to terminate the connection "
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server raise value
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 1017, in _attach_volume
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server return self._connect_device(conn)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 1059, in _connect_device
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server vol_handle = connector.connect_volume(conn['data'])
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/rbd.py", line 204, in connect_volume
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server rbd_handle = self._get_rbd_handle(connection_properties)
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/connectors/rbd.py", line 122, in _get_rbd_handle
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server rbd_client = linuxrbd.RBDClient(user, pool, conffile=conf,
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/linuxrbd.py", line 60, in __init__
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server self.client, self.ioctx = self.connect()
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/os_brick/initiator/linuxrbd.py", line 71, in connect
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server client = self.rados.Rados(rados_id=self.rbd_user,
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "rados.pyx", line 692, in rados.Rados.__init__
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "rados.pyx", line 578, in rados.requires.wrapper.validate_func
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "rados.pyx", line 743, in rados.Rados.__setup
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "rados.pyx", line 578, in rados.requires.wrapper.validate_func
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server File "rados.pyx", line 826, in rados.Rados.conf_read_file
2020-08-07 11:17:09.906 44689 ERROR oslo_messaging.rpc.server rados.InvalidArgumentError: [errno 22] RADOS invalid argument (error calling conf_read_file)

The patchset (https://review.opendev.org/#/c/730376/) has been submitted for os_brick. Depending on how this is resolved, this will be backported to ussuri.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to charm-cinder-backup-swift-proxy (master)

Reviewed: https://review.opendev.org/743345
Committed: https://git.openstack.org/cgit/openstack/charm-cinder-backup-swift-proxy/commit/?id=8ffd5a407196cd99590018700ab7d7de3b94ad9b
Submitter: Zuul
Branch: master

commit 8ffd5a407196cd99590018700ab7d7de3b94ad9b
Author: Alex Kavanagh <email address hidden>
Date: Mon Jul 27 20:09:24 2020 +0100

    Release sync for 20.08

    - Classic charms: sync charm-helpers.
    - Classic ceph based charms: also sync charms.ceph
    - Reactive charms: trigger a rebuild
    - sync tox.ini
    - sync requirements.txt and test-requirements.txt
    - Disable focal-ussuri-swift-v3 test due to os-brick bug and
      ceph-octopus

    Related-Bug: #1890821

    Change-Id: I63a7f44ff3d630e184ffb254202026d7c9c51347

Felipe Reyes (freyes)
tags: added: unstable-test
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-cinder-backup-swift-proxy (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-cinder-backup-swift-proxy/+/842818

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-cinder-backup-swift-proxy (stable/victoria)

Change abandoned by "Felipe Reyes <email address hidden>" on branch: stable/victoria
Review: https://review.opendev.org/c/openstack/charm-cinder-backup-swift-proxy/+/842818

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.