When creating a cinder backup of a volume when using ceph as a back end and the python3 version of cinder, the backup is in error state with fail_reason: data must be bytes. This clearly is a python3 problem. But it is unclear to me which package is at fault. python3-cinder, ceph-common, python3-rados. $ cinder backup-list +--------------------------------------+--------------------------------------+--------+----------+------+--------------+---------------+ | ID | Volume ID | Status | Name | Size | Object Count | Container | +--------------------------------------+--------------------------------------+--------+----------+------+--------------+---------------+ | 75bfe189-6ac4-4bea-bb53-e523a8f1a9af | 64c5a514-0641-4b25-bd8f-092f40249400 | error | demo-bak | 1 | 0 | cinder-backup | +--------------------------------------+--------------------------------------+--------+----------+------+--------------+---------------+ $ cinder backup-show 75bfe189-6ac4-4bea-bb53-e523a8f1a9af +-----------------------------------+--------------------------------------+ | Property | Value | +-----------------------------------+--------------------------------------+ | availability_zone | nova | | container | cinder-backup | | created_at | 2018-10-19T15:14:10.000000 | | data_timestamp | 2018-10-19T15:14:10.000000 | | description | None | | fail_reason | data must be bytes | | has_dependent_backups | False | | id | 75bfe189-6ac4-4bea-bb53-e523a8f1a9af | | is_incremental | False | | metadata | {} | | name | demo-bak | | object_count | 0 | | os-backup-project-attr:project_id | a399969417f146b0b7c8fce20c65c1e3 | | size | 1 | | snapshot_id | None | | status | error | | updated_at | 2018-10-19T15:14:16.000000 | | volume_id | 64c5a514-0641-4b25-bd8f-092f40249400 | +-----------------------------------+--------------------------------------+ cinder-backup.log: e7545f300fb23 38ecce7606e84e17b28e7545f300fb23] Create backup started, backup: dadf4f46-91f0-457b-9c47-edef63381c8e volume: 5c0d8a28-e391-4533-8b9c-5304b4afeefb. 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server [req-8099dad4-2f4a-4698-b1c1-0fb4a3f2da0e b5f325fae3904bae80a9366efaf12766 143dc30f79f34c649e4b50d64dc7c62a - 38ecce7606e84e17b28e7545f300fb23 38ecce7606e84e17b28e7545f300fb23] Exception during message handling: TypeError: data must be bytes 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 425, in create_backup 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err)) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server self.force_reraise() 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server raise value 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 414, in create_backup 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 502, in _run_backup 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server tpool.Proxy(device_path)) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 950, in backup 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server self._backup_metadata(backup) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 903, in _backup_metadata 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server vol_meta_backup.set(json_meta) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 139, in set 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server meta_obj.write(json_meta) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 186, in doit 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 144, in proxy_call 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 125, in execute 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/support/six.py", line 625, in reraise 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server raise value 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 83, in tworker 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs) 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 3939, in rados.set_object_locator.retfunc 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 3949, in rados.set_object_namespace.retfunc 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 3986, in rados.Object.write 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 517, in rados.requires.wrapper.validate_func 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 505, in rados.requires.check_type 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server TypeError: data must be bytes 2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server Packages: ii cinder-api 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - API server ii cinder-backup 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - Scheduler server ii cinder-common 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - common files ii cinder-scheduler 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - Scheduler server ii cinder-volume 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - Volume server ii python3-cinder 2:13.0.0-0ubuntu3~cloud0 all Cinder Python 3 libraries ii ceph-common 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster ii libcephfs2 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Ceph distributed file system client library ii python-cephfs 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Python libraries for the Ceph libcephfs library ii librados2 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 RADOS distributed object store client library ii libradosstriper1 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 RADOS striping interface ii python-rados 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Python libraries for the Ceph librados library ii python3-rados 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Python 3 libraries for the Ceph librados library ProblemType: Bug DistroRelease: Ubuntu 18.04 Package: python3-cinder 2:13.0.0-0ubuntu3~cloud0 [origin: Canonical] ProcVersionSignature: Ubuntu 4.15.0-36.39-generic 4.15.18 Uname: Linux 4.15.0-36-generic x86_64 ApportVersion: 2.20.9-0ubuntu7.4 Architecture: amd64 CrashDB: { "impl": "launchpad", "project": "cloud-archive", "bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml", } Date: Fri Oct 19 23:20:06 2018 Ec2AMI: ami-000001ca Ec2AMIManifest: FIXME Ec2AvailabilityZone: nova Ec2InstanceType: m1.blue Ec2Kernel: unavailable Ec2Ramdisk: unavailable ExecutablePath: /usr/bin/python3-cinder-backup InterpreterPath: /usr/bin/python3.6 PackageArchitecture: all ProcEnviron: PATH=(custom, no user) LANG=C.UTF-8 SHELL=/bin/false Python3Details: /usr/bin/python3.6, Python 3.6.6, python3-minimal, 3.6.5-3ubuntu1 PythonDetails: /usr/bin/python2.7, Python 2.7.15rc1, python-minimal, 2.7.15~rc1-1 SourcePackage: cinder UpgradeStatus: No upgrade log present (probably fresh install)