| 2018-10-24 20:43:59 |
Corey Bryant |
description |
When creating a cinder backup of a volume when using ceph as a back end and the python3 version of cinder, the backup is in error state with fail_reason: data must be bytes.
This clearly is a python3 problem. But it is unclear to me which package is at fault. python3-cinder, ceph-common, python3-rados.
$ cinder backup-list
+--------------------------------------+--------------------------------------+--------+----------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+--------+----------+------+--------------+---------------+
| 75bfe189-6ac4-4bea-bb53-e523a8f1a9af | 64c5a514-0641-4b25-bd8f-092f40249400 | error | demo-bak | 1 | 0 | cinder-backup |
+--------------------------------------+--------------------------------------+--------+----------+------+--------------+---------------+
$ cinder backup-show 75bfe189-6ac4-4bea-bb53-e523a8f1a9af
+-----------------------------------+--------------------------------------+
| Property | Value |
+-----------------------------------+--------------------------------------+
| availability_zone | nova |
| container | cinder-backup |
| created_at | 2018-10-19T15:14:10.000000 |
| data_timestamp | 2018-10-19T15:14:10.000000 |
| description | None |
| fail_reason | data must be bytes |
| has_dependent_backups | False |
| id | 75bfe189-6ac4-4bea-bb53-e523a8f1a9af |
| is_incremental | False |
| metadata | {} |
| name | demo-bak |
| object_count | 0 |
| os-backup-project-attr:project_id | a399969417f146b0b7c8fce20c65c1e3 |
| size | 1 |
| snapshot_id | None |
| status | error |
| updated_at | 2018-10-19T15:14:16.000000 |
| volume_id | 64c5a514-0641-4b25-bd8f-092f40249400 |
+-----------------------------------+--------------------------------------+
cinder-backup.log:
e7545f300fb23 38ecce7606e84e17b28e7545f300fb23] Create backup started, backup: dadf4f46-91f0-457b-9c47-edef63381c8e volume: 5c0d8a28-e391-4533-8b9c-5304b4afeefb.
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server [req-8099dad4-2f4a-4698-b1c1-0fb4a3f2da0e b5f325fae3904bae80a9366efaf12766 143dc30f79f34c649e4b50d64dc7c62a - 38ecce7606e84e17b28e7545f300fb23 38ecce7606e84e17b28e7545f300fb23] Exception during message handling: TypeError: data must be bytes
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 425, in create_backup
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err))
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server raise value
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 414, in create_backup
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 502, in _run_backup
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server tpool.Proxy(device_path))
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 950, in backup
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server self._backup_metadata(backup)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 903, in _backup_metadata
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server vol_meta_backup.set(json_meta)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 139, in set
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server meta_obj.write(json_meta)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 186, in doit
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 125, in execute
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/support/six.py", line 625, in reraise
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server raise value
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 83, in tworker
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 3939, in rados.set_object_locator.retfunc
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 3949, in rados.set_object_namespace.retfunc
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 3986, in rados.Object.write
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 517, in rados.requires.wrapper.validate_func
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 505, in rados.requires.check_type
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server TypeError: data must be bytes
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server
Packages:
ii cinder-api 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - API server
ii cinder-backup 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - Scheduler server
ii cinder-common 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - common files
ii cinder-scheduler 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - Scheduler server
ii cinder-volume 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - Volume server
ii python3-cinder 2:13.0.0-0ubuntu3~cloud0 all Cinder Python 3 libraries
ii ceph-common 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs2 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Ceph distributed file system client library
ii python-cephfs 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Python libraries for the Ceph libcephfs library
ii librados2 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 RADOS distributed object store client library
ii libradosstriper1 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 RADOS striping interface
ii python-rados 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Python libraries for the Ceph librados library
ii python3-rados 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Python 3 libraries for the Ceph librados library
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: python3-cinder 2:13.0.0-0ubuntu3~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.15.0-36.39-generic 4.15.18
Uname: Linux 4.15.0-36-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.4
Architecture: amd64
CrashDB:
{
"impl": "launchpad",
"project": "cloud-archive",
"bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
}
Date: Fri Oct 19 23:20:06 2018
Ec2AMI: ami-000001ca
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.blue
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ExecutablePath: /usr/bin/python3-cinder-backup
InterpreterPath: /usr/bin/python3.6
PackageArchitecture: all
ProcEnviron:
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/false
Python3Details: /usr/bin/python3.6, Python 3.6.6, python3-minimal, 3.6.5-3ubuntu1
PythonDetails: /usr/bin/python2.7, Python 2.7.15rc1, python-minimal, 2.7.15~rc1-1
SourcePackage: cinder
UpgradeStatus: No upgrade log present (probably fresh install) |
[Impact]
When creating a cinder backup of a volume when using ceph as a back end and the python3 version of cinder, the backup is in error state with fail_reason: data must be bytes.
This clearly is a python3 problem. But it is unclear to me which package is at fault. python3-cinder, ceph-common, python3-rados.
$ cinder backup-list
+--------------------------------------+--------------------------------------+--------+----------+------+--------------+---------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+--------+----------+------+--------------+---------------+
| 75bfe189-6ac4-4bea-bb53-e523a8f1a9af | 64c5a514-0641-4b25-bd8f-092f40249400 | error | demo-bak | 1 | 0 | cinder-backup |
+--------------------------------------+--------------------------------------+--------+----------+------+--------------+---------------+
$ cinder backup-show 75bfe189-6ac4-4bea-bb53-e523a8f1a9af
+-----------------------------------+--------------------------------------+
| Property | Value |
+-----------------------------------+--------------------------------------+
| availability_zone | nova |
| container | cinder-backup |
| created_at | 2018-10-19T15:14:10.000000 |
| data_timestamp | 2018-10-19T15:14:10.000000 |
| description | None |
| fail_reason | data must be bytes |
| has_dependent_backups | False |
| id | 75bfe189-6ac4-4bea-bb53-e523a8f1a9af |
| is_incremental | False |
| metadata | {} |
| name | demo-bak |
| object_count | 0 |
| os-backup-project-attr:project_id | a399969417f146b0b7c8fce20c65c1e3 |
| size | 1 |
| snapshot_id | None |
| status | error |
| updated_at | 2018-10-19T15:14:16.000000 |
| volume_id | 64c5a514-0641-4b25-bd8f-092f40249400 |
+-----------------------------------+--------------------------------------+
cinder-backup.log:
e7545f300fb23 38ecce7606e84e17b28e7545f300fb23] Create backup started, backup: dadf4f46-91f0-457b-9c47-edef63381c8e volume: 5c0d8a28-e391-4533-8b9c-5304b4afeefb.
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server [req-8099dad4-2f4a-4698-b1c1-0fb4a3f2da0e b5f325fae3904bae80a9366efaf12766 143dc30f79f34c649e4b50d64dc7c62a - 38ecce7606e84e17b28e7545f300fb23 38ecce7606e84e17b28e7545f300fb23] Exception during message handling: TypeError: data must be bytes
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 425, in create_backup
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err))
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server self.force_reraise()
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server raise value
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 414, in create_backup
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/manager.py", line 502, in _run_backup
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server tpool.Proxy(device_path))
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 950, in backup
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server self._backup_metadata(backup)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 903, in _backup_metadata
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server vol_meta_backup.set(json_meta)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/backup/drivers/ceph.py", line 139, in set
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server meta_obj.write(json_meta)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 186, in doit
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 125, in execute
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/support/six.py", line 625, in reraise
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server raise value
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 83, in tworker
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 3939, in rados.set_object_locator.retfunc
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 3949, in rados.set_object_namespace.retfunc
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 3986, in rados.Object.write
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 517, in rados.requires.wrapper.validate_func
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server File "rados.pyx", line 505, in rados.requires.check_type
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server TypeError: data must be bytes
2018-10-18 22:17:20.284 15452 ERROR oslo_messaging.rpc.server
Packages:
ii cinder-api 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - API server
ii cinder-backup 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - Scheduler server
ii cinder-common 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - common files
ii cinder-scheduler 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - Scheduler server
ii cinder-volume 2:13.0.0-0ubuntu3~cloud0 all Cinder storage service - Volume server
ii python3-cinder 2:13.0.0-0ubuntu3~cloud0 all Cinder Python 3 libraries
ii ceph-common 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs2 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Ceph distributed file system client library
ii python-cephfs 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Python libraries for the Ceph libcephfs library
ii librados2 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 RADOS distributed object store client library
ii libradosstriper1 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 RADOS striping interface
ii python-rados 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Python libraries for the Ceph librados library
ii python3-rados 13.2.1+dfsg1-0ubuntu2~cloud0 amd64 Python 3 libraries for the Ceph librados library
ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: python3-cinder 2:13.0.0-0ubuntu3~cloud0 [origin: Canonical]
ProcVersionSignature: Ubuntu 4.15.0-36.39-generic 4.15.18
Uname: Linux 4.15.0-36-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.4
Architecture: amd64
CrashDB:
{
"impl": "launchpad",
"project": "cloud-archive",
"bug_pattern_url": "http://people.canonical.com/~ubuntu-archive/bugpatterns/bugpatterns.xml",
}
Date: Fri Oct 19 23:20:06 2018
Ec2AMI: ami-000001ca
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.blue
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
ExecutablePath: /usr/bin/python3-cinder-backup
InterpreterPath: /usr/bin/python3.6
PackageArchitecture: all
ProcEnviron:
PATH=(custom, no user)
LANG=C.UTF-8
SHELL=/bin/false
Python3Details: /usr/bin/python3.6, Python 3.6.6, python3-minimal, 3.6.5-3ubuntu1
PythonDetails: /usr/bin/python2.7, Python 2.7.15rc1, python-minimal, 2.7.15~rc1-1
SourcePackage: cinder
UpgradeStatus: No upgrade log present (probably fresh install)
[Test Case]
Deploy OpenStack with a python 3 cinder having a ceph backend. Create a cinder volume backup and ensure it is successful.
[Regression Potential]
The patch that is cherry-picked has landed upstream, minimizing the potential of regression. |
|
