Description of problem: Cinder volume backup using Google as backup backend fails. We suspect this regression was introduced by move to Python3.
Version-Release number of selected component (if applicable):
rhel8
puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch
openstack-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch
python3-cinderclient-4.2.0-0.20190520060354.953243d.el8ost.noarch
python3-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch
python3-google-api-client-1.6.5-3.el8.noarch
How reproducible:
Every time
Steps to Reproduce:
1. Configure Google backup.
Cinder backup service is enabled and up
$ cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup | controller-0 | nova | enabled | up | 2019-06-05T11:17:47.000000 | - |
2. Create a volume
vol id -> 0336a629-25c4-4e05-be06-de51a75afc9d
3. If you try to backup the volume, backup fails
cinder backup logs reports the trace show below
$ cinder backup-list
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
| d256989b-a88f-4dcf-95d2-1c7afa12be76 | 0336a629-25c4-4e05-be06-de51a75afc9d | error | - | 1 | 0 | mycinderbucket |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
e trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:170
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server [req-4f591f5c-031a-417e-974c-37cdb2ced102 c45b531877804ab097b8f587497031db ac390c9999244f56aa05cb13e181f0bc - default default] Exception during message handling: cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 121, in func_wrapper
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 312, in close
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server md5 = md5.encode('utf-8')
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server AttributeError: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 406, in create_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err))
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server raise value
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 395, in create_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 479, in _run_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server backup, tpool.Proxy(device_file))
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 625, in backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server object_meta, extra_metadata)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 374, in _backup_chunk
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server writer.write(output_data)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 221, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server return proxy_call(self._autowrap, self._obj.__exit__, *exc)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server raise value
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 293, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server self.close()
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 127, in func_wrapper
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server raise exception.GCSConnectionFailure(reason=err)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server
2019-06-05 09:25:57.656 21 DEBUG oslo_service.periodic_task [req-dbe4b355-7960-4100-a621-4fba6a5c05ac - - - - -] Running periodic task BackupManager._report_driver_status run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217
Actual results:
Cinder backup fails
Expected results:
Cinder backup should successfully complete the backup.
Least this worked on previous openstack release.
Workaround for now is to not run under Python 3. May need to deprecate driver if there is no maintainer for it.