Ubuntu
cinder package

Google backup driver fails to backup a volume

Bug #1831735 reported by Tzach Shefi
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Sean McGinnis
cinder (Ubuntu)
New
Undecided
Unassigned

Bug Description

Description of problem: Cinder volume backup using Google as backup backend fails. We suspect this regression was introduced by move to Python3.

Version-Release number of selected component (if applicable):
rhel8
puppet-cinder-14.4.1-0.20190420083336.1cf0604.el8ost.noarch
openstack-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch
python3-cinderclient-4.2.0-0.20190520060354.953243d.el8ost.noarch
python3-cinder-14.0.1-0.20190524082243.520e6e0.el8ost.noarch

python3-google-api-client-1.6.5-3.el8.noarch

How reproducible:
Every time

Steps to Reproduce:
1. Configure Google backup.

Cinder backup service is enabled and up
$ cinder service-list
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+-------------------------+------+---------+-------+----------------------------+-----------------+
| cinder-backup | controller-0 | nova | enabled | up | 2019-06-05T11:17:47.000000 | - |

2. Create a volume
vol id -> 0336a629-25c4-4e05-be06-de51a75afc9d

3. If you try to backup the volume, backup fails
cinder backup logs reports the trace show below

$ cinder backup-list
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
| ID | Volume ID | Status | Name | Size | Object Count | Container |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+
| d256989b-a88f-4dcf-95d2-1c7afa12be76 | 0336a629-25c4-4e05-be06-de51a75afc9d | error | - | 1 | 0 | mycinderbucket |
+--------------------------------------+--------------------------------------+--------+------+------+--------------+----------------+

e trace_logging_wrapper /usr/lib/python3.6/site-packages/os_brick/utils.py:170
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server [req-4f591f5c-031a-417e-974c-37cdb2ced102 c45b531877804ab097b8f587497031db ac390c9999244f56aa05cb13e181f0bc - default default] Exception during message handling: cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 121, in func_wrapper
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 312, in close
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server md5 = md5.encode('utf-8')
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server AttributeError: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 406, in create_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err))
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server raise value
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 395, in create_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 479, in _run_backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server backup, tpool.Proxy(device_file))
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 625, in backup
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server object_meta, extra_metadata)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/chunkeddriver.py", line 374, in _backup_chunk
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server writer.write(output_data)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 221, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server return proxy_call(self._autowrap, self._obj.__exit__, *exc)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server raise value
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 293, in __exit__
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server self.close()
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 127, in func_wrapper
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server raise exception.GCSConnectionFailure(reason=err)
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server cinder.exception.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-06-05 09:25:26.581 21 ERROR oslo_messaging.rpc.server
2019-06-05 09:25:57.656 21 DEBUG oslo_service.periodic_task [req-dbe4b355-7960-4100-a621-4fba6a5c05ac - - - - -] Running periodic task BackupManager._report_driver_status run_periodic_tasks /usr/lib/python3.6/site-packages/oslo_service/periodic_task.py:217

Actual results:
Cinder backup fails

Expected results:
Cinder backup should successfully complete the backup.
Least this worked on previous openstack release.

Revision history for this message
Tzach Shefi (tshefi) wrote :
  • Logs Edit (214.4 KiB, application/zip)
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

Workaround for now is to not run under Python 3. May need to deprecate driver if there is no maintainer for it.

tags: added: backup drivers google
summary: - Cinder's Google backup driver fails to backup a volume
+ Google backup driver fails to backup a volume
Changed in cinder:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/676403

Changed in cinder:
assignee: nobody → Pablo (pcaruana)
status: New → In Progress
OpenStack Infra (hudson-openstack)
Changed in cinder:
assignee: Pablo (pcaruana) → Sean McGinnis (sean-mcginnis)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.opendev.org/676403
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=38d581996b436bae9f294be9b030a8e1c427f1a6
Submitter: Zuul
Branch: master

commit 38d581996b436bae9f294be9b030a8e1c427f1a6
Author: Pablo Caruana <email address hidden>
Date: Wed Aug 14 14:37:59 2019 +0200

    Google backup: correct string encoding between py 2 and 3

    Correcting encode() and decode() methods of 8-bit and Unicode
    string objects.

    In Python 2, Both 8-bit and Unicode strings had both methods
    and the type of the return value was based on the specific
    encoding passed in.

    In Python 3, Text strings only have an encode() method, and
    that method can only be used with codecs that produce bytes
    objects. Similarly bytes and bytearray objects only have a
    decode() method which can only be used with codecs that
    produce string objects.

    Change-Id: I214a786858655b8caf9612f3da5403c7d0bb7387
    Closes-Bug: #1831735

Changed in cinder:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/684722

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 15.0.0.0rc1

This issue was fixed in the openstack/cinder 15.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/stein)

Reviewed: https://review.opendev.org/684722
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=47726fb3ad104698327e40efb94b088d311ce543
Submitter: Zuul
Branch: stable/stein

commit 47726fb3ad104698327e40efb94b088d311ce543
Author: Pablo Caruana <email address hidden>
Date: Wed Aug 14 14:37:59 2019 +0200

    Google backup: correct string encoding between py 2 and 3

    Correcting encode() and decode() methods of 8-bit and Unicode
    string objects.

    In Python 2, Both 8-bit and Unicode strings had both methods
    and the type of the return value was based on the specific
    encoding passed in.

    In Python 3, Text strings only have an encode() method, and
    that method can only be used with codecs that produce bytes
    objects. Similarly bytes and bytearray objects only have a
    decode() method which can only be used with codecs that
    produce string objects.

    Change-Id: I214a786858655b8caf9612f3da5403c7d0bb7387
    Closes-Bug: #1831735
    (cherry picked from commit 38d581996b436bae9f294be9b030a8e1c427f1a6)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 14.0.2

This issue was fixed in the openstack/cinder 14.0.2 release.

Revision history for this message
Tzach Shefi (tshefi) wrote :
Download full text (6.0 KiB)

Don't know if I should reopen, same issue or other related issue.
openstack-cinder-15.0.1-0.20191114132949.b91f514.el8ost.noarch

Again failing to backup a volume using Cinder's Google backup driver.

5a7c386672 - default default] Exception during message handling: cinder.backup.drivers.gcs.GCSConnectionFailure: Google Cloud Storage connection failure: 'bytes' object has no attribute 'encode'
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 134, in func_wrapper
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/gcs.py", line 325, in close
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server md5 = md5.encode('utf-8')
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server AttributeError: 'bytes' object has no attribute 'encode'
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 417, in create_backup
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err))
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2019-12-04 15:27:02.236 24 ERROR oslo_messaging.rpc.server rai...

Read more...

Revision history for this message
Daniele Caiti (danycaiti) wrote :

Bug still present on Train, verified on Ubuntu 18.04. Patch ready to solve it.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.