cinder backup python3 encoding issue

[Danny Webb]

Hi,

While using restoring a volume from ceph -> Nimble storage array with a target volume larger than the original backup the padding fails due to a TypeError:

2021-04-23 09:43:02.208 20 DEBUG cinder.backup.drivers.ceph [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Transferred chunk 63 of 80 (476724K/s) _transfer_data /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:418
2021-04-23 09:43:02.486 20 DEBUG cinder.backup.drivers.ceph [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Transferred chunk 64 of 80 (472558K/s) _transfer_data /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:418
2021-04-23 09:43:02.494 20 DEBUG cinder.backup.drivers.ceph [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Discarding 2147483648 bytes from offset 8589934592 _discard_bytes /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:364
2021-04-23 09:43:02.567 20 DEBUG cinder.backup.drivers.ceph [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Writing zeroes chunk 0 _discard_bytes /usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py:379
2021-04-23 09:43:02.594 20 DEBUG oslo_concurrency.processutils [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdc execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:379
2021-04-23 09:43:02.802 20 DEBUG oslo_concurrency.processutils [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf chown 0 /dev/sdc" returned: 0 in 0.208s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:416
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager [req-b45d0762-dab0-4331-9bd9-cd56e1e9c727 b84534a24a41480e8390f3082b58d88f 1784657cf8964dff82479f4bfc11a941 - default default] Restoring backup 36da785e-9e59-4d70-8dd9-058bb970b40f to volume a61e038b-999d-4446-9624-cfe4c1efca63 failed.: TypeError: a bytes-like object is required, not 'str'
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager Traceback (most recent call last):
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/manager.py", line 654, in _run_restore
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager tpool.Proxy(device_file))
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py", line 1269, in restore
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager self._restore_volume(backup, target_volume, volume_file)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py", line 1236, in _restore_volume
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager length, src_snap=restore_point)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py", line 1038, in _full_restore
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager dest_file, dest_name, length)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py", line 406, in _transfer_data
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager length - dest.tell())
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/cinder/backup/drivers/ceph.py", line 380, in _discard_bytes
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager volume.write(zeroes)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager result = proxy_call(self._autowrap, f, *args, **kwargs)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager rv = execute(f, *args, **kwargs)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager six.reraise(c, e, tb)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager raise value
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager rv = meth(*args, **kwargs)
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager TypeError: a bytes-like object is required, not 'str'
2021-04-23 09:43:02.803 20 ERROR cinder.backup.manager

https://github.com/openstack/cinder/blob/stable/ussuri/cinder/backup/drivers/ceph.py#L372

This can be fixed by encoding the zero stream like:

            else:
                zeroes = '\0' * self.chunk_size
                zeroes = zeroes.encode('utf-8')

though I'm not 100% sure if that does anything untowards or properly zeroes the remaining space.

The cinder version this is on:
cinder (16.2.1) (though I believe it will affect all versions including master)

[Sofia Enriquez (lsofia-enriquez)]

Looks like a RBD related issue, i'll take a look.

[Brian Rosmaita (brian-rosmaita)]

looks like we can do:

zeroes = bytearray(self.chunk_size)

and down a few lines further in the code

zeroes = bytearray(rem)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/791059

[Ber1in (berl1n)]

I also encountered this problem in actual use. I saw that the code review has been submitted, but it has not been merged. Are there any other problems that need to be solved?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/920073

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/920073
Committed: https://opendev.org/openstack/cinder/commit/c13370cdec32ae6b74f3653981519af567be40de
Submitter: "Zuul (22348)"
Branch: master

commit c13370cdec32ae6b74f3653981519af567be40de
Author: bolin.wu <email address hidden>
Date: Tue May 21 13:47:33 2024 +0800

    Fix: [Ceph] Backup Driver Python3 Encoding Issue

    Using python3 the Ceph backup driver faces encoding issues.
    In order to solve this we should start using bytearray for
    creating a zero-filled instance with a given length.

    Closes-Bug: #1925809
    Change-Id: I367df9b52f787403a54fb148713d122e9bb2a54f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2024.1
Review: https://review.opendev.org/c/openstack/cinder/+/924942

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/cinder/+/924942
Committed: https://opendev.org/openstack/cinder/commit/da46f4d775e02b2a36eac8a8fb139728f173e3ab
Submitter: "Zuul (22348)"
Branch: stable/2024.1

commit da46f4d775e02b2a36eac8a8fb139728f173e3ab
Author: bolin.wu <email address hidden>
Date: Tue May 21 13:47:33 2024 +0800

    Fix: [Ceph] Backup Driver Python3 Encoding Issue

    Using python3 the Ceph backup driver faces encoding issues.
    In order to solve this we should start using bytearray for
    creating a zero-filled instance with a given length.

    Closes-Bug: #1925809
    Change-Id: I367df9b52f787403a54fb148713d122e9bb2a54f
    (cherry picked from commit c13370cdec32ae6b74f3653981519af567be40de)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 25.0.0.0rc1 release candidate.