cinder backup S3 driver failure: signed integer is greater than maximum

Bug #1992292 reported by Sang Tran
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Low
Unassigned

Bug Description

Step to reproduce:
1. Set backup_sha_block_size_bytes = 32768
2. Backup full volume over 1TB --> Success (Note: the sha checksum file will be over 2GB)
3. Test create incremental backup --> Fail (log below)
4. Try to restore the (2) backup --> Fail (same log below)

It looks similar to this python bug https://bugs.python.org/issue42853

2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server [req-3b52d138-cbcb-4a62-80f2-fe19b92e8e76 c3ae4c60a7e54e3197a181cb3f971722 4dfa07e07fda4f5f8520687d4912af9a - - -] Exception during message handling: cinder.backup.drivers.s3.S3ConnectionFailure: S3 connection failure: signed integer is greater than maximum
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 163, in func_wrapper
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 375, in read
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server return resp.get('Body').read()
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/botocore/response.py", line 77, in read
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server chunk = self._raw_stream.read(amt)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/response.py", line 515, in read
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server data = self._fp.read() if not fp_closed else b""
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/http/client.py", line 472, in read
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server s = self._safe_read(self.length)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/http/client.py", line 613, in _safe_read
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server data = self.fp.read(amt)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/socket.py", line 669, in readinto
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server return self._sock.recv_into(b)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 252, in recv_into
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server return self._base_recv(nbytes, flags, into=True, buffer_=buffer)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 267, in _base_recv
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server read = self.read(nbytes, buffer_)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 186, in read
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server return self._call_trampolining(
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 161, in _call_trampolining
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server return func(*a, **kw)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/ssl.py", line 1099, in read
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server return self._sslobj.read(len, buffer)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server OverflowError: signed integer is greater than maximum
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/utils.py", line 890, in wrapper
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 410, in create_backup
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err))
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server self.force_reraise()
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server raise self.value
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 399, in create_backup
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server updates = self._run_backup(context, backup, volume)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 482, in _run_backup
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server updates = backup_service.backup(
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 517, in backup
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server parent_backup_shafile = self._read_sha256file(parent_backup)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 313, in _read_sha256file
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server sha256file_json = reader.read()
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 167, in func_wrapper
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server raise S3ConnectionFailure(reason=err)
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server cinder.backup.drivers.s3.S3ConnectionFailure: S3 connection failure: signed integer is greater than maximum
2022-08-07 01:00:42.469 693 ERROR oslo_messaging.rpc.server

Tags: backup s3
Eric Harney (eharney)
tags: added: backup s3
Changed in cinder:
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.