Environment:
- OpenStack: yoga
- cinder-backup driver: cinder.backup.drivers.s3.S3BackupDriver
Step to reproduce:
1. Create volume size >= 2TB
2. Backup full volume with cinder.backup.drivers.s3.S3BackupDriver (Success)
3. Backup incremental with the parent from the previous step.
3. Volume backup error with the logs as follows
2023-09-22 01:15:27.823 666 ERROR root [req-38ce711e-599c-45f5-bc47-563194445146 c6d27dedb5ca466181b1cfe34f513144 646c2722c6d5434996043a33900f8948 - - -] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 163, in func_wrapper\n return func(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 377, in read\n return resp.get(\'Body\').read()\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/botocore/response.py", line 95, in read\n chunk = self._raw_stream.read(amt)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/response.py", line 515, in read\n data = self._fp.read() if not fp_closed else b""\n', ' File "/usr/lib/python3.8/http/client.py", line 472, in read\n s = self._safe_read(self.length)\n', ' File "/usr/lib/python3.8/http/client.py", line 613, in _safe_read\n data = self.fp.read(amt)\n', ' File "/usr/lib/python3.8/socket.py", line 669, in readinto\n return self._sock.recv_into(b)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 253, in recv_into\n return self._base_recv(nbytes, flags, into=True, buffer_=buffer)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 268, in _base_recv\n read = self.read(nbytes, buffer_)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 187, in read\n return self._call_trampolining(\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining\n return func(*a, **kw)\n', ' File "/usr/lib/python3.8/ssl.py", line 1099, in read\n return self._sslobj.read(len, buffer)\n', 'OverflowError: signed integer is greater than maximum\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 488, in continue_backup\n updates = backup_service.backup(\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 653, in backup\n parent_backup_shafile = self._read_sha256file(parent_backup)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 326, in _read_sha256file\n sha256file_json = reader.read()\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 167, in func_wrapper\n raise S3ConnectionFailure(reason=err)\n', 'cinder.backup.drivers.s3.S3ConnectionFailure: S3 connection failure: signed integer is greater than maximum\n']
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server [req-38ce711e-599c-45f5-bc47-563194445146 c6d27dedb5ca466181b1cfe34f513144 646c2722c6d5434996043a33900f8948 - - -] Exception during message handling: TypeError: 'NoneType' object is not subscriptable
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 163, in func_wrapper
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 377, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return resp.get('Body').read()
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/botocore/response.py", line 95, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server chunk = self._raw_stream.read(amt)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/response.py", line 515, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server data = self._fp.read() if not fp_closed else b""
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/http/client.py", line 472, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server s = self._safe_read(self.length)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/http/client.py", line 613, in _safe_read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server data = self.fp.read(amt)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/socket.py", line 669, in readinto
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return self._sock.recv_into(b)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 253, in recv_into
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return self._base_recv(nbytes, flags, into=True, buffer_=buffer)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 268, in _base_recv
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server read = self.read(nbytes, buffer_)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 187, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return self._call_trampolining(
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return func(*a, **kw)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/ssl.py", line 1099, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return self._sslobj.read(len, buffer)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server OverflowError: signed integer is greater than maximum
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 488, in continue_backup
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server updates = backup_service.backup(
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 653, in backup
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server parent_backup_shafile = self._read_sha256file(parent_backup)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 326, in _read_sha256file
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server sha256file_json = reader.read()
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 167, in func_wrapper
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server raise S3ConnectionFailure(reason=err)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server cinder.backup.drivers.s3.S3ConnectionFailure: S3 connection failure: signed integer is greater than maximum
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 526, in continue_backup
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err))
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server raise self.value
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 498, in continue_backup
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context(
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/message/api.py", line 140, in create_from_request_context
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server self.create(context=context,
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/message/api.py", line 104, in create
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server action[0],
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server
Fix proposed to branch: master /review. opendev. org/c/openstack /cinder/ +/911411
Review: https:/