Cinder

Cinder backup fail with s3 driver and volume size exceed 2TB

Bug #2038280 reported by Sang Tran
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Cinder
In Progress
Undecided
Unassigned

Bug Description

Environment:
- OpenStack: yoga
- cinder-backup driver: cinder.backup.drivers.s3.S3BackupDriver

Step to reproduce:
1. Create volume size >= 2TB
2. Backup full volume with cinder.backup.drivers.s3.S3BackupDriver (Success)
3. Backup incremental with the parent from the previous step.
3. Volume backup error with the logs as follows

2023-09-22 01:15:27.823 666 ERROR root [req-38ce711e-599c-45f5-bc47-563194445146 c6d27dedb5ca466181b1cfe34f513144 646c2722c6d5434996043a33900f8948 - - -] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 163, in func_wrapper\n return func(*args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 377, in read\n return resp.get(\'Body\').read()\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/botocore/response.py", line 95, in read\n chunk = self._raw_stream.read(amt)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/response.py", line 515, in read\n data = self._fp.read() if not fp_closed else b""\n', ' File "/usr/lib/python3.8/http/client.py", line 472, in read\n s = self._safe_read(self.length)\n', ' File "/usr/lib/python3.8/http/client.py", line 613, in _safe_read\n data = self.fp.read(amt)\n', ' File "/usr/lib/python3.8/socket.py", line 669, in readinto\n return self._sock.recv_into(b)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 253, in recv_into\n return self._base_recv(nbytes, flags, into=True, buffer_=buffer)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 268, in _base_recv\n read = self.read(nbytes, buffer_)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 187, in read\n return self._call_trampolining(\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining\n return func(*a, **kw)\n', ' File "/usr/lib/python3.8/ssl.py", line 1099, in read\n return self._sslobj.read(len, buffer)\n', 'OverflowError: signed integer is greater than maximum\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 488, in continue_backup\n updates = backup_service.backup(\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 653, in backup\n parent_backup_shafile = self._read_sha256file(parent_backup)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 326, in _read_sha256file\n sha256file_json = reader.read()\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 167, in func_wrapper\n raise S3ConnectionFailure(reason=err)\n', 'cinder.backup.drivers.s3.S3ConnectionFailure: S3 connection failure: signed integer is greater than maximum\n']
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server [req-38ce711e-599c-45f5-bc47-563194445146 c6d27dedb5ca466181b1cfe34f513144 646c2722c6d5434996043a33900f8948 - - -] Exception during message handling: TypeError: 'NoneType' object is not subscriptable
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 163, in func_wrapper
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 377, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return resp.get('Body').read()
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/botocore/response.py", line 95, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server chunk = self._raw_stream.read(amt)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/urllib3/response.py", line 515, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server data = self._fp.read() if not fp_closed else b""
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/http/client.py", line 472, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server s = self._safe_read(self.length)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/http/client.py", line 613, in _safe_read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server data = self.fp.read(amt)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/socket.py", line 669, in readinto
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return self._sock.recv_into(b)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 253, in recv_into
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return self._base_recv(nbytes, flags, into=True, buffer_=buffer)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 268, in _base_recv
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server read = self.read(nbytes, buffer_)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 187, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return self._call_trampolining(
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/green/ssl.py", line 162, in _call_trampolining
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return func(*a, **kw)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.8/ssl.py", line 1099, in read
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return self._sslobj.read(len, buffer)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server OverflowError: signed integer is greater than maximum
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 488, in continue_backup
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server updates = backup_service.backup(
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 653, in backup
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server parent_backup_shafile = self._read_sha256file(parent_backup)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 326, in _read_sha256file
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server sha256file_json = reader.read()
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/drivers/s3.py", line 167, in func_wrapper
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server raise S3ConnectionFailure(reason=err)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server cinder.backup.drivers.s3.S3ConnectionFailure: S3 connection failure: signed integer is greater than maximum
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 526, in continue_backup
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err))
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server raise self.value
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 498, in continue_backup
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context(
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/message/api.py", line 140, in create_from_request_context
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server self.create(context=context,
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/message/api.py", line 104, in create
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server action[0],
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable
2023-09-22 01:15:30.530 666 ERROR oslo_messaging.rpc.server

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/911411

Changed in cinder:
status: New → In Progress
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.