When reading volume file as a chunk, sometimes OSError is raised.
I cannot find the rootcause for this problems. If this due to hardwares or disk write/read problem, can we implement a retry mechanism?
2024-05-29 17:33:50.022 672 ERROR root [req-721220e3-ad6e-4b1c-a05f-f164d42c1d47 c6d27dedb5ca466181b1cfe34f513144 a1e905408c304b5abba8edd635ce2465 - - -] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 492, in continue_backup\n updates = backup_service.backup(\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 725, in backup\n data = volume_file.read(read_bytes)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call\n rv = execute(f, *args, **kwargs)\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute\n six.reraise(c, e, tb)\n', ' File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker\n rv = meth(*args, **kwargs)\n', 'OSError: [Errno 5] Input/output error\n']
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server [req-721220e3-ad6e-4b1c-a05f-f164d42c1d47 c6d27dedb5ca466181b1cfe34f513144 a1e905408c304b5abba8edd635ce2465 - - -] Exception during message handling: TypeError: 'NoneType' object is not subscriptable
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 492, in continue_backup
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server updates = backup_service.backup(
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/chunkeddriver.py", line 725, in backup
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server data = volume_file.read(read_bytes)
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 193, in doit
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server result = proxy_call(self._autowrap, f, *args, **kwargs)
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 151, in proxy_call
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server rv = execute(f, *args, **kwargs)
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 132, in execute
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb)
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 719, in reraise
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server raise value
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/eventlet/tpool.py", line 86, in tworker
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs)
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server OSError: [Errno 5] Input/output error
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 530, in continue_backup
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server volume_utils.update_backup_error(backup, str(err))
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server self.force_reraise()
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server raise self.value
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/backup/manager.py", line 502, in continue_backup
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server self.message_api.create_from_request_context(
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/message/api.py", line 140, in create_from_request_context
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server self.create(context=context,
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/message/api.py", line 104, in create
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server action[0],
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not subscriptable
2024-05-29 17:33:50.806 672 ERROR oslo_messaging.rpc.server
