Looking at a recent failure there is a very obvious stacktrace in the cinder-backup logs:

http://logs.openstack.org/04/649804/1/check/tempest-full/e2ad258/controller/logs/screen-c-bak.txt.gz?level=TRACE#_Apr_11_08_42_32_094879

Apr 11 08:42:32.094879 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR cinder.backup.manager [None req-1726f3a1-3b67-49d5-926c-646d59c39710 tempest-VolumesBackupsTest-391112369 None] Restoring backup 87ecac88-4dbb-4f58-b0de-3d79e4525008 to volume 883e3950-d0eb-41ba-985f-e9928c5efe38 failed.: OSError: [Errno 5] Input/output error
Apr 11 08:42:32.094879 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR cinder.backup.manager Traceback (most recent call last):
Apr 11 08:42:32.094879 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR cinder.backup.manager   File "/opt/stack/cinder/cinder/backup/manager.py", line 644, in _run_restore
Apr 11 08:42:32.094879 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR cinder.backup.manager     tpool.Proxy(device_file))
Apr 11 08:42:32.094879 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR cinder.backup.manager   File "/opt/stack/cinder/cinder/backup/chunkeddriver.py", line 788, in restore
Apr 11 08:42:32.094879 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR cinder.backup.manager     restore_func(backup1, volume_id, metadata, volume_file, backup)
Apr 11 08:42:32.094879 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR cinder.backup.manager   File "/opt/stack/cinder/cinder/backup/chunkeddriver.py", line 733, in _restore_v1
Apr 11 08:42:32.094879 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR cinder.backup.manager     os.fsync(fileno)
Apr 11 08:42:32.094879 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR cinder.backup.manager OSError: [Errno 5] Input/output error
Apr 11 08:42:32.094879 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR cinder.backup.manager 
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server [None req-1726f3a1-3b67-49d5-926c-646d59c39710 tempest-VolumesBackupsTest-391112369 None] Exception during message handling: OSError: [Errno 5] Input/output error
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/backup/manager.py", line 597, in restore_backup
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     backup.save()
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/backup/manager.py", line 586, in restore_backup
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     self._run_restore(context, backup, volume)
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/backup/manager.py", line 644, in _run_restore
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     tpool.Proxy(device_file))
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/backup/chunkeddriver.py", line 788, in restore
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     restore_func(backup1, volume_id, metadata, volume_file, backup)
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/backup/chunkeddriver.py", line 733, in _restore_v1
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     os.fsync(fileno)
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server OSError: [Errno 5] Input/output error
Apr 11 08:43:32.619862 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server 
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server [None req-52d4d53a-f756-4c79-bd59-aa9fbf3e6275 tempest-VolumesBackupsAdminTest-159108100 None] Exception during message handling: IOError: [Errno 5] Input/output error
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/backup/manager.py", line 406, in create_backup
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     self._update_backup_error(backup, six.text_type(err))
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     self.force_reraise()
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/backup/manager.py", line 395, in create_backup
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     updates = self._run_backup(context, backup, volume)
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/backup/manager.py", line 479, in _run_backup
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     backup, tpool.Proxy(device_file))
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/opt/stack/cinder/cinder/backup/chunkeddriver.py", line 584, in backup
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     data = volume_file.read(read_bytes)
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 190, in doit
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 148, in proxy_call
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 129, in execute
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server IOError: [Errno 5] Input/output error
Apr 11 08:43:32.833541 ubuntu-bionic-ovh-gra1-0004997343 cinder-backup[14242]: ERROR oslo_messaging.rpc.server 

Looks like there is some eventlet monkey business and an unhandled I/O error.