Cinder

backup tests failing due to timeouts/unexpected status on backup; i/o errors in c-bak logs (amqp issues)

Bug #1813217 reported by Matt Riedemann
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Confirmed
Undecided
Unassigned

Bug Description

http://logs.openstack.org/40/627540/8/gate/tempest-full/47abf62/controller/logs/screen-c-bak.txt.gz?level=TRACE

Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server [None req-5c1544a1-0dd3-4325-8a2b-f868cc8df2f0 tempest-VolumesBackupsAdminTest-1331377977 None] Exception during message handling: OSError: [Errno 5] Input/output error
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server Traceback (most recent call last):
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/manager.py", line 597, in restore_backup
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server backup.save()
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server self.force_reraise()
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/manager.py", line 586, in restore_backup
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server self._run_restore(context, backup, volume)
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/manager.py", line 644, in _run_restore
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server tpool.Proxy(device_file))
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/chunkeddriver.py", line 788, in restore
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server restore_func(backup1, volume_id, metadata, volume_file, backup)
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server File "/opt/stack/cinder/cinder/backup/chunkeddriver.py", line 733, in _restore_v1
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server os.fsync(fileno)
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server OSError: [Errno 5] Input/output error
Jan 24 20:06:02.943403 ubuntu-bionic-inap-mtl01-0002105527 cinder-backup[13809]: ERROR oslo_messaging.rpc.server

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Exception%20during%20message%20handling%3A%20OSError%3A%20%5BErrno%205%5D%20Input%2Foutput%20error%5C%22%20AND%20tags%3A%5C%22screen-c-bak.txt%5C%22&from=7d

37 hits in 7 days, check and gate, all failures.

Matt Riedemann (mriedem)
Changed in cinder:
status: New → Confirmed
tags: added: backup-service gate-failure
Revision history for this message
Matt Riedemann (mriedem) wrote :

This is likely a duplicate of bug 1483434.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.