Cinder

Failed to copy image to volume: qemu-img: error writing zeroes at sector 0: Input/output error

Bug #1568727 reported by changxh on 2016-04-11

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Cinder	Invalid	Undecided	Unassigned

Bug Description

When I attempt to create a volume from image, I get this error message "Failed to copy image to volume: qemu-img: error writing zeroes at sector 0: Input/output error". Sometimes it will be succeed, but other times it will not (with the same image). But reboot the storage node, it will success.

ubunt 14.04.3
openstack kilo
backend huawei v3 5600
FC

2016-04-06 09:49:16.380 59750 INFO cinder.volume.drivers.huawei.huawei_driver [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] terminate_connection_fc, return data is: {'driver_volume_type': 'fibre_channel', 'data': {}}.
2016-04-06 09:49:16.381 59750 INFO cinder.volume.drivers.huawei.huawei_driver [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Detach Volume, metadata is: {}.
2016-04-06 09:49:16.382 59750 ERROR cinder.volume.flows.manager.create_volume [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Failed to copy image fdd1363f-f1df-4907-bb00-25bafd7b0c44 to volume: 91268015-7a4a-4b34-b73f-c30179c89251
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume Traceback (most recent call last):
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 501, in _copy_image_to_volume
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume copy_image_to_volume(context, volume_ref, image_service, image_id)
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume return f(*args, **kwargs)
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/dist-packages/cinder/volume/driver.py", line 561, in copy_image_to_volume
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume size=volume['size'])
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 226, in fetch_to_raw
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume run_as_root=run_as_root)
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 317, in fetch_to_volume_format
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume run_as_root=run_as_root)
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 147, in convert_image
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume out_format, run_as_root=run_as_root)
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/dist-packages/cinder/image/image_utils.py", line 121, in _convert_image
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume utils.execute(*cmd, run_as_root=run_as_root)
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/dist-packages/cinder/utils.py", line 143, in execute
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume return processutils.execute(*cmd, **kwargs)
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 266, in execute
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume cmd=sanitized_cmd)
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume ProcessExecutionError: Unexpected error while running command.
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf qemu-img convert -t none -O raw /var/lib/cinder/conversion/tmpzeZ4gu /dev/disk/by-path/pci-0000:04:00.5-fc-0x260990032590dfd9-lun-2
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume Exit code: 1
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume Stdout: u''
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume Stderr: u'qemu-img: error writing zeroes at sector 0: Input/output error\n'
2016-04-06 09:49:16.382 59750 TRACE cinder.volume.flows.manager.create_volume
2016-04-06 09:49:16.389 59750 WARNING cinder.volume.manager [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fb075799-6b23-4ddb-98b3-509237b39606) transitioned into state 'FAILURE' from state 'RUNNING'
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager Traceback (most recent call last):
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 67, in _execute_task
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager result = task.execute(**arguments)
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 653, in execute
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager **volume_spec)
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 605, in _create_from_image
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager image_id, image_location, image_service)
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 506, in _copy_image_to_volume
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager raise exception.ImageCopyFailure(reason=ex.stderr)
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager ImageCopyFailure: Failed to copy image to volume: qemu-img: error writing zeroes at sector 0: Input/output error
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager
2016-04-06 09:49:16.389 59750 TRACE cinder.volume.manager
2016-04-06 09:49:16.393 59750 WARNING cinder.volume.manager [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (fb075799-6b23-4ddb-98b3-509237b39606) transitioned into state 'REVERTED' from state 'REVERTING'
2016-04-06 09:49:16.396 59750 WARNING cinder.volume.manager [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (c1c99d5a-c25a-4c3b-8331-689d10ea5c7f) transitioned into state 'REVERTED' from state 'REVERTING'
2016-04-06 09:49:16.398 59750 WARNING cinder.volume.manager [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (64f35edf-9b93-4576-b404-007576ed981f) transitioned into state 'REVERTED' from state 'REVERTING'
2016-04-06 09:49:16.401 59750 WARNING cinder.volume.manager [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (dc860fbb-888c-424f-8239-336b214744c7) transitioned into state 'REVERTED' from state 'REVERTING'
2016-04-06 09:49:16.482 59750 ERROR cinder.volume.flows.manager.create_volume [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Volume 91268015-7a4a-4b34-b73f-c30179c89251: create failed
2016-04-06 09:49:16.483 59750 WARNING cinder.volume.manager [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e6599d7f-35ea-486b-a0ab-c325e958f490) transitioned into state 'REVERTED' from state 'REVERTING'
2016-04-06 09:49:16.485 59750 WARNING cinder.volume.manager [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Flow 'volume_create_manager' (850b4e24-2dd7-43c0-85bf-9470277eec00) transitioned into state 'REVERTED' from state 'RUNNING'
2016-04-06 09:49:16.536 59750 ERROR oslo_messaging.rpc.dispatcher [req-3e5eda3c-8b43-4f18-9b0f-2c3ab5b44d28 - - - - -] Exception during message handling: Failed to copy image to volume: qemu-img: error writing zeroes at sector 0: Input/output error
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 468, in create_volume
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher _run_flow()
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 456, in _run_flow
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher flow_engine.run()
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 96, in run
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher for _state in self.run_iter():
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 153, in run_iter
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher failure.Failure.reraise_if_any(failures.values())
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/types/failure.py", line 244, in reraise_if_any
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher failures[0].reraise()
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/types/failure.py", line 251, in reraise
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher six.reraise(*self._exc_info)
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 67, in _execute_task
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher result = task.execute(**arguments)
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 653, in execute
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher **volume_spec)
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 605, in _create_from_image
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher image_id, image_location, image_service)
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/flows/manager/create_volume.py", line 506, in _copy_image_to_volume
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher raise exception.ImageCopyFailure(reason=ex.stderr)
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher ImageCopyFailure: Failed to copy image to volume: qemu-img: error writing zeroes at sector 0: Input/output error
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher
2016-04-06 09:49:16.536 59750 TRACE oslo_messaging.rpc.dispatcher
2016-04-06 09:49:29.472 59750 INFO cinder.volume.manager [req-54fc88b1-f472-4c98-ab4a-4231a2a16953 eaec0399b8c84b5eb657ff0dbaf69aab b40b0627516647209de3d8dcf5f898a5 - - -] volume 91268015-7a4a-4b34-b73f-c30179c89251: deleting
2016-04-06 09:49:29.474 59750 INFO cinder.volume.drivers.huawei.huawei_driver [req-54fc88b1-f472-4c98-ab4a-4231a2a16953 eaec0399b8c84b5eb657ff0dbaf69aab b40b0627516647209de3d8dcf5f898a5 - - -] Delete volume: kSaAFXpKSzS3P8MBeciSUQ, array lun id: 23.
2016-04-06 09:49:29.537 59750 INFO cinder.volume.drivers.huawei.rest_client [req-54fc88b1-f472-4c98-ab4a-4231a2a16953 eaec0399b8c84b5eb657ff0dbaf69aab b40b0627516647209de3d8dcf5f898a5 - - -]

See original description

Tags:

changxh (1558187275-s) on 2016-04-11

description:

updated

Mitsuhiro Tanino (mitsuhiro-tanino) on 2016-04-11

tags:

added: huawei qemu-img

Revision history for this message

Mitsuhiro Tanino (mitsuhiro-tanino) wrote on 2016-04-11:

Similar bug was reported here. https://bugs.launchpad.net/cinder/+bug/1531711
Seems this might be timing issue but root cause is not clear.

Revision history for this message

Mitsuhiro Tanino (mitsuhiro-tanino) wrote on 2016-05-15:

FYI, Nimble Array will provide software update for the problem because the root cause of their problem was that WRITE SAME SCSI Command is handled properly on their storage.

I suppose root cause of this bug is on the storage side too, so please contact to the Huawei driver team. I'll close this bug as an Invalid but if you need further help, please reopen the bug.

Thanks,

Changed in cinder:
status:	New → Invalid

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.