os-brick

Bug #1461308
Comment #3

Comment 3 for bug 1461308

Revision history for this message

John Griffith (john-griffith) wrote on 2015-06-26:

Looks like it didn't fix things anyway. Have seen this in the RBD gate test, but also locally on CI systems a few times this week.

2015-06-26 11:06:35.905 ERROR cinder.volume.flows.manager.create_volume [req-d786eb8d-faf2-4904-af22-616008f0aa2a None] Failed to copy image 190d5982-24b2-4b24-aaa7-2e88bce83d35 to volume: d0437a9f-d181-41e6-93d3-f88803c323b5
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume Traceback (most recent call last):
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 515, in _copy_image_to_volume
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume copy_image_to_volume(context, volume_ref, image_service, image_id)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume File "/opt/stack/cinder/cinder/volume/driver.py", line 568, in copy_image_to_volume
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume attach_info, volume = self._attach_volume(context, volume, properties)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume File "/opt/stack/cinder/cinder/volume/driver.py", line 712, in _attach_volume
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume return (self._connect_device(conn), volume)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume File "/opt/stack/cinder/cinder/volume/driver.py", line 724, in _connect_device
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume device = connector.connect_volume(conn['data'])
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 251, in inner
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume return f(*args, **kwargs)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 320, in connect_volume
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume if self._connect_to_iscsi_portal(props):
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 551, in _connect_to_iscsi_portal
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume self._run_iscsiadm(connection_properties, ())
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 468, in _run_iscsiadm
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume check_exit_code=check_exit_code)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 260, in execute
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume cmd=sanitized_cmd)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume ProcessExecutionError: Unexpected error while running command.
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf iscsiadm -m node -T iqn.2010-01.com.solidfire:l74j.uuid-d0437a9f-d181-41e6-93d3-f88803c323b5.120446 -p 10.10.64.3:3260
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume Exit code: 6
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume Stdout: u''
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume Stderr: u'iscsiadm: Could not execute operation on all records: encountered iSCSI database failure\n'
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume
2015-06-26 11:06:35.915 WARNING cinder.volume.manager [req-d786eb8d-faf2-4904-af22-616008f0aa2a None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7d5b6af9-1815-4211-8ec5-0997056ea731) transitioned into state 'FAILURE' from state 'RUNNING'
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager Traceback (most recent call last):
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 67, in _execute_task
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager result = task.execute(**arguments)
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 664, in execute
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager **volume_spec)
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 619, in _create_from_image
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager image_id, image_location, image_service)
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 520, in _copy_image_to_volume
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager raise exception.ImageCopyFailure(reason=ex.stderr)
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager ImageCopyFailure: Failed to copy image to volume: iscsiadm: Could not execute operation on all records: encountered iSCSI database failure
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager
2015-06-26 11:06:35.915 DEBUG taskflow.engines.action_engine.completer [req-d786eb8d-faf2-4904-af22-616008f0aa2a None] Applying resolver '_RevertAll(strategy=REVERT_ALL)' to resolve failure 'Failure: cinder.exception.ImageCopyFailure: Failed to copy image to volume: iscsiadm: Could not execute operation on all records: encountered iSCSI database failure

Looks like it didn't fix things anyway.  Have seen this in the RBD gate test, but also locally on CI systems a few times this week.

2015-06-26 11:06:35.905 ERROR cinder.volume.flows.manager.create_volume [req-d786eb8d-faf2-4904-af22-616008f0aa2a None] Failed to copy image 190d5982-24b2-4b24-aaa7-2e88bce83d35 to volume: d0437a9f-d181-41e6-93d3-f88803c323b5
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume Traceback (most recent call last):
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 515, in _copy_image_to_volume
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume     copy_image_to_volume(context, volume_ref, image_service, image_id)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume     return f(*args, **kwargs)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume   File "/opt/stack/cinder/cinder/volume/driver.py", line 568, in copy_image_to_volume
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume     attach_info, volume = self._attach_volume(context, volume, properties)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume   File "/opt/stack/cinder/cinder/volume/driver.py", line 712, in _attach_volume
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume     return (self._connect_device(conn), volume)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume   File "/opt/stack/cinder/cinder/volume/driver.py", line 724, in _connect_device
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume     device = connector.connect_volume(conn['data'])
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 251, in inner
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume     return f(*args, **kwargs)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume   File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 320, in connect_volume
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume     if self._connect_to_iscsi_portal(props):
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume   File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 551, in _connect_to_iscsi_portal
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume     self._run_iscsiadm(connection_properties, ())
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume   File "/usr/local/lib/python2.7/dist-packages/os_brick/initiator/connector.py", line 468, in _run_iscsiadm
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume     check_exit_code=check_exit_code)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 260, in execute
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume     cmd=sanitized_cmd)
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume ProcessExecutionError: Unexpected error while running command.
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf iscsiadm -m node -T iqn.2010-01.com.solidfire:l74j.uuid-d0437a9f-d181-41e6-93d3-f88803c323b5.120446 -p 10.10.64.3:3260
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume Exit code: 6
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume Stdout: u''
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume Stderr: u'iscsiadm: Could not execute operation on all records: encountered iSCSI database failure\n'
2015-06-26 11:06:35.905 11515 ERROR cinder.volume.flows.manager.create_volume 
2015-06-26 11:06:35.915 WARNING cinder.volume.manager [req-d786eb8d-faf2-4904-af22-616008f0aa2a None] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (7d5b6af9-1815-4211-8ec5-0997056ea731) transitioned into state 'FAILURE' from state 'RUNNING'
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager Traceback (most recent call last):
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager   File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 67, in _execute_task
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager     result = task.execute(**arguments)
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 664, in execute
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager     **volume_spec)
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 619, in _create_from_image
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager     image_id, image_location, image_service)
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager   File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 520, in _copy_image_to_volume
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager     raise exception.ImageCopyFailure(reason=ex.stderr)
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager ImageCopyFailure: Failed to copy image to volume: iscsiadm: Could not execute operation on all records: encountered iSCSI database failure
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager 
2015-06-26 11:06:35.915 11515 ERROR cinder.volume.manager 
2015-06-26 11:06:35.915 DEBUG taskflow.engines.action_engine.completer [req-d786eb8d-faf2-4904-af22-616008f0aa2a None] Applying resolver '_RevertAll(strategy=REVERT_ALL)' to resolve failure 'Failure: cinder.exception.ImageCopyFailure: Failed to copy image to volume: iscsiadm: Could not execute operation on all records: encountered iSCSI database failure