OpenStack Compute (nova)

Instance Boot from volume fails with iSER

Bug #1253578 reported by Shlomi Sasson
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Invalid
Low
Unassigned

Bug Description

steps to reproduce:
1. Create new volume with image source.
2. Launch a new instance boot from the above volume using iSER transport.

Nova-Compute side fails with the below logs:
2013-11-12 16:25:49.508 7157 INFO nova.virt.libvirt.driver [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Creating image 2013-11-12 16:25:56.048 7157 ERROR nova.compute.manager [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Instance failed to spa
wn 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Traceback (most recent call last):
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1407, in _spawn 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] block_device_info)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2070, in spawn 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] block_device_info, context=context)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3184, in _create_domain_and_network 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] disk_info)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1030, in volume_driver_method 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return method(connection_info, *args, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 246, in inner 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return f(*args, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 536, in connect_volume 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] self._connect_to_iser_portal(iser_properties)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 643, in _connect_to_iser_portal 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] "iser")
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 227, in _iscsiadm_update 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return self._run_iscsiadm(iscsi_properties, iscsi_command, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 218, in _run_iscsiadm 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] check_exit_code=check_exit_code)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/utils.py", line 177, in execute 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return processutils.execute(*cmd, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 178, in execute 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] cmd=' '.join(cmd))
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] ProcessExecutionError: Unexpected error while running command. 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm m node -T iqn.2010-10.org.iser.openstack:volume-55341fa3-cbd5-4c52-87
53-e3171dd8dcc4 -p 11.131.1.100:3260 --op update -n iface.transport_name -v iser 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Exit code: 15
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Stdout: '' 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Stderr: 'iscsiadm: Cannot modify node/iface transport name while a session is using it. Log out the session then update record.\n'
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] 2013-11-12 16:25:56.218 7157 AUDIT nova.compute.manager [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Terminating instance
2013-11-12 16:25:57.157 7157 ERROR nova.virt.libvirt.driver [] [instance: 19646297-0eec-4182-9c17-d9172faf5663] During wait destroy, instance disappeared. 2013-11-12 16:25:57.667 7157 INFO nova.virt.libvirt.driver [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Deleting instance files /var/lib/nova/instances/19646297-0eec-4182-9c17-d9172faf5663
2013-11-12 16:25:57.668 7157 INFO nova.virt.libvirt.driver [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Deletion of /var/lib/nova/instances/19646297-0eec-4182-9c17-d9172faf5663 complete

This happens due to attempt to change iface transport while a session is active.

See original description
Shlomi Sasson (shlomis)
Changed in nova:
assignee: nobody → Shlomi Sasson (shlomis)
Itzik Brown (itzikb1)
Changed in nova:
status: New → Confirmed
Shlomi Sasson (shlomis)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/havana)

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/58138

Shlomi Sasson (shlomis)
Changed in nova:
status: Confirmed → Fix Committed
Revision history for this message
Thierry Carrez (ttx) wrote :

Could not find the commit that closes this, please add it before setting to FixCommitted again

Changed in nova:
status: Fix Committed → Triaged
Revision history for this message
Sean Dague (sdague) wrote :

Does this remain an issue 1 year later

Changed in nova:
status: Triaged → Incomplete
importance: Undecided → Low
assignee: Shlomi Sasson (shlomis) → nobody
Sean Dague (sdague)
Changed in nova:
status: Incomplete → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.