| 2013-11-21 12:00:50 |
Shlomi Sasson |
description |
steps to reproduce:
1. Create new volume with image source.
2. Launch a new instance from the above volume.
Nova-Compute side fails with the below logs:
2013-11-12 16:25:49.508 7157 INFO nova.virt.libvirt.driver [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Creating image 2013-11-12 16:25:56.048 7157 ERROR nova.compute.manager [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Instance failed to spa
wn 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Traceback (most recent call last):
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1407, in _spawn 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] block_device_info)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2070, in spawn 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] block_device_info, context=context)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3184, in _create_domain_and_network 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] disk_info)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1030, in volume_driver_method 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return method(connection_info, *args, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 246, in inner 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return f(*args, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 536, in connect_volume 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] self._connect_to_iser_portal(iser_properties)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 643, in _connect_to_iser_portal 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] "iser")
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 227, in _iscsiadm_update 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return self._run_iscsiadm(iscsi_properties, iscsi_command, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 218, in _run_iscsiadm 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] check_exit_code=check_exit_code)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/utils.py", line 177, in execute 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return processutils.execute(*cmd, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 178, in execute 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] cmd=' '.join(cmd))
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] ProcessExecutionError: Unexpected error while running command. 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm m node -T iqn.2010-10.org.iser.openstack:volume-55341fa3-cbd5-4c52-87
53-e3171dd8dcc4 -p 11.131.1.100:3260 --op update -n iface.transport_name -v iser 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Exit code: 15
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Stdout: '' 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Stderr: 'iscsiadm: Cannot modify node/iface transport name while a session is using it. Log out the session then update record.\n'
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] 2013-11-12 16:25:56.218 7157 AUDIT nova.compute.manager [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Terminating instance
2013-11-12 16:25:57.157 7157 ERROR nova.virt.libvirt.driver [] [instance: 19646297-0eec-4182-9c17-d9172faf5663] During wait destroy, instance disappeared. 2013-11-12 16:25:57.667 7157 INFO nova.virt.libvirt.driver [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Deleting instance files /var/lib/nova/instances/19646297-0eec-4182-9c17-d9172faf5663
2013-11-12 16:25:57.668 7157 INFO nova.virt.libvirt.driver [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Deletion of /var/lib/nova/instances/19646297-0eec-4182-9c17-d9172faf5663 complete
This happens due to attempt to change iface transport while a session is active. |
steps to reproduce:
1. Create new volume with image source.
2. Launch a new instance boot from the above volume using iSER transport.
Nova-Compute side fails with the below logs:
2013-11-12 16:25:49.508 7157 INFO nova.virt.libvirt.driver [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Creating image 2013-11-12 16:25:56.048 7157 ERROR nova.compute.manager [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Instance failed to spa
wn 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Traceback (most recent call last):
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1407, in _spawn 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] block_device_info)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2070, in spawn 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] block_device_info, context=context)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3184, in _create_domain_and_network 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] disk_info)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1030, in volume_driver_method 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return method(connection_info, *args, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 246, in inner 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return f(*args, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 536, in connect_volume 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] self._connect_to_iser_portal(iser_properties)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 643, in _connect_to_iser_portal 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] "iser")
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 227, in _iscsiadm_update 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return self._run_iscsiadm(iscsi_properties, iscsi_command, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 218, in _run_iscsiadm 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] check_exit_code=check_exit_code)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/utils.py", line 177, in execute 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] return processutils.execute(*cmd, **kwargs)
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 178, in execute 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] cmd=' '.join(cmd))
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] ProcessExecutionError: Unexpected error while running command. 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm m node -T iqn.2010-10.org.iser.openstack:volume-55341fa3-cbd5-4c52-87
53-e3171dd8dcc4 -p 11.131.1.100:3260 --op update -n iface.transport_name -v iser 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Exit code: 15
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Stdout: '' 2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] Stderr: 'iscsiadm: Cannot modify node/iface transport name while a session is using it. Log out the session then update record.\n'
2013-11-12 16:25:56.048 7157 TRACE nova.compute.manager [instance: 19646297-0eec-4182-9c17-d9172faf5663] 2013-11-12 16:25:56.218 7157 AUDIT nova.compute.manager [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Terminating instance
2013-11-12 16:25:57.157 7157 ERROR nova.virt.libvirt.driver [] [instance: 19646297-0eec-4182-9c17-d9172faf5663] During wait destroy, instance disappeared. 2013-11-12 16:25:57.667 7157 INFO nova.virt.libvirt.driver [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Deleting instance files /var/lib/nova/instances/19646297-0eec-4182-9c17-d9172faf5663
2013-11-12 16:25:57.668 7157 INFO nova.virt.libvirt.driver [req-233b2283-94b3-4340-b77e-76f4efbcc2c4 2cb6dbc75b0e4df1aed19b1253f5344e 330f12e1871348e09e03ffea2e9523ff] [instance: 19646297-0eec-4182-9c17-d9172faf5663] Deletion of /var/lib/nova/instances/19646297-0eec-4182-9c17-d9172faf5663 complete
This happens due to attempt to change iface transport while a session is active. |
|
