Attaching Nova volumes fails if open-iscsi daemon is not already running

Bug #1308649 reported by Dirk Mueller
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Low
Sean Dague

Bug Description

nova-compute traces on first volume attach if open-iscsi is not running:

2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] Traceback (most recent call last):
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4142, in _attach_volume
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] do_check_attach=False, do_driver_attach=True)
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/virt/block_device.py", line 44, in wrapped
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] ret_val = method(obj, context, *args, **kwargs)
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/virt/block_device.py", line 248, in attach
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] connector)
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] six.reraise(self.type_, self.value, self.tb)
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/virt/block_device.py", line 239, in attach
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] device_type=self['device_type'], encryption=encryption)
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1224, in attach_volume
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] disk_info)
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1183, in volume_driver_method
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] return method(connection_info, *args, **kwargs)
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] return f(*args, **kwargs)
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/contextlib.py", line 34, in __exit__
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] self.gen.throw(type, value, traceback)
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/openstack/common/lockutils.py", line 212, in lock
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] yield sem
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] return f(*args, **kwargs)
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] File "/usr/lib64/python2.6/site-packages/nova/virt/libvirt/volume.py", line 285, in connect_volume
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] for ip, iqn in self._get_target_portals_from_iscsiadm_output(out):
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481] ValueError: too many values to unpack
2014-04-16 13:19:44.359 6079 TRACE nova.compute.manager [instance: ddc07867-67a9-4af7-b7d0-d75dff53b481]

Thats because it runs "iscsiadm -m discovery -t sendtargets -p", which prints upon first run:

Loading iscsi modules: done
Starting iSCSI initiator service: done
Setting up iSCSI targets: unused
192.168.204.82:3260,1 iqn.2010-10.org.openstack:volume-f9b12623-6ce3-4dac-a71f-09ad4249bdd3
192.168.204.82:3260,1 iqn.2010-10.org.openstack:volume-f9b12623-6ce3-4dac-a71f-09ad4249bdd3

aka there is some garbage before the actual output that it looks for. this is due to /etc/iscsid.conf having configured to start the daemon:

from /etc/iscsid.conf:

# Default for upstream open-iscsi scripts (uncomment to activate).
iscsid.startup = /usr/sbin/rcopen-iscsi start

Tags: volumes
Dirk Mueller (dmllr)
Changed in nova:
assignee: nobody → Dirk Mueller (dmllr)
Tracy Jones (tjones-i)
tags: added: volumes
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/120958

Changed in nova:
assignee: Dirk Mueller (dmllr) → Sean Dague (sdague)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/120958
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4ab7b6b7ad8b6047ec108c8f98bb1c0f310b9899
Submitter: Jenkins
Branch: master

commit 4ab7b6b7ad8b6047ec108c8f98bb1c0f310b9899
Author: Sean Dague <email address hidden>
Date: Thu Sep 11 21:13:03 2014 -0400

    Fix parsing sloppiness from iscsiadm discover

    As reported by a user, iscsiadm might spew some other random gorp when
    it starts up, and if it does, nova explodes poorly. Put a modicum of
    sanity testing into the parsing of the output to make this less
    likely.

    Change-Id: Iaf55438bb3ec412187158e80f9fbc1d8e7071207
    Closes-Bug: #1308649

Changed in nova:
status: In Progress → Fix Committed
Sean Dague (sdague)
Changed in nova:
importance: Undecided → Low
Thierry Carrez (ttx)
Changed in nova:
milestone: none → juno-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-rc1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.