KeyError: 'sr_uuid' when booting from volume on xenapi

Bug #1087308 reported by Armando Migliaccio
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Armando Migliaccio

Bug Description

I get this stack trace when I try the 'boot from volume feature' on Grizzly (Cinder configured with the iscsi Driver):

2012-11-19 18:01:55 DEBUG nova.openstack.common.lockutils [req-75241d7f-d5d9-4a88-9d38-70b50db25f36 demo demo] Got semaphore "compute_resources" for method "update_usage"... inner /opt/stack/nova/nova/openstack/common/lockutils.py:185
2012-11-19 18:01:55 2475 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 115, in wrapped
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in _exit_
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp return f(*args, **kw)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 176, in decorated_function
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp pass
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in _exit_
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 162, in decorated_function
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 203, in decorated_function
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp kwargs['instance']['uuid'], e, sys.exc_info())
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in _exit_
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 191, in decorated_function
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 933, in run_instance
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp do_run_instance()
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 228, in inner
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 932, in do_run_instance
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, instance)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 608, in _run_instance
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in _exit_
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 596, in _run_instance
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp is_first_time, request_spec, filter_properties)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 582, in _run_instance
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp injected_files, admin_password)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 840, in _spawn
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp block_device_info)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/xenapi/driver.py", line 172, in spawn
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp admin_password, network_info, block_device_info)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/xenapi/vmops.py", line 408, in spawn
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp undo_mgr.rollback_and_reraise(msg=msg, instance=instance)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/utils.py", line 1154, in rollback_and_reraise
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp self._rollback()
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in _exit_
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/xenapi/vmops.py", line 391, in spawn
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp vdis = create_disks_step(undo_mgr, disk_image_type, image_meta)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/xenapi/vmops.py", line 134, in inner
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp rv = f(*args, **kwargs)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/xenapi/vmops.py", line 275, in create_disks_step
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp block_device_info)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/xenapi/vmops.py", line 246, in _create_disks
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp block_device_info=block_device_info)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/xenapi/vm_utils.py", line 520, in get_vdis_for_instance
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp dev_params)
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/xenapi/vm_utils.py", line 458, in get_vdis_for_boot_from_vol
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp sr_uuid = dev_params['sr_uuid']
2012-11-19 18:01:55 2475 TRACE nova.openstack.common.rpc.amqp KeyError: 'sr_uuid'

I get the same error with Folsom stable (with slightly different line numbers).

I think this has to do with the fact that the definition of get_vdis_for_boot_from_vol is tightly coupled with the XenAPI Storage Manager back-end (which is the one where keys like sr_uuid are coming from); however I see no good reason why this is. I think this can safely work with an iscsi driver.

I know that there's some work going on in refactoring how xenapi handles volume-related requests, so I would suggest that the Grizzly and Folsom fixes follow different paths.

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

I think I have a fix for Folsom stable, I'll be pushing it shortly.

Changed in nova:
assignee: nobody → Armando Migliaccio (armando-migliaccio)
Tom Fifield (fifieldt)
Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/17726
Committed: http://github.com/openstack/nova/commit/503d5729547f135951d75d5237b398fe1700aaf2
Submitter: Jenkins
Branch: master

commit 503d5729547f135951d75d5237b398fe1700aaf2
Author: Armando Migliaccio <email address hidden>
Date: Fri Dec 7 21:06:43 2012 +0000

    Fixes KeyError: 'sr_uuid' when booting from volume on xenapi

    This change fixes the capability for a XenServer/XCP hypervisor
    to boot instances from an iscsi volume running on Cinder.

    This patch takes into account that Cinder can be configured
    with an iscsi back-end, instead of a xensm one, where keys like
    sr_uuid are not being specified. It works through the magic of
    determining the SR, and the vdi associated with the iscsi target
    and lun.

    This patch also does a bit of clean-up of code/comments that no
    longer apply.

    Addresses bug #1087308

    Change-Id: Ie7d65eeb965b3468e4407981788725e2b43bff5e

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

A patch for the folsom stable branch

Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-2 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.