DevicePathInUse exception in devstack-vm-quantum

[Ben Nemec]

I just got this during verification of one of my changes. I don't think it's related to the change (https://review.openstack.org/#/c/33478/) so I'm reporting it here before I reverify.

Full log: http://logs.openstack.org/33478/1/gate/gate-tempest-devstack-vm-quantum/32609/logs/screen-n-cpu.txt.gz

Also, this was for stable/grizzly. I'm not sure how to specify that in LP.

2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/exception.py", line 117, in wrapped
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/exception.py", line 94, in wrapped
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 209, in decorated_function
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp pass
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 195, in decorated_function
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 237, in decorated_function
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 224, in decorated_function
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 2854, in reserve_block_device_name
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp return do_reserve()
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/openstack/common/lockutils.py", line 242, in inner
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/manager.py", line 2843, in do_reserve
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp context, instance, bdms, device)
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/new/nova/nova/compute/utils.py", line 165, in get_device_name_for_instance
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp raise exception.DevicePathInUse(path=device)
2013-06-20 19:26:25.981 23879 TRACE nova.openstack.common.rpc.amqp DevicePathInUse: The supplied device path (/dev/vdb) is in use.

[melanie witt (melwitt)]

The devstack volumes.sh exercise creates a volume and then issues 'nova volume-attach' to attach the volume to /dev/vdb.

http://logs.openstack.org/33478/1/gate/gate-tempest-devstack-vm-quantum/32609/console.html.gz

Somehow, /dev/vdb seems to be in the block_device.instance_block_mapping. In the screen-n-cpu.txt log, the instance has u'default_ephemeral_device': u'/dev/vdb' which looks related as the default_ephemeral_device does get returned as part of the mappings. If the requested device is found in the mappings, DevicePathInUse will be raised.

I'm not yet sure when/how default_ephemeral_device gets set. I looked at a recent gate run and didn't see it set to anything other than None.

http://logs.openstack.org/35361/15/gate/gate-tempest-devstack-vm-neutron/4343/logs/screen-n-cpu.txt.gz

[Nikola Đipanov (ndipanov)]

The real solution here is to completely disregard the passed in device name when running libvirt/KVM as it cannot honor it. The API option is there only for hypervisors that support it such as Xen.

Roughly this can be done by the following steps:

1) Exposing a way for drivers to generate device names - much like is done now when booting (but not when attaching volumes to running instances for which we use a general helper method compute_utils.get_device_name_for_instance)
2) Make libvirt disregard what is passed into the new method and simply use it's own assignment methods. Fall back on the utils method in the other driver's implementations (much like is done when booting)

[Joe Gordon (jogo)]

Not seeing this stacktrace anymore, marking as incomplete.

[Sean Dague (sdague)]

Old incomplete bug, moving to invalid