Spawning a number of instances using a single request (or number of requests close enough to each other) can result in the same nbd device chosen for more than one instance. For example, see the log of a collision below. The python threads switch after the device is selected and qemu-nbd is spawned, but before the qemu-nbd gets a chance to connect and create a /sys/block/nbd*/pid file.
Solution is most likely to put a lock around the _inner_get_dev() part of NbdMount.
2013-08-01 12:18:02.893 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Get nbd device /dev/nbd14 for /var/lib/nova/instances/0fde8031-0152-49ca-bd59-fd7a9353afc0/disk _inner_get_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:87
2013-08-01 12:18:02.893 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -c /dev/nbd14 /var/lib/nova/instances/0fde8031-0152-49ca-bd59-fd7a9353afc0/disk execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
2013-08-01 12:18:02.924 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Get nbd device /dev/nbd14 for /var/lib/nova/instances/10474a10-4bfa-41f2-8bd6-9add1e6aff1f/disk _inner_get_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:87
2013-08-01 12:18:02.924 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -c /dev/nbd14 /var/lib/nova/instances/10474a10-4bfa-41f2-8bd6-9add1e6aff1f/disk execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
2013-08-01 12:18:02.968 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Map dev /dev/nbd14 map_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:135
2013-08-01 12:18:02.968 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Mount /dev/nbd14p1 on /tmp/openstack-vfs-localfsnW0rfa mnt_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:188
2013-08-01 12:18:02.968 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
2013-08-01 12:18:03.000 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Map dev /dev/nbd14 map_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:135
2013-08-01 12:18:03.000 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf kpartx -a /dev/nbd14 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa
Stderr: 'mount: special device /dev/nbd14p1 does not exist\n' mnt_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:193
2013-08-01 12:18:03.044 23130 DEBUG nova.virt.disk.mount.api [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Unmap dev /dev/nbd14 unmap_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/api.py:179
2013-08-01 12:18:03.045 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Release nbd device /dev/nbd14 unget_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:126
2013-08-01 12:18:03.045 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -d /dev/nbd14 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa
Stderr: 'mount: special device /dev/nbd14p1 does not exist\n') setup /usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/localfs.py:81
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf mount /dev/nbd14p1 /tmp/openstack-vfs-localfsnW0rfa
Stderr: 'mount: special device /dev/nbd14p1 does not exist\n')
2013-08-01 12:18:04.117 23130 DEBUG nova.virt.disk.mount.nbd [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Release nbd device /dev/nbd14 unget_dev /usr/lib/python2.7/dist-packages/nova/virt/disk/mount/nbd.py:126
2013-08-01 12:18:04.118 23130 DEBUG nova.openstack.common.processutils [req-4ed4037b-a707-4b25-9037-5cf3443f8267 10363648477724 10909817428811] Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf qemu-nbd -d /dev/nbd14 execute /usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py:142
Fix proposed to branch: master /review. openstack. org/39927
Review: https:/