Comment 23 for bug 1348204

Based on the new logging for the original libvirt error, I haven't seen these before:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRmFpbGVkIHRvIGF0dGFjaCB2b2x1bWUgYXQgbW91bnRwb2ludFwiIEFORCB0YWdzOlwic2NyZWVuLW4tY3B1LnR4dFwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiJjdXN0b20iLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsiZnJvbSI6IjIwMTQtMTEtMDNUMjA6NTE6MjIrMDA6MDAiLCJ0byI6IjIwMTQtMTEtMTRUMjA6NTE6MjIrMDA6MDAiLCJ1c2VyX2ludGVydmFsIjoiMCJ9LCJzdGFtcCI6MTQxNTk5ODQ2NTA4MH0=

1. In this case, it looks like the connection to libvirt just drops:

2014-11-12 03:08:52.797 5908 WARNING nova.virt.libvirt.driver [-] Connection to libvirt lost: 0
2014-11-12 03:08:52.800 ERROR nova.virt.libvirt.driver [req-f3264cfc-36d6-41af-a3aa-2d913e16a343 TestEncryptedCinderVolumes-1650787953 TestEncryptedCinderVolumes-658618010] [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] Failed to attach volume at mountpoint: /dev/vdb
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] Traceback (most recent call last):
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1420, in attach_volume
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] virt_dom.attachDeviceFlags(conf.to_xml(), flags)
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] rv = execute(f, *args, **kwargs)
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] six.reraise(c, e, tb)
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] rv = meth(*args, **kwargs)
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 439, in attachDeviceFlags
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593] libvirtError: End of file while reading data: Input/output error
2014-11-12 03:08:52.800 5908 TRACE nova.virt.libvirt.driver [instance: 14f6ba42-0e39-494d-a8d1-ca04f2843593]

2. In this case, it looks like it can't open the device because it's busy:

2014-11-07 10:47:31.273 ERROR nova.virt.libvirt.driver [req-eea5a7d5-6518-4564-bc7b-eb7f59345c63 TestEncryptedCinderVolumes-1947534427 TestEncryptedCinderVolumes-10971448] [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] Failed to attach volume at mountpoint: /dev/vdb
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] Traceback (most recent call last):
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1420, in attach_volume
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] virt_dom.attachDeviceFlags(conf.to_xml(), flags)
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] rv = execute(f, *args, **kwargs)
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] six.reraise(c, e, tb)
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] rv = meth(*args, **kwargs)
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 513, in attachDeviceFlags
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] libvirtError: operation failed: open disk image file failed
2014-11-07 10:47:31.273 1695 TRACE nova.virt.libvirt.driver [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562]

...

2014-11-07 10:47:32.168 ERROR nova.virt.block_device [req-eea5a7d5-6518-4564-bc7b-eb7f59345c63 TestEncryptedCinderVolumes-1947534427 TestEncryptedCinderVolumes-10971448] [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] Driver failed to attach volume bce86fa3-82d4-46e4-b9af-973bd798abfc at /dev/vdb
2014-11-07 10:47:32.168 1695 TRACE nova.virt.block_device [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] Traceback (most recent call last):
2014-11-07 10:47:32.168 1695 TRACE nova.virt.block_device [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] File "/opt/stack/new/nova/nova/virt/block_device.py", line 253, in attach
2014-11-07 10:47:32.168 1695 TRACE nova.virt.block_device [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] device_type=self['device_type'], encryption=encryption)
2014-11-07 10:47:32.168 1695 TRACE nova.virt.block_device [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1428, in attach_volume
2014-11-07 10:47:32.168 1695 TRACE nova.virt.block_device [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] raise exception.DeviceIsBusy(device=disk_dev)
2014-11-07 10:47:32.168 1695 TRACE nova.virt.block_device [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562] DeviceIsBusy: The supplied device (vdb) is busy.
2014-11-07 10:47:32.168 1695 TRACE nova.virt.block_device [instance: 3d0c6ca7-c48d-440a-bb64-dee1e525a562]