Instance creation fails with NovaException: Error mounting with libguestfs

[Yuriy Yekovenko]

The issue was discovered with the following steps:

1. Deploy Fuel 4.0.
2. Clone tempest (Havana stable branch) to Controller node.
3. Configure tempest.
4. Run
nosetests -sv tempest/api/compute/servers/test_create_server.py

This script creates instances with code:
cls.create_server(name=cls.name,
                                     meta=cls.meta,
                                     accessIPv4=cls.accessIPv4,
                                     accessIPv6=cls.accessIPv6,
                                     personality=personality,
                                     disk_config=cls.disk_config)

This code sends request with body:
{"server": {"name": "server82800933", "imageRef": "2c97f855-a19d-4c08-8f6d-7b74df66befe", "flavorRef": 1, "OS-DCF:diskConfig": "AUTO", "accessIPv4": "1.1.1.1", "accessIPv6": "0000:0000:0000:0000:0000:babe:220.12.22.2", "metadata": {"hello": "world"}, "personality": [{"path": "/test.txt", "contents": "VGhpcyBpcyBhIHRlc3QgZmlsZS4="}]}}

Result: instance in ERROR status.
/var/log/nova-all.log contains:

<0>Mar 5 15:29:07 node-13 ¿<182>nova-nova.virt.libvirt.driver INFO: Creating image
<0>Mar 5 15:29:08 node-13 ¿<182>nova-nova.virt.libvirt.driver INFO: Injecting metadata into image 2c97f855-a19d-4c08-8f6d-7b74df66befe
<0>Mar 5 15:29:08 node-13 ¿<182>nova-nova.virt.libvirt.driver INFO: Injecting files into image 2c97f855-a19d-4c08-8f6d-7b74df66befe
<0>Mar 5 15:29:08 node-13 ¿<179>nova-nova.virt.libvirt.driver ERROR: Error injecting data into image 2c97f855-a19d-4c08-8f6d-7b74df66befe (Error mounting /var/lib/nova/instances/d5069cc1-f144-4493-8ee6-9abe32c6dd44/disk with libguestfs (command failed: LC_ALL=C '/usr/libexec/qemu-kvm' -nographic -help
errno: File exists

If qemu is located on a non-standard path, try setting the LIBGUESTFS_QEMU
environment variable. There may also be errors printed above.))
<0>Mar 5 15:29:08 node-13 ¿<179>nova-nova.compute.manager ERROR: Instance failed to spawn
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1417, in _spawn
    block_device_info)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2067, in spawn
    admin_pass=admin_password)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2500, in _create_image
    instance=instance)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2494, in _create_image
    mandatory=('files',))
  File "/usr/lib/python2.6/site-packages/nova/virt/disk/api.py", line 325, in inject_data
    fs.setup()
  File "/usr/lib/python2.6/site-packages/nova/virt/disk/vfs/guestfs.py", line 129, in setup
    {'imgfile': self.imgfile, 'e': e})
NovaException: Error mounting /var/lib/nova/instances/d5069cc1-f144-4493-8ee6-9abe32c6dd44/disk with libguestfs (command failed: LC_ALL=C '/usr/libexec/qemu-kvm' -nographic -help
errno: File exists

If qemu is located on a non-standard path, try setting the LIBGUESTFS_QEMU
environment variable. There may also be errors printed above.)

The following steps helped to fix this issue:

1. qemu-kvm is located in /usr/bin folder but libguestfs expect it in /usr/libexec/ folder.
So, on each Compute node do this :
ln -s /usr/bin/qemu-kvm /usr/libexec/qemu-kvm

2. Apply fix for bug https://bugzilla.redhat.com/show_bug.cgi?id=894241#c11 :
On each Compute node, add these to the supermin hostfiles (/usr/lib64/guestfs/supermin.d/hostfiles):
/usr/share/augeas/lenses/dist/quote.aug
/usr/share/augeas/lenses/dist/simplevars.aug

[Vitaly Parakhin (vparakhin)]

Vladimir,

Regarding the issues :

1) we're going to add creation of /usr/libexec/qemu-kvm symlink to the qemu-kvm rpm package
2) current augeas-libs version (0.9.0-4.el6) does not contain the "lenses" entries specified above, so we're going to replace it with current version (1.0.0-5.el6) from upstream
Also, in order to include these descriptions to the supermin hostfiles, we will build our own libguestfs package from CentOS 6.4 source (thus preserving the version).

Please let us know if that sounds good to you.

[Vitaly Parakhin (vparakhin)]

Packages that contain fix are available here:

https://gerrit.mirantis.com/#/c/13642/
https://gerrit.mirantis.com/#/c/13647/
https://gerrit.mirantis.com/#/c/13640/

Please test and confirm.

[Vladimir Kuklin (vkuklin)]

please, create also backports for stable/4.1

[Vitaly Parakhin (vparakhin)]

Also, the following packages are available for 4.1.1:

https://gerrit.mirantis.com/#/c/13740/
https://gerrit.mirantis.com/#/c/13743/
https://gerrit.mirantis.com/#/c/13758/

[Meg McRoberts (dreidellhasa)]

I need clarification about what to include in Release Notes.