Failed deployment of VM due to the use of guestfs.GuestFS (close_on_exit = False)

[Jorge]

After Bug 1261475 was fixed, A new issue in Ice-house was introduced. Basically now that ice-house is doing the call guestfs.GuestFS (close_on_exit = False), there are different requirements on the version of libguestfs.x86_64 and python-libguestfs.x86_64.

During the deployment of a VM I would get the following error on the compute node:

2014-01-20 16:51:32.395 7723 ERROR nova.compute.manager [req-482a2eec-238c-4696-8aab-acc0d404e627 45a71543cb6f4022ac4f56855a7ff197 4a5f610e660347b6a629a27ffea92240] [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] Error: __init__() got an unexpected keyword argument 'close_on_exit'
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] Traceback (most recent call last):
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1062, in _build_instance
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] set_access_ip=set_access_ip)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 362, in decorated_function
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] return function(self, context, *args, **kwargs)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1497, in _spawn
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] six.reraise(self.type_, self.value, self.tb)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1483, in _spawn
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] block_device_info)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2174, in spawn
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] admin_pass=admin_password)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2477, in _create_image
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] project_id=instance['project_id'])
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 187, in cache
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] *args, **kwargs)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 337, in create_image
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] copy_qcow2_image(base, self.path, size)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] return f(*args, **kwargs)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 303, in copy_qcow2_image
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] disk.extend(target, size, use_cow=True)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/virt/disk/api.py", line 149, in extend
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] if not is_image_partitionless(image, use_cow):
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/virt/disk/api.py", line 189, in is_image_partitionless
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] fs.setup()
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/virt/disk/vfs/guestfs.py", line 111, in setup
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] self.handle = tpool.Proxy(guestfs.GuestFS(close_on_exit=False))
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] TypeError: __init__() got an unexpected keyword argument 'close_on_exit'

After doing some research I found out that the close_on_exit parameter is supported only on the newer versions of libguestfs.x86_64 and python-libguestfs.x86_64.

In my system (RHEL 6.4) I had
libguestfs.x86_64 1:1.16.34-2.el6
python-libguestfs.x86_64 1:1.16.34-2.el6

After updating to the following versions the deploy operation started working again.
libguestfs.x86_64 1:1.20.11-2.el6
python-libguestfs.x86_64 1:1.20.11-2.el6

[Tom Deckers (tdeckers)]

Hi Jorge,

How did you install the later versions of libguestfs and python-libguestfs? I'm relying on EPEL and they're not available yet there. A manual download of libguestfs and install bring up a whole bunch of additional dependencies... It's going to be a nightmare to install them all manually.

Thanks!
Tom.

[Tom Deckers (tdeckers)]

I have removed the close_on_exit parameter locally as a workaround. That seems to do the trick too.

Regards,
Tom.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70354
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fd621ef05a024b8319f2c67f7e5c084b9ab77755
Submitter: Jenkins
Branch: master

commit fd621ef05a024b8319f2c67f7e5c084b9ab77755
Author: Russell Bryant <email address hidden>
Date: Fri Jan 31 09:28:50 2014 -0500

    Deal with old versions of libguestfs

    Commit 02abbef960b12deaa7a911788dbc56f3a0bd555a added the use of the
    close_on_exit parameter. This is supported by libguestfs 1.20 and up.
    Catch the exeption raised when this parameter isn't supported and retry
    without it.

    Change-Id: Ic47c1cb997ece3710f8c4465733148fcfa94068d
    Closes-Bug: #1271562
    Related-Bug: #1261475

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/70419

[Jorge (jorgedr)]

Hello, how can I tell what version of OpenStack the fix was merged into?

[Jorge (jorgedr)]

Tom,

I got the library from one of my company's repositories. I think they have access to beta packages. After installing it in one of our systems I could no longer find the packages anymore so I did the same thing that you did. That is why I want to know what version of open stack this fix is into. Our product is going to be be based on ice-house so I need to make sure we are going to pick the fix for it.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70419
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a154ff8e54478d6b2a2396eada97eb5229d74811
Submitter: Jenkins
Branch: master

commit a154ff8e54478d6b2a2396eada97eb5229d74811
Author: Russell Bryant <email address hidden>
Date: Fri Jan 31 14:30:13 2014 -0500

    Test guestfs without support for close_on_exit

    A previous commit, fd621ef05a024b8319f2c67f7e5c084b9ab77755, fixed support for
    versions of libguestfs that did not support the close_on_exit parameter. This
    patch adds unit test coverage for that code path.

    Change-Id: Ic3cc0057e24ac00e6435a9555a19c076ddc9b02e
    Related-Bug: #1271562
    Related-Bug: #1261475