Nova should disable libguestfs' automatic cleanup

[Daniel Berrange]

By default libguestfs will register an atexit() handler to cleanup any open libguestfs handles when the process exits. Since libguestfs does not provide any mutex locking in its APIs, the atexit handlers are not safe in multi-threaded processes. If they run they are liable to cause memory corruption as multiple threads access the same libguestfs handle. As such at atexit handlers should be disabled in any multi-threaded program using libguestfs. eg by using

  guestfs.GuestFS (close_on_exit = False)

instead of

  guestfs.GuestFS()

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/64384

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/64384
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=02abbef960b12deaa7a911788dbc56f3a0bd555a
Submitter: Jenkins
Branch: master

commit 02abbef960b12deaa7a911788dbc56f3a0bd555a
Author: Michael Still <email address hidden>
Date: Mon Dec 30 10:55:20 2013 +1100

    Disable libguestfs' default atexit handlers.

    These aren't safe for multi-threaded applications, per the bug
    report from Daniel Berrange.

    Closes-bug: #1261475

    Change-Id: I4cc5eaded2e21f1b2cd55615242a27b9773f0b42

[Jorge (jorgedr)]

I hit a bug on Ice-house due to this fix. Basically now that ice-house is doing the call guestfs.GuestFS (close_on_exit = False), there are different requirements on the version of libguestfs.x86_64 and python-libguestfs.x86_64.

During the deployment of a VM I would get the following error on the compute node:

2014-01-20 16:51:32.395 7723 ERROR nova.compute.manager [req-482a2eec-238c-4696-8aab-acc0d404e627 45a71543cb6f4022ac4f56855a7ff197 4a5f610e660347b6a629a27ffea92240] [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] Error: __init__() got an unexpected keyword argument 'close_on_exit'
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] Traceback (most recent call last):
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1062, in _build_instance
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] set_access_ip=set_access_ip)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 362, in decorated_function
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] return function(self, context, *args, **kwargs)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1497, in _spawn
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] six.reraise(self.type_, self.value, self.tb)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1483, in _spawn
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] block_device_info)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2174, in spawn
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] admin_pass=admin_password)
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2477, in _create_image
2014-01-20 16:51:32.395 7723 TRACE nova.compute.manager [instance: d4083a7f-0c2d-44b3-a79b-7f498ea75ecd] project_id=instance['project_id'])
2014-01-20 16:51:32.395 7723 ...

[Daniel Berrange (berrange)]

Yes, please report new bugs for new problems. We'll have to make this new usage conditional based on version

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70354
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fd621ef05a024b8319f2c67f7e5c084b9ab77755
Submitter: Jenkins
Branch: master

commit fd621ef05a024b8319f2c67f7e5c084b9ab77755
Author: Russell Bryant <email address hidden>
Date: Fri Jan 31 09:28:50 2014 -0500

    Deal with old versions of libguestfs

    Commit 02abbef960b12deaa7a911788dbc56f3a0bd555a added the use of the
    close_on_exit parameter. This is supported by libguestfs 1.20 and up.
    Catch the exeption raised when this parameter isn't supported and retry
    without it.

    Change-Id: Ic47c1cb997ece3710f8c4465733148fcfa94068d
    Closes-Bug: #1271562
    Related-Bug: #1261475

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/70419

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70419
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a154ff8e54478d6b2a2396eada97eb5229d74811
Submitter: Jenkins
Branch: master

commit a154ff8e54478d6b2a2396eada97eb5229d74811
Author: Russell Bryant <email address hidden>
Date: Fri Jan 31 14:30:13 2014 -0500

    Test guestfs without support for close_on_exit

    A previous commit, fd621ef05a024b8319f2c67f7e5c084b9ab77755, fixed support for
    versions of libguestfs that did not support the close_on_exit parameter. This
    patch adds unit test coverage for that code path.

    Change-Id: Ic3cc0057e24ac00e6435a9555a19c076ddc9b02e
    Related-Bug: #1271562
    Related-Bug: #1261475