Enabling guestfs.debug can hang guestfs.launch

[Matt Riedemann]

I noticed this while debugging this change:

https://review.openstack.org/#/c/524710/

When we enable guestfs.debug, we're receiving events from guestfs and logging them.

However, guestfs.launch is run in an eventlet thread pool, and running things within an eventlet thread pool for native apps is a no-no for doing logging because it can cause a greenthread switch that makes us lose the return from the guestfs.launch call, which eventually means we hang when trying to create the instance.

So we need to just run guestfs.launch non-threaded if guestfs.debug is enabled.

From the logs, you can see the guestfs.launch thread switch here:

http://logs.openstack.org/50/524750/1/check/legacy-tempest-dsvm-neutron-full-centos-7/a7f051e/logs/screen-n-cpu.txt.gz#_Dec_04_13_43_19_941869

And the logging callback here:

http://logs.openstack.org/50/524750/1/check/legacy-tempest-dsvm-neutron-full-centos-7/a7f051e/logs/screen-n-cpu.txt.gz#_Dec_04_15_25_29_320753

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/526748

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/526748
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7c30da13842469e641d171b6008a57d5db8c5d5e
Submitter: Zuul
Branch: master

commit 7c30da13842469e641d171b6008a57d5db8c5d5e
Author: Matt Riedemann <email address hidden>
Date: Fri Dec 8 13:15:50 2017 -0500

    Don't launch guestfs in a thread pool if guestfs.debug is enabled

    When guestfs.debug is enabled, we're handling callback events
    from guestfs and logging them at debug level. When guestfs
    is launched to inspect capabilities, that is currently done
    in an eventlet thread pool. Because of the concurrent logging
    along with the eventlet thread, we can hit an issue where eventlet
    tries to switch threads and fails and then we hang the launch
    call to guestfs, which hangs creating an instance.

    This change simply avoids using a thread pool to launch guestfs
    if guestfs.debug is True.

    Change-Id: I0ffe93a031154b123c8beff96a695df5a280b935
    Closes-Bug: #1737214

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/539541

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/539541
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e521616afd14e6eedc0d3bfca0adf2179f4e05c2
Submitter: Zuul
Branch: stable/pike

commit e521616afd14e6eedc0d3bfca0adf2179f4e05c2
Author: Matt Riedemann <email address hidden>
Date: Fri Dec 8 13:15:50 2017 -0500

    Don't launch guestfs in a thread pool if guestfs.debug is enabled

    When guestfs.debug is enabled, we're handling callback events
    from guestfs and logging them at debug level. When guestfs
    is launched to inspect capabilities, that is currently done
    in an eventlet thread pool. Because of the concurrent logging
    along with the eventlet thread, we can hit an issue where eventlet
    tries to switch threads and fails and then we hang the launch
    call to guestfs, which hangs creating an instance.

    This change simply avoids using a thread pool to launch guestfs
    if guestfs.debug is True.

    Change-Id: I0ffe93a031154b123c8beff96a695df5a280b935
    Closes-Bug: #1737214
    (cherry picked from commit 7c30da13842469e641d171b6008a57d5db8c5d5e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.1 release.