In python3 test_fixtures.TestOSAPIFixture.test_responds_to_version stalls in epoll()

[Chris Dent]

When running the py34 unit tests, nova.tests.unit.test_fixtures.TestOSAPIFixture.test_responds_to_version will block in epoll() for up to 900 seconds. This sometimes causes the gate to timeout as the combined time of building the environment, running the tests, and gathering the information can be too much (despite all the tests passing).

The problem appears to be a deadlock/race in eventlet itself when working with the same file from different greenthreads.

see email for more discussion: http://lists.openstack.org/pipermail/openstack-dev/2016-March/089496.html

a fix is proposed and will be linked here momentarily

[Chris Dent (cdent)]

The related review: https://review.openstack.org/#/c/293372/

[Davanum Srinivas (DIMS) (dims-v)]

Here's what the threads look like when it's stuck:
http://paste.openstack.org/show/490793/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/294568

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chris Dent (<email address hidden>) on branch: master
Review: https://review.openstack.org/293372
Reason: this fix is considered too weird, better to just block the test in py3, done in: I47e90bb613bfba76bb504a5bd0955206120b5556

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/294568
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f6b11c59641d55020a5ada25cba6ec4de58431a9
Submitter: Jenkins
Branch: master

commit f6b11c59641d55020a5ada25cba6ec4de58431a9
Author: Chris Dent <email address hidden>
Date: Fri Mar 18 12:36:32 2016 +0000

    Blacklist TestOSAPIFixture.test_responds_to_version in python3

    In python3 the test blocks in epoll() for 10-15 minutes. This can
    lead to gate job timeouts (as a result of the cumulative time being
    extended by this one test). The root cause has been tracked to eventlet
    itself.

    Change-Id: I47e90bb613bfba76bb504a5bd0955206120b5556
    Related-Bug: #1558105

[Augustina Ragwitz (auggy)]

Does this fix actually close the bug or just partially close it? If it closes, the tag "closes-bug" should be used. Since this fix is already merged, if it does actually fix the bug then this bug should be marked as Fix Committed.

[ChangBo Guo(gcb) (glongwave)]

It takes about 2.5 seconds to run this test method. Not sure this only occurred run all unit tests.

tox -e py34 nova.tests.unit.test_fixtures.TestOSAPIFixture.test_responds_to_version
..............
．．．．．．．．．．．．
======
Totals
======
Ran: 1 tests in 29.2162 sec.
 - Passed: 1
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 2.5056 sec.

==============
Worker Balance
==============
 - Worker 0 (1 tests) => 0:00:02.505606

Slowest Tests:

Test id Runtime (s)
----------------------------------------------------------------------- -----------
nova.tests.unit.test_fixtures.TestOSAPIFixture.test_responds_to_version 2.506

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/415612

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/415612
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4173e4192142d9ba0d02e92082522b7fb651abe0
Submitter: Jenkins
Branch: master

commit 4173e4192142d9ba0d02e92082522b7fb651abe0
Author: ChangBo Guo(gcb) <email address hidden>
Date: Thu Dec 29 14:20:29 2016 +0800

    Enable TestOSAPIFixture.test_responds_to_version on Python 3

    It takes about 2.5 seconds to run this test, seems the issue
    has gone. Not sure this is fixed by eventlet now. Let's enable
    it now and figure out root cause and fix the issue.

    Related-Bug: #1558105
    Partially-Implements: blueprint goal-python35

    Change-Id: I6826a4ba3ea5656471cdffbaa2883b522b4b22f0

[Sean Dague (sdague)]

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

[Sean Dague (sdague)]

This appears fixed, please reopen if still an issue