unit test intermittent failure

[Armando Migliaccio]

An example:

http://logs.openstack.org/98/248498/3/gate/gate-neutron-python27/e2b5fd2/testr_results.html.gz

The logstash query:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22test_failing_async_process_handle_error_once%5C%22

[Armando Migliaccio (armando-migliaccio)]

Wonder if this is the culprit:

https://review.openstack.org/#/c/234770

[Hong Hui Xiao (xiaohhui)]

I scan the code and think it might be caused by the following reason.

The _wait_process is blocked by other GreenThreads(too many threads for the test machine) . So, when the failing_process exit, the _wait_process is still not be executed. Therefore, the call_count is 0.

I can simulate the error by using spawn_after at [1]. I will propose a patch to address this problem.

[1] https://github.com/openstack/neutron/blob/master/neutron/agent/linux/async_process.py#L148

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/249232

[John Schwarz (jschwarz)]

I can confirm the culprit is the patchset Armando linked to. Lets merge Hong's patch to see if it fixes the errors.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/249232
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=dcd0498c17ae860c55a92f9f31b3e3fa0460b78e
Submitter: Jenkins
Branch: master

commit dcd0498c17ae860c55a92f9f31b3e3fa0460b78e
Author: Hong Hui Xiao <email address hidden>
Date: Tue Nov 24 09:01:48 2015 -0500

    Wait for the watch process in test case

    Because the _watch_process and the failing_process are asynchronous,
    there might be a chance that failing_process exit and _watch_process
    is not executed.

    If the _watch_process is blocked, the method that will be asserted
    will not be called. This will fail the UT, but it is intermittent.

    Change-Id: Ic951c1b91c5a10462f548544a5e8d482c52ad665
    Closes-Bug: #1519160

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/277347

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/277347
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=306636e64dc884484aad772dd6994299d30c2a06
Submitter: Jenkins
Branch: stable/liberty

commit 306636e64dc884484aad772dd6994299d30c2a06
Author: Hong Hui Xiao <email address hidden>
Date: Tue Nov 24 09:01:48 2015 -0500

    Wait for the watch process in test case

    Because the _watch_process and the failing_process are asynchronous,
    there might be a chance that failing_process exit and _watch_process
    is not executed.

    If the _watch_process is blocked, the method that will be asserted
    will not be called. This will fail the UT, but it is intermittent.

    Change-Id: Ic951c1b91c5a10462f548544a5e8d482c52ad665
    Closes-Bug: #1519160
    Related-Bug: #1543040
    Related Bug: #1506021
    (cherry picked from commit dcd0498c17ae860c55a92f9f31b3e3fa0460b78e)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 7.0.4 release.