Mitigate frequent fixtures._fixtures.timeout.TimeoutException in UTs and FTs

[Rodolfo Alonso]

Since [0] was introduced, it's very frequent to have "fixtures._fixtures.timeout.TimeoutException" exceptions during the execution of UTs and FTs. Because the privsep includes the synchronized decorator, the synchronization wait is done inside the privsep context. This is prone to timeouts if the wait is too long.

[0] https://review.opendev.org/#/c/631275/
[1] https://148a66b404dde523de26-17406e3478c64e603d8ff3ea0aac16c8.ssl.cf5.rackcdn.com/680393/1/check/neutron-functional-python27/59e721e/testr_results.html.gz
[2] https://0668011f33af6364883c-c555fae2d8c498523cc4b2c363541725.ssl.cf1.rackcdn.com/679852/11/gate/neutron-functional/6b7c424/testr_results.html.gz

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/681432

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/681432
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=63715ea361ea94fb48842c83deef1e5850bcc44d
Submitter: Zuul
Branch: master

commit 63715ea361ea94fb48842c83deef1e5850bcc44d
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Sep 11 10:13:19 2019 +0000

    Re execute a test case if fixtures.TimeoutException is thrown

    Since [1] was introduced, it's very frequent to have
    "fixtures._fixtures.timeout.TimeoutException" exceptions during the
    execution of UTs and FTs. Because the privsep includes the synchronized
    decorator, the synchronization wait is done inside the privsep context.
    This is prone to timeouts if the wait is too long.

    Until we can reorder the decorators of ip_lib [2] or we can remove the
    sync decorators [3], this patch can mitigate the errors in the CI.

    [1]https://review.opendev.org/#/c/631275/
    [2]https://review.opendev.org/#/c/666853/
    [3]https://review.opendev.org/#/c/657608/

    Closes-Bug: #1843478

    Change-Id: If865c4683645f9bd11f5e1b528bade0547505bfd

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/689658

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/689658
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4f59db2fab33e39f3d101885263e5f6483ad2b94
Submitter: Zuul
Branch: master

commit 4f59db2fab33e39f3d101885263e5f6483ad2b94
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sun Oct 20 19:16:48 2019 +0000

    Reset timeout exception in DietTestCase when retrying

    In case of fixtures.TimeoutException, DietTestCase retries the
    execution of the test case (introduced in [1]). But the timeout
    fixture was not reset. This can lead to a general execution timeout
    if a test do not exit in a proper time.

    [1] If865c4683645f9bd11f5e1b528bade0547505bfd

    Related-Bug: #1843478
    Partial-Bug: #1848944
    Change-Id: Ib6ef58b5259ea4bf151e866f6244ff87b195b139

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/705183

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/705758

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/705759

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705758
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9db46c5794493bcd7e768c97458aba105242c75f
Submitter: Zuul
Branch: stable/train

commit 9db46c5794493bcd7e768c97458aba105242c75f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sun Oct 20 19:16:48 2019 +0000

    Reset timeout exception in DietTestCase when retrying

    In case of fixtures.TimeoutException, DietTestCase retries the
    execution of the test case (introduced in [1]). But the timeout
    fixture was not reset. This can lead to a general execution timeout
    if a test do not exit in a proper time.

    [1] If865c4683645f9bd11f5e1b528bade0547505bfd

    Related-Bug: #1843478
    Partial-Bug: #1848944
    Change-Id: Ib6ef58b5259ea4bf151e866f6244ff87b195b139
    (cherry picked from commit 4f59db2fab33e39f3d101885263e5f6483ad2b94)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705183
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=6589f06e4646ef0a3df3f6f65f92b3733d58e7fb
Submitter: Zuul
Branch: stable/stein

commit 6589f06e4646ef0a3df3f6f65f92b3733d58e7fb
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Sep 11 10:13:19 2019 +0000

    Re execute a test case if fixtures.TimeoutException is thrown

    Since [1] was introduced, it's very frequent to have
    "fixtures._fixtures.timeout.TimeoutException" exceptions during the
    execution of UTs and FTs. Because the privsep includes the synchronized
    decorator, the synchronization wait is done inside the privsep context.
    This is prone to timeouts if the wait is too long.

    Until we can reorder the decorators of ip_lib [2] or we can remove the
    sync decorators [3], this patch can mitigate the errors in the CI.

    Backporting for stable/stein as mentioned change [1] is in this branch

    [1]https://review.opendev.org/#/c/631275/
    [2]https://review.opendev.org/#/c/666853/
    [3]https://review.opendev.org/#/c/657608/

    Closes-Bug: #1843478

    Change-Id: If865c4683645f9bd11f5e1b528bade0547505bfd
    (cherry picked from commit 63715ea361ea94fb48842c83deef1e5850bcc44d)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/705759
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4ff4ff4864fb62a0b581a1821e02dec24581d09a
Submitter: Zuul
Branch: stable/stein

commit 4ff4ff4864fb62a0b581a1821e02dec24581d09a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Sun Oct 20 19:16:48 2019 +0000

    Reset timeout exception in DietTestCase when retrying

    In case of fixtures.TimeoutException, DietTestCase retries the
    execution of the test case (introduced in [1]). But the timeout
    fixture was not reset. This can lead to a general execution timeout
    if a test do not exit in a proper time.

    [1] If865c4683645f9bd11f5e1b528bade0547505bfd

    Related-Bug: #1843478
    Partial-Bug: #1848944
    Change-Id: Ib6ef58b5259ea4bf151e866f6244ff87b195b139
    (cherry picked from commit 4f59db2fab33e39f3d101885263e5f6483ad2b94)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.1.0 release.