[FT] "TestMonitorDaemon._run_monitor" failing radomly, initial message not written

[Rodolfo Alonso]

The method ``TestMonitorDaemon._run_monitor`` is randomly failing. The initial message expected after "keepalived_state_change" process is started is not written in the logs.

Logs: https://8271f43479f81f7d3395-4bdeef087e9d95514555f2932706a956.ssl.cf1.rackcdn.com/893555/1/check/neutron-functional-with-uwsgi/90dde54/testr_results.html

Snippet: https://paste.opendev.org/show/bJCAjIqiUOcWB0xBhPnX/

[Miro Tomaska (mtomaska)]

It appears this is the same issue we fixed few months ago [1][2]. There we have raised the log file read timeout by 3 seconds, totaling to 18 seconds. The Traceback[3] from this failure states that timeout was set to 15 seconds which makes me believe this is an older branch. If I am reading the job output correctly this run on Xena[4] which would explain the failure.

@Rodolfo Alonso, should I just backport[2] to earlier branches? Thanks!

[1]https://bugs.launchpad.net/neutron/+bug/2003022
[2]https://review.opendev.org/c/openstack/neutron/+/870679
[3]https://paste.opendev.org/show/bJCAjIqiUOcWB0xBhPnX/
[4]https://opendev.org/openstack/neutron/src/commit/067364fce50ca4105b3a98f9838d7fecc8e6baec/neutron/tests/functional/agent/l3/test_keepalived_state_change.py#L99

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hi Miro:

That could work, I didn't check the branch of the logs. But as you said, if the timeout is 15 seconds, most probably these are stable ones. Please propose backports for the stable branches and let's check if that solves the issue.

Regards.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/895667

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/neutron/+/895668

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/895669

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/895670

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895667
Committed: https://opendev.org/openstack/neutron/commit/e27a4af2b1e07043e7a7a1595b810cd9c8e22238
Submitter: "Zuul (22348)"
Branch: stable/zed

commit e27a4af2b1e07043e7a7a1595b810cd9c8e22238
Author: Miguel Lavalle <email address hidden>
Date: Mon Jan 16 16:41:20 2023 -0600

    Add 3 secs to wait for keepalived state change

    We add 3 seconds to the keepalived state change to avoid timeout
    failures in test_read_queue_change_state.

    Change-Id: Ic54bd6f699b91b53f22c93c972d8def6c99d0eb7
    Closes-Bug: #2035382
    Related-Bug: #2003022
    (cherry picked from commit f28e4165993f0c226bd5af54ad6d8c6bb48bff69)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895668
Committed: https://opendev.org/openstack/neutron/commit/edf33d2217f73b8775c76f3cac2fc75ce00632a1
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit edf33d2217f73b8775c76f3cac2fc75ce00632a1
Author: Miguel Lavalle <email address hidden>
Date: Mon Jan 16 16:41:20 2023 -0600

    Add 3 secs to wait for keepalived state change

    We add 3 seconds to the keepalived state change to avoid timeout
    failures in test_read_queue_change_state.

    Change-Id: Ic54bd6f699b91b53f22c93c972d8def6c99d0eb7
    Closes-Bug: #2035382
    Related-Bug: #2003022
    (cherry picked from commit f28e4165993f0c226bd5af54ad6d8c6bb48bff69)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895669
Committed: https://opendev.org/openstack/neutron/commit/9579ce4c35a1ae35370f9305f8f167ab2d4f4c85
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 9579ce4c35a1ae35370f9305f8f167ab2d4f4c85
Author: Miguel Lavalle <email address hidden>
Date: Mon Jan 16 16:41:20 2023 -0600

    Add 3 secs to wait for keepalived state change

    We add 3 seconds to the keepalived state change to avoid timeout
    failures in test_read_queue_change_state.

    Change-Id: Ic54bd6f699b91b53f22c93c972d8def6c99d0eb7
    Closes-Bug: #2035382
    Related-Bug: #2003022
    (cherry picked from commit f28e4165993f0c226bd5af54ad6d8c6bb48bff69)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/895670
Committed: https://opendev.org/openstack/neutron/commit/ba0c6a176590206e6de5a513ed757f1c818cc8af
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit ba0c6a176590206e6de5a513ed757f1c818cc8af
Author: Miguel Lavalle <email address hidden>
Date: Mon Jan 16 16:41:20 2023 -0600

    Add 3 secs to wait for keepalived state change

    We add 3 seconds to the keepalived state change to avoid timeout
    failures in test_read_queue_change_state.

    Change-Id: Ic54bd6f699b91b53f22c93c972d8def6c99d0eb7
    Closes-Bug: #2035382
    Related-Bug: #2003022
    (cherry picked from commit f28e4165993f0c226bd5af54ad6d8c6bb48bff69)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.5.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 21.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron wallaby-eom release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron xena-eom release.