Fullstack test neutron.tests.fullstack.test_l3_agent.TestHAL3Agent. test_ha_router_restart_agents_no_packet_lost fails often

[Slawek Kaplonski]

Example of failure: http://logs.openstack.org/95/572295/1/check/neutron-fullstack/14122fa/logs/testr_results.html.gz

Happened about 50 times in last week: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Could%20not%20ping%20the%20other%20VM%2C%20L2%20agent%20restart%20leads%20to%20network%20disruption%5C%22%20AND%20build_name%3A%5C%22neutron-fullstack%5C%22

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/572568

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/572742

[OpenStack Infra (hudson-openstack)]

Change abandoned by Manjeet Singh Bhatia (<email address hidden>) on branch: master
Review: https://review.openstack.org/572568

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/572742
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=32f14aa8a030e0d619ddd4989a3ea54a481f20e5
Submitter: Zuul
Branch: master

commit 32f14aa8a030e0d619ddd4989a3ea54a481f20e5
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jun 6 13:42:37 2018 +0200

    [Fullstack] Change time waiting for async ping results

    During agents restart there is async ping run and there is
    called function to wait until all async ping workers will
    finish their job.

    In TestHAL3Agent.test_ha_router_restart_agents_no_packet_lost
    there are 60 pings sent with 1 second timeout so default
    wait_until_true timeout which is set to 60 seconds might not
    be enough in some cases.
    Because of that wait_until_true timeout is now set as
    twice higher value than is needed to number of packets to send
    with ping_timeout.

    This should give enough time to finish all workers.

    Change-Id: Ia7c3755c2ba5029bdab3c1dd30b305f3bde19740
    Closes-Bug: #1775183

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/573603

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/573603
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9eef1db160521076d8243f1980e681f0f04ecbc6
Submitter: Zuul
Branch: stable/queens

commit 9eef1db160521076d8243f1980e681f0f04ecbc6
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jun 6 13:42:37 2018 +0200

    [Fullstack] Change time waiting for async ping results

    During agents restart there is async ping run and there is
    called function to wait until all async ping workers will
    finish their job.

    In TestHAL3Agent.test_ha_router_restart_agents_no_packet_lost
    there are 60 pings sent with 1 second timeout so default
    wait_until_true timeout which is set to 60 seconds might not
    be enough in some cases.
    Because of that wait_until_true timeout is now set as
    twice higher value than is needed to number of packets to send
    with ping_timeout.

    This should give enough time to finish all workers.

    Change-Id: Ia7c3755c2ba5029bdab3c1dd30b305f3bde19740
    Closes-Bug: #1775183
    (cherry picked from commit 32f14aa8a030e0d619ddd4989a3ea54a481f20e5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.3 release.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/577648

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/577648

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577648
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5522da67cfb33cf2c781fb0db8ebaff622fd65f9
Submitter: Zuul
Branch: stable/pike

commit 5522da67cfb33cf2c781fb0db8ebaff622fd65f9
Author: Slawek Kaplonski <email address hidden>
Date: Wed Jun 6 13:42:37 2018 +0200

    [Fullstack] Change time waiting for async ping results

    During agents restart there is async ping run and there is
    called function to wait until all async ping workers will
    finish their job.

    In TestHAL3Agent.test_ha_router_restart_agents_no_packet_lost
    there are 60 pings sent with 1 second timeout so default
    wait_until_true timeout which is set to 60 seconds might not
    be enough in some cases.
    Because of that wait_until_true timeout is now set as
    twice higher value than is needed to number of packets to send
    with ping_timeout.

    This should give enough time to finish all workers.

    Change-Id: Ia7c3755c2ba5029bdab3c1dd30b305f3bde19740
    Closes-Bug: #1775183
    (cherry picked from commit 32f14aa8a030e0d619ddd4989a3ea54a481f20e5)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.6 release.