Functional test neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase. test_ha_router_lifecycle failing

[Slawek Kaplonski]

Functional test neutron.tests.functional.agent.l3.test_ha_router.LinuxBridgeL3HATestCase. test_ha_router_lifecycle is failing from time to time.

Example of failure: http://logs.openstack.org/68/623268/14/gate/neutron-functional-python27/4dc7fb8/logs/testr_results.html.gz

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%2081%2C%20in%20test_ha_router_lifecycle%5C%22

[Slawek Kaplonski (slaweq)]

I looked into logs from 3 examples of this failure.
In all cases it failed in https://github.com/openstack/neutron/blob/master/neutron/tests/functional/agent/l3/framework.py#L277
and in all 3 cases I saw in logs that router was transitioned to master, e.g. here: http://logs.openstack.org/68/623268/14/gate/neutron-functional-python27/4dc7fb8/logs/dsvm-functional-logs/neutron.tests.functional.agent.l3.test_ha_router.L3HATestCase.test_ha_router_lifecycle.txt.gz#_2019-02-18_14_29_25_033

which shouldn't happen according to test code.
So my theory is that it's some race condition issue. Router is set as backup but then keepalived is spawned and switch router to be master. That triggers configuration of everything in router's namespace and it fails if that will happen "too fast"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/640400

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/640400
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e6351ab11e2cfaea18ab963e32052ad110d2b899
Submitter: Zuul
Branch: master

commit e6351ab11e2cfaea18ab963e32052ad110d2b899
Author: Slawek Kaplonski <email address hidden>
Date: Fri Mar 1 16:11:59 2019 +0100

    [Functional] Don't assert that HA router don't have IPs configured

    In functional tests of HA router, in
    L3AgentTestFramework._router_lifecycle method there was assertion
    that HA router at the beginning don't have IPs configured in
    router's namespace.

    That could lead to test failure because sometimes keepalived process
    switched router from standby to master before this assertion was
    done and IPs were already configured.

    There is alsmost no value in doing this assertion as it's just after
    router was created so it is "normal" that there is no IP addresses
    configured yet.
    Because of that this patch removes this assertion.

    Change-Id: Ib509a7226eb94483a0aaf2d930f329e419b8e135
    Closes-Bug: #1816489

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/645229

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/645230

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/645231

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/645230
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b0c8dde359e3a17b27547cb57c104c58fc67dfc8
Submitter: Zuul
Branch: stable/queens

commit b0c8dde359e3a17b27547cb57c104c58fc67dfc8
Author: Slawek Kaplonski <email address hidden>
Date: Fri Mar 1 16:11:59 2019 +0100

    [Functional] Don't assert that HA router don't have IPs configured

    In functional tests of HA router, in
    L3AgentTestFramework._router_lifecycle method there was assertion
    that HA router at the beginning don't have IPs configured in
    router's namespace.

    That could lead to test failure because sometimes keepalived process
    switched router from standby to master before this assertion was
    done and IPs were already configured.

    There is alsmost no value in doing this assertion as it's just after
    router was created so it is "normal" that there is no IP addresses
    configured yet.
    Because of that this patch removes this assertion.

    Change-Id: Ib509a7226eb94483a0aaf2d930f329e419b8e135
    Closes-Bug: #1816489
    (cherry picked from commit e6351ab11e2cfaea18ab963e32052ad110d2b899)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/645231
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e2f93a2703b339f54a3cbeb5b19e5caa423fea83
Submitter: Zuul
Branch: stable/pike

commit e2f93a2703b339f54a3cbeb5b19e5caa423fea83
Author: Slawek Kaplonski <email address hidden>
Date: Fri Mar 1 16:11:59 2019 +0100

    [Functional] Don't assert that HA router don't have IPs configured

    In functional tests of HA router, in
    L3AgentTestFramework._router_lifecycle method there was assertion
    that HA router at the beginning don't have IPs configured in
    router's namespace.

    That could lead to test failure because sometimes keepalived process
    switched router from standby to master before this assertion was
    done and IPs were already configured.

    There is alsmost no value in doing this assertion as it's just after
    router was created so it is "normal" that there is no IP addresses
    configured yet.
    Because of that this patch removes this assertion.

    Change-Id: Ib509a7226eb94483a0aaf2d930f329e419b8e135
    Closes-Bug: #1816489
    (cherry picked from commit e6351ab11e2cfaea18ab963e32052ad110d2b899)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/645229
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e5202b9fb1ae2bba9d7642f8d9bfbff0a3b1d1a9
Submitter: Zuul
Branch: stable/rocky

commit e5202b9fb1ae2bba9d7642f8d9bfbff0a3b1d1a9
Author: Slawek Kaplonski <email address hidden>
Date: Fri Mar 1 16:11:59 2019 +0100

    [Functional] Don't assert that HA router don't have IPs configured

    In functional tests of HA router, in
    L3AgentTestFramework._router_lifecycle method there was assertion
    that HA router at the beginning don't have IPs configured in
    router's namespace.

    That could lead to test failure because sometimes keepalived process
    switched router from standby to master before this assertion was
    done and IPs were already configured.

    There is alsmost no value in doing this assertion as it's just after
    router was created so it is "normal" that there is no IP addresses
    configured yet.
    Because of that this patch removes this assertion.

    Change-Id: Ib509a7226eb94483a0aaf2d930f329e419b8e135
    Closes-Bug: #1816489
    (cherry picked from commit e6351ab11e2cfaea18ab963e32052ad110d2b899)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.6 release.