Functional tests for dvr ha routers are broken

[Slawek Kaplonski]

As a result of debugging issues reported in https://bugs.launchpad.net/neutron/+bug/1818614 I found an issue with functional tests:

test_dvr_ha_router_failover_with_gw,
test_dvr_ha_router_failover_without_gw,
test_dvr_ha_router_failover_with_gw_and_floatingip

from module neutron.tests.functional.agent.l3.test_dvr_router are broken.

First thing which is wrong there is fact that those tests creates 2 independent routers on 2 agents. It's not one router in "master" and "standby" mode. Please check in https://github.com/openstack/neutron/blob/68fd13af40ce2752d2919ca569d3d3fe00ff6444/neutron/tests/functional/agent/l3/test_dvr_router.py#L1503

Second issue is that it assumes that always router on agent1 will be "master" and that may be different sometimes as it depends on keepalived negotiations.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/641996

[LIU Yulong (dragon889)]

Log and examples:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_ha_router_failover_with_gw%5C%22%20AND%20message%3A%20%5C%22FAILED%5C%22
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_ha_router_failover_without_gw%5C%22%20AND%20message%3A%20%5C%22FAILED%5C%22
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22neutron.tests.functional.agent.l3.test_dvr_router.TestDvrRouter.test_dvr_ha_router_failover_with_gw_and_floatingip%5C%22%20AND%20message%3A%20%5C%22FAILED%5C%22

[LIU Yulong (dragon889)]

@Slawek,
For the first thing:
for that 2 routers creation, we have this:
https://github.com/openstack/neutron/blob/master/neutron/tests/common/l3_test_common.py#L164
They will finally have same VRRP virtual_router_id 1. We can say it is one router. So for only one test case it is fine.
But for concurrently running of same logical, it will still have such virtual_router_id 1, this does not make sense.

Then it raises your second question:
test case 1 has master VRRP instance in that 'master-agent1',
test case 2 will also try to use same virtual_router_id 1 to wait another master VRRP instance is up in 'master-agent1'.
They have same virtual_router_id 1, and our router is non-preemptive, so test case 2 will not get that router up in agent1.

Make sense?

I will try to give each failover case an independent vrrp_id to avoid such issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/642220

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/641996
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c2ab2626f8d4a1847ddc7d52b9449bc54c052ed7
Submitter: Zuul
Branch: master

commit c2ab2626f8d4a1847ddc7d52b9449bc54c052ed7
Author: Slawek Kaplonski <email address hidden>
Date: Fri Mar 8 13:34:15 2019 +0100

    Mark dvr_ha_router_failover tests as unstable

    According to related bug, those tests are now failing
    quite often due to reasons described in bug report.
    Lets mark them as unstable for now to make our gate
    more stable until we will fix those tests in proper way.

    Change-Id: Id803f2d23375a73322dbcc7337480d28c33cea82
    Related-Bug: #1819160

[LIU Yulong (dragon889)]

No more failure since 3-15-2019 after this fix:
https://review.openstack.org/#/c/642295/
IMO, this bug may also be fixed by it.

[LIU Yulong (dragon889)]

New log search:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22bug%201819160%5C%22%20and%20message%3A%20%5C%22SKIPPED%5C%22

[LIU Yulong (dragon889)]

Sorry, this log search:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%20%5C%22bug%201819160%5C%22%20AND%20message%3A%20%5C%22SKIPPED%5C%22

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/645225

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/647784

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/647784
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8cc480bd01d21f05296c77db231b706a7dc5b475
Submitter: Zuul
Branch: master

commit 8cc480bd01d21f05296c77db231b706a7dc5b475
Author: LIU Yulong <email address hidden>
Date: Tue Mar 26 23:47:11 2019 +0800

    Set HA failover bridges veth pair devices UP

    The test bridge veth pair devices is not up which cause the
    VRRP advertisement packet can not pass to each HA port. Then
    multiple master router is up. This patch just sets the veth
    pair devices up.

    Closes-Bug: #1819160
    Change-Id: I0e0d0311d73bce83d3c7341e7a0167917818b1ff

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/645225
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8f35331c918f83381951fc4367f43b2805d556ef
Submitter: Zuul
Branch: master

commit 8f35331c918f83381951fc4367f43b2805d556ef
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Mar 21 16:19:32 2019 +0000

    Check master/backup router status in DVR functional tests

    When two routers are created at the same time, we can't assume the
    status of each one. Instead of this, the status of each router is
    first checked and then compared to the other router status.

    Change-Id: If20a3a414986ea29fbfd50616761c14e5b249b2c
    Closes-Bug: #1819160

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/642220
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c69a87405af6b632a3edd9c6fe5ee63a5bc07eb1
Submitter: Zuul
Branch: master

commit c69a87405af6b632a3edd9c6fe5ee63a5bc07eb1
Author: LIU Yulong <email address hidden>
Date: Sat Mar 9 11:59:54 2019 +0800

    Give some HA router case specific resources

    1. give each HA failover case an independent vrrp_id
    2. give each HA port an independent IP address, so the
    interface IPs for router HA ports will be:
        169.254.192.100 and 169.254.192.101
        169.254.192.102 and 169.254.192.103
        169.254.192.104 and 169.254.192.105
        169.254.192.106 and 169.254.192.107
    VIP of each case will be:
        169.254.0.10/24
        169.254.0.11/24
        169.254.0.12/24
        169.254.0.13/24
        169.254.0.14/24

    Closes-Bug: #1819160
    Change-Id: I1216d96af40449ec16a852cc1f6c4f15c85f4546

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.openstack.org/653386

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.openstack.org/653391

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.openstack.org/653392

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/653393

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/653394

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/653396

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/653397

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/653398

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/653399

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/653400

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/653402

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/653403

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653397
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0e808b4a6aa901171b6e93b5225d570d24ad3c07
Submitter: Zuul
Branch: stable/queens

commit 0e808b4a6aa901171b6e93b5225d570d24ad3c07
Author: LIU Yulong <email address hidden>
Date: Tue Mar 26 23:47:11 2019 +0800

    Set HA failover bridges veth pair devices UP

    The test bridge veth pair devices is not up which cause the
    VRRP advertisement packet can not pass to each HA port. Then
    multiple master router is up. This patch just sets the veth
    pair devices up.

    Closes-Bug: #1819160
    Change-Id: I0e0d0311d73bce83d3c7341e7a0167917818b1ff
    (cherry picked from commit 8cc480bd01d21f05296c77db231b706a7dc5b475)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653398
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8ba589994236862834aa1bf35907b84ffa3fda9b
Submitter: Zuul
Branch: stable/queens

commit 8ba589994236862834aa1bf35907b84ffa3fda9b
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Mar 21 16:19:32 2019 +0000

    Check master/backup router status in DVR functional tests

    When two routers are created at the same time, we can't assume the
    status of each one. Instead of this, the status of each router is
    first checked and then compared to the other router status.

    Change-Id: If20a3a414986ea29fbfd50616761c14e5b249b2c
    Closes-Bug: #1819160
    (cherry picked from commit 8f35331c918f83381951fc4367f43b2805d556ef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653393
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=827d6434d76fe52226ff418644389ac35b685a52
Submitter: Zuul
Branch: stable/rocky

commit 827d6434d76fe52226ff418644389ac35b685a52
Author: LIU Yulong <email address hidden>
Date: Tue Mar 26 23:47:11 2019 +0800

    Set HA failover bridges veth pair devices UP

    The test bridge veth pair devices is not up which cause the
    VRRP advertisement packet can not pass to each HA port. Then
    multiple master router is up. This patch just sets the veth
    pair devices up.

    Closes-Bug: #1819160
    Change-Id: I0e0d0311d73bce83d3c7341e7a0167917818b1ff
    (cherry picked from commit 8cc480bd01d21f05296c77db231b706a7dc5b475)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653400
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a5ca42d2428ab3de74b4459e9510dc414d4e14b5
Submitter: Zuul
Branch: stable/pike

commit a5ca42d2428ab3de74b4459e9510dc414d4e14b5
Author: LIU Yulong <email address hidden>
Date: Tue Mar 26 23:47:11 2019 +0800

    Set HA failover bridges veth pair devices UP

    The test bridge veth pair devices is not up which cause the
    VRRP advertisement packet can not pass to each HA port. Then
    multiple master router is up. This patch just sets the veth
    pair devices up.

    Closes-Bug: #1819160
    Change-Id: I0e0d0311d73bce83d3c7341e7a0167917818b1ff
    (cherry picked from commit 8cc480bd01d21f05296c77db231b706a7dc5b475)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653402
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9e8ea8edbca61138116159ba9ae6acdd96cf8f94
Submitter: Zuul
Branch: stable/pike

commit 9e8ea8edbca61138116159ba9ae6acdd96cf8f94
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Mar 21 16:19:32 2019 +0000

    Check master/backup router status in DVR functional tests

    When two routers are created at the same time, we can't assume the
    status of each one. Instead of this, the status of each router is
    first checked and then compared to the other router status.

    Conflicts:
        neutron/tests/functional/agent/l3/test_dvr_router.py

    Change-Id: If20a3a414986ea29fbfd50616761c14e5b249b2c
    Closes-Bug: #1819160
    (cherry picked from commit 8f35331c918f83381951fc4367f43b2805d556ef)
    (cherry picked from commit 8ba589994236862834aa1bf35907b84ffa3fda9b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653403
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=26f6d5fd73c311c48768f570bf1bc088b746310a
Submitter: Zuul
Branch: stable/pike

commit 26f6d5fd73c311c48768f570bf1bc088b746310a
Author: LIU Yulong <email address hidden>
Date: Sat Mar 9 11:59:54 2019 +0800

    Give some HA router case specific resources

    1. give each HA failover case an independent vrrp_id
    2. give each HA port an independent IP address, so the
    interface IPs for router HA ports will be:
        169.254.192.100 and 169.254.192.101
        169.254.192.102 and 169.254.192.103
        169.254.192.104 and 169.254.192.105
        169.254.192.106 and 169.254.192.107
    VIP of each case will be:
        169.254.0.10/24
        169.254.0.11/24
        169.254.0.12/24
        169.254.0.13/24
        169.254.0.14/24

    Conflicts:
        neutron/tests/functional/agent/l3/test_dvr_router.py

    Closes-Bug: #1819160
    Change-Id: I1216d96af40449ec16a852cc1f6c4f15c85f4546
    (cherry picked from commit c69a87405af6b632a3edd9c6fe5ee63a5bc07eb1)
    (cherry picked from commit 2c5957f56d46d99b8eb1e7651afcbf8fa9f426a0)
    (cherry picked from commit c50bdf232992f0431f15cbe7cfb9f7eaa3a9ff0c)
    (cherry picked from commit 7b2a8f795fa3986e00cb5178e9e95a90951c2be4)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653394
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8280d75723200ceec9109dc7cc8c992d1f001314
Submitter: Zuul
Branch: stable/rocky

commit 8280d75723200ceec9109dc7cc8c992d1f001314
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Mar 21 16:19:32 2019 +0000

    Check master/backup router status in DVR functional tests

    When two routers are created at the same time, we can't assume the
    status of each one. Instead of this, the status of each router is
    first checked and then compared to the other router status.

    Change-Id: If20a3a414986ea29fbfd50616761c14e5b249b2c
    Closes-Bug: #1819160
    (cherry picked from commit 8f35331c918f83381951fc4367f43b2805d556ef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653396
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c50bdf232992f0431f15cbe7cfb9f7eaa3a9ff0c
Submitter: Zuul
Branch: stable/rocky

commit c50bdf232992f0431f15cbe7cfb9f7eaa3a9ff0c
Author: LIU Yulong <email address hidden>
Date: Sat Mar 9 11:59:54 2019 +0800

    Give some HA router case specific resources

    1. give each HA failover case an independent vrrp_id
    2. give each HA port an independent IP address, so the
    interface IPs for router HA ports will be:
        169.254.192.100 and 169.254.192.101
        169.254.192.102 and 169.254.192.103
        169.254.192.104 and 169.254.192.105
        169.254.192.106 and 169.254.192.107
    VIP of each case will be:
        169.254.0.10/24
        169.254.0.11/24
        169.254.0.12/24
        169.254.0.13/24
        169.254.0.14/24

    Conflicts:
        neutron/tests/common/l3_test_common.py

    Closes-Bug: #1819160
    Change-Id: I1216d96af40449ec16a852cc1f6c4f15c85f4546
    (cherry picked from commit c69a87405af6b632a3edd9c6fe5ee63a5bc07eb1)
    (cherry picked from commit 2c5957f56d46d99b8eb1e7651afcbf8fa9f426a0)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653399
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b3a12e15f428afbd461ed0b4ecfb5bae59178809
Submitter: Zuul
Branch: stable/queens

commit b3a12e15f428afbd461ed0b4ecfb5bae59178809
Author: LIU Yulong <email address hidden>
Date: Sat Mar 9 11:59:54 2019 +0800

    Give some HA router case specific resources

    1. give each HA failover case an independent vrrp_id
    2. give each HA port an independent IP address, so the
    interface IPs for router HA ports will be:
        169.254.192.100 and 169.254.192.101
        169.254.192.102 and 169.254.192.103
        169.254.192.104 and 169.254.192.105
        169.254.192.106 and 169.254.192.107
    VIP of each case will be:
        169.254.0.10/24
        169.254.0.11/24
        169.254.0.12/24
        169.254.0.13/24
        169.254.0.14/24

    Conflicts:
        neutron/tests/common/l3_test_common.py

    Closes-Bug: #1819160
    Change-Id: I1216d96af40449ec16a852cc1f6c4f15c85f4546
    (cherry picked from commit c69a87405af6b632a3edd9c6fe5ee63a5bc07eb1)
    (cherry picked from commit 2c5957f56d46d99b8eb1e7651afcbf8fa9f426a0)
    (cherry picked from commit c50bdf232992f0431f15cbe7cfb9f7eaa3a9ff0c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653391
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4e43ce541483b9026f02d3b55cfe91d5db99c64e
Submitter: Zuul
Branch: stable/stein

commit 4e43ce541483b9026f02d3b55cfe91d5db99c64e
Author: LIU Yulong <email address hidden>
Date: Tue Mar 26 23:47:11 2019 +0800

    Set HA failover bridges veth pair devices UP

    The test bridge veth pair devices is not up which cause the
    VRRP advertisement packet can not pass to each HA port. Then
    multiple master router is up. This patch just sets the veth
    pair devices up.

    Closes-Bug: #1819160
    Change-Id: I0e0d0311d73bce83d3c7341e7a0167917818b1ff
    (cherry picked from commit 8cc480bd01d21f05296c77db231b706a7dc5b475)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653386
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=78555ee3462a99f1f8f0c6b9dd6464d598ecf289
Submitter: Zuul
Branch: stable/stein

commit 78555ee3462a99f1f8f0c6b9dd6464d598ecf289
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu Mar 21 16:19:32 2019 +0000

    Check master/backup router status in DVR functional tests

    When two routers are created at the same time, we can't assume the
    status of each one. Instead of this, the status of each router is
    first checked and then compared to the other router status.

    Change-Id: If20a3a414986ea29fbfd50616761c14e5b249b2c
    Closes-Bug: #1819160
    (cherry picked from commit 8f35331c918f83381951fc4367f43b2805d556ef)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/653392
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5d0c27e8360831b365f85da1551cc09fb6b4ddfc
Submitter: Zuul
Branch: stable/stein

commit 5d0c27e8360831b365f85da1551cc09fb6b4ddfc
Author: LIU Yulong <email address hidden>
Date: Sat Mar 9 11:59:54 2019 +0800

    Give some HA router case specific resources

    1. give each HA failover case an independent vrrp_id
    2. give each HA port an independent IP address, so the
    interface IPs for router HA ports will be:
        169.254.192.100 and 169.254.192.101
        169.254.192.102 and 169.254.192.103
        169.254.192.104 and 169.254.192.105
        169.254.192.106 and 169.254.192.107
    VIP of each case will be:
        169.254.0.10/24
        169.254.0.11/24
        169.254.0.12/24
        169.254.0.13/24
        169.254.0.14/24

    Closes-Bug: #1819160
    Change-Id: I1216d96af40449ec16a852cc1f6c4f15c85f4546
    (cherry picked from commit c69a87405af6b632a3edd9c6fe5ee63a5bc07eb1)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron pike-eol release.