L3 HA connectivity to GW port can be broken after reboot of backup node

[Slawek Kaplonski]

When neutron router is on some network node in backup state (other network node is "active" for this router), and such network node will be rebooted it may happen that connectivity to router's gateway port will be broken.
It can happen due to race between L3 agent and OVS agent and is easier to reproduce when You have many routers in backup state on such node.
I was testing it with 10 routers, all in backup state. In such case 1 or 2 routers had got broken connectivity after reboot of host.

It is like that because when L3 agent adds interface to the router, it checks if there is any IPv6 link-local address on interface and if there is, it flush such IPv6 addresses and adds them to keepalived config. So keepalived can manage such IPs as any other IP address from this interface.
But the problem is that when IPv6 address is removed from the interface, it sends MLDv2 packets to unsubsribe from multicast group. And if those packets will go out from host e.g. to ToR switch, such switch will learn that MAC address of gw port is on wrong host (this rebooted one instead of one where router is in master state).

Thos MLDv2 packets aren't send to the wire for each router but only for some of them due to race.
Basically new qg-XXX port is created in br-int by L3 agent with DEAD_VLAN_TAG (4095) and than both agents, L3 and OVS are configuring it. If L3 agent flush IPv6 addresses from this interface BEFORE OVS agent sets proper tag (local_vlan_id) for the port, than all is fine because MLDv2 packets are dropped. But if L3 agent will flush AFTER tag is changed, than MLDv2 packets are send to the wire and cause ingress connectivity break.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/702856

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/707406

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/712474

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/702856

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707406
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c52029c39aa824a67095fbbf9e59eff769d92587
Submitter: Zuul
Branch: master

commit c52029c39aa824a67095fbbf9e59eff769d92587
Author: LIU Yulong <email address hidden>
Date: Thu Oct 31 19:06:37 2019 +0800

    Do not link up HA router gateway in backup node

    L3 router will set its devices link up by default.
    For HA routers, the gateway device will be pluged
    in all scheduled hosts. When the gateway deivce is
    up in backup node, it will send out IPv6 related
    packets (MLDv2) according to some kernal config.
    This will cause the physical fabric think that the
    gateway MAC is now working in the backup node. And
    finally the master node L3 traffic will be broken.

    This patch sets the backup gateway device link down
    by default. When the VRRP sets the master state in
    one host, the L3 agent state change procedure will
    do link up action for the gateway device.

    Closes-Bug: #1859832
    Change-Id: I8dca2c1a2f8cb467cfb44420f0eea54ca0932b05

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/717718

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/717720

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/717740

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/717741

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717718
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fe62f4db26419d1977fca8d88aba189fae106202
Submitter: Zuul
Branch: stable/train

commit fe62f4db26419d1977fca8d88aba189fae106202
Author: LIU Yulong <email address hidden>
Date: Thu Oct 31 19:06:37 2019 +0800

    Do not link up HA router gateway in backup node

    L3 router will set its devices link up by default.
    For HA routers, the gateway device will be pluged
    in all scheduled hosts. When the gateway deivce is
    up in backup node, it will send out IPv6 related
    packets (MLDv2) according to some kernal config.
    This will cause the physical fabric think that the
    gateway MAC is now working in the backup node. And
    finally the master node L3 traffic will be broken.

    This patch sets the backup gateway device link down
    by default. When the VRRP sets the master state in
    one host, the L3 agent state change procedure will
    do link up action for the gateway device.

    Closes-Bug: #1859832
    Change-Id: I8dca2c1a2f8cb467cfb44420f0eea54ca0932b05
    (cherry picked from commit c52029c39aa824a67095fbbf9e59eff769d92587)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717720
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b9a29681003b7f975d7bc9687399cafb5549010a
Submitter: Zuul
Branch: stable/stein

commit b9a29681003b7f975d7bc9687399cafb5549010a
Author: LIU Yulong <email address hidden>
Date: Thu Oct 31 19:06:37 2019 +0800

    Do not link up HA router gateway in backup node

    L3 router will set its devices link up by default.
    For HA routers, the gateway device will be pluged
    in all scheduled hosts. When the gateway deivce is
    up in backup node, it will send out IPv6 related
    packets (MLDv2) according to some kernal config.
    This will cause the physical fabric think that the
    gateway MAC is now working in the backup node. And
    finally the master node L3 traffic will be broken.

    This patch sets the backup gateway device link down
    by default. When the VRRP sets the master state in
    one host, the L3 agent state change procedure will
    do link up action for the gateway device.

    Closes-Bug: #1859832
    Change-Id: I8dca2c1a2f8cb467cfb44420f0eea54ca0932b05
    (cherry picked from commit c52029c39aa824a67095fbbf9e59eff769d92587)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717740
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=69249ae8c1273ef6c6ab5e29870f2e878e3d0e50
Submitter: Zuul
Branch: stable/rocky

commit 69249ae8c1273ef6c6ab5e29870f2e878e3d0e50
Author: LIU Yulong <email address hidden>
Date: Thu Oct 31 19:06:37 2019 +0800

    Do not link up HA router gateway in backup node

    L3 router will set its devices link up by default.
    For HA routers, the gateway device will be pluged
    in all scheduled hosts. When the gateway deivce is
    up in backup node, it will send out IPv6 related
    packets (MLDv2) according to some kernal config.
    This will cause the physical fabric think that the
    gateway MAC is now working in the backup node. And
    finally the master node L3 traffic will be broken.

    This patch sets the backup gateway device link down
    by default. When the VRRP sets the master state in
    one host, the L3 agent state change procedure will
    do link up action for the gateway device.

    Conflicts:
        neutron/agent/l3/router_info.py

    Closes-Bug: #1859832
    Change-Id: I8dca2c1a2f8cb467cfb44420f0eea54ca0932b05
    (cherry picked from commit c52029c39aa824a67095fbbf9e59eff769d92587)
    (cherry picked from commit b9a29681003b7f975d7bc9687399cafb5549010a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/717741
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=647b24288e93d83f7dd1c3db88a0cf5efac0ff3e
Submitter: Zuul
Branch: stable/queens

commit 647b24288e93d83f7dd1c3db88a0cf5efac0ff3e
Author: LIU Yulong <email address hidden>
Date: Thu Oct 31 19:06:37 2019 +0800

    Do not link up HA router gateway in backup node

    L3 router will set its devices link up by default.
    For HA routers, the gateway device will be pluged
    in all scheduled hosts. When the gateway deivce is
    up in backup node, it will send out IPv6 related
    packets (MLDv2) according to some kernal config.
    This will cause the physical fabric think that the
    gateway MAC is now working in the backup node. And
    finally the master node L3 traffic will be broken.

    This patch sets the backup gateway device link down
    by default. When the VRRP sets the master state in
    one host, the L3 agent state change procedure will
    do link up action for the gateway device.

    Conflicts:
        neutron/agent/l3/router_info.py
        neutron/agent/linux/interface.py

    Closes-Bug: #1859832
    Change-Id: I8dca2c1a2f8cb467cfb44420f0eea54ca0932b05
    (cherry picked from commit c52029c39aa824a67095fbbf9e59eff769d92587)
    (cherry picked from commit b9a29681003b7f975d7bc9687399cafb5549010a)
    (cherry picked from commit 41e8689234e87b6bf77a9849776d6a05cfcb1a71)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/712474
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Vasyl Saienko (vsaienko)]

The solution implemented in this bug now put all HA capabilities of dataplane traffic into python processes (which for example during upgrade might not be running at all). And on real loaded environment switching router from Backup to Master might takes an hours. Please consider reverting this change, or move part that brings interface UP into keepalived notification script which executed on transiion from Backup to MASTER

[LIU Yulong (dragon889)]

@Vasyl Saienko,

thanks for the information. IMO, you should add a new bug for that issue "Backup to Master might takes an hours". Some valuable information for new bugs are logs, config options, and how to reproduce it.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/836198

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/839671

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/839671
Committed: https://opendev.org/openstack/neutron/commit/5288593fafe6636fc14b8873465866d20de26935
Submitter: "Zuul (22348)"
Branch: master

commit 5288593fafe6636fc14b8873465866d20de26935
Author: Damian Dabrowski <email address hidden>
Date: Thu Apr 28 02:54:25 2022 +0200

    [L3-HA] Disable automatic link-local address assignment for HA routers

    In order to get both [1] and [2] fixed, we set
    `net.ipv6.conf.all.addr_gen_mode=1` in HA router namespace to
    prevent auto-assigning link-local address(lla) to the interfaces.
    We don't need lla auto-assignment as keepalived manages them.
    With this change, we will have link-local addresses only on active
    router, which will prevent 'dadfailed' and MLD packets will not be
    sent from standby router.

    Previously we also reverted [3] to always keep qg-* interface up on both
    active&standby router's instance, no matter if keepalived is started or
    not.
    Without link-local address assigned, backup router's instance won't
    send any packets, so I see no reason to keep qg-* interface down.

    [1] https://bugs.launchpad.net/neutron/+bug/1952907
    [2] https://bugs.launchpad.net/neutron/+bug/1859832
    [3] https://review.opendev.org/c/openstack/neutron/+/834162

    Closes-Bug: #1952907
    Related-Bug: #1859832
    Depends-On: https://review.opendev.org/c/openstack/neutron/+/834162
    Change-Id: I306f14aa6b7e8bb69a81f441be337bc1a584d3b2

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/836198
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "liuyulong <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/712474