packet loss during backup L3 HA agent restart

Bug #1771841 reported by Slawek Kaplonski
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Slawek Kaplonski

Bug Description

When backup L3 agent in HA deployment is restarted, there is 20-30 seconds of data plane connectivity break to FloatingIP.
It looks that during start of L3 agent, it first enable IPv6 forwarding: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L106 and then disable it for non master node: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/ha_router.py#L419
That cause send of MLDv2 packet:

08:38:29.815180 Out fa:16:3e:9e:de:0e ethertype IPv6 (0x86dd), length 112: :: > ff02::16: HBH ICMP6, multicast listener report v2, 2 group record(s), length 48

and that breaks somehow connectivity as packets are sent to backup node instead of master one.

tags: added: queens-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/569289

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Related fix proposed to branch: master
Review: https://review.openstack.org/569424

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/569289

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/569424

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/569083
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3e9e2a5b4b2654864bb0e2a7d9b841eb03240147
Submitter: Zuul
Branch: master

commit 3e9e2a5b4b2654864bb0e2a7d9b841eb03240147
Author: Slawek Kaplonski <email address hidden>
Date: Thu May 17 12:16:52 2018 +0200

    Disable IPv6 forwarding by default on HA routers

    In case of HA routers IPv6 forwarding is not disabled by default and
    then enabled only on master node.
    Before this patch it was done in opposite way, so forwarding was
    enabled by default and then disabled on backup nodes.
    When forwarding was enabled/disabled for qg- port, MLDv2 packets are
    sent and that might lead to temportary packets loss as packets to
    FIP were sent to this backup node instead of master one.

    Related-Bug: #1771841

    Change-Id: Ia6b772e91c1f94612ca29d7082eca999372e60d6

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/571984

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/pike)

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/572055

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/571984
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2ed6e4d7ab253f23fd81d84b127139b9deb438fd
Submitter: Zuul
Branch: stable/queens

commit 2ed6e4d7ab253f23fd81d84b127139b9deb438fd
Author: Slawek Kaplonski <email address hidden>
Date: Thu May 17 12:16:52 2018 +0200

    Disable IPv6 forwarding by default on HA routers

    In case of HA routers IPv6 forwarding is not disabled by default and
    then enabled only on master node.
    Before this patch it was done in opposite way, so forwarding was
    enabled by default and then disabled on backup nodes.
    When forwarding was enabled/disabled for qg- port, MLDv2 packets are
    sent and that might lead to temportary packets loss as packets to
    FIP were sent to this backup node instead of master one.

    Related-Bug: #1771841

    Change-Id: Ia6b772e91c1f94612ca29d7082eca999372e60d6
    (cherry picked from commit 3e9e2a5b4b2654864bb0e2a7d9b841eb03240147)

tags: added: in-stable-queens
tags: added: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/572055
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ba9fd196a255170ae48149447a99e3d872d49976
Submitter: Zuul
Branch: stable/pike

commit ba9fd196a255170ae48149447a99e3d872d49976
Author: Slawek Kaplonski <email address hidden>
Date: Thu May 17 12:16:52 2018 +0200

    Disable IPv6 forwarding by default on HA routers

    In case of HA routers IPv6 forwarding is not disabled by default and
    then enabled only on master node.
    Before this patch it was done in opposite way, so forwarding was
    enabled by default and then disabled on backup nodes.
    When forwarding was enabled/disabled for qg- port, MLDv2 packets are
    sent and that might lead to temportary packets loss as packets to
    FIP were sent to this backup node instead of master one.

    Related-Bug: #1771841

    Change-Id: Ia6b772e91c1f94612ca29d7082eca999372e60d6
    (cherry picked from commit 3e9e2a5b4b2654864bb0e2a7d9b841eb03240147)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/ocata)

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/577396

Changed in neutron:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/ocata)

Reviewed: https://review.openstack.org/577396
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5e47f30572b7b9224a6ac6df59f0f1b6794c935f
Submitter: Zuul
Branch: stable/ocata

commit 5e47f30572b7b9224a6ac6df59f0f1b6794c935f
Author: Slawek Kaplonski <email address hidden>
Date: Thu May 17 12:16:52 2018 +0200

    Disable IPv6 forwarding by default on HA routers

    In case of HA routers IPv6 forwarding is not disabled by default and
    then enabled only on master node.
    Before this patch it was done in opposite way, so forwarding was
    enabled by default and then disabled on backup nodes.
    When forwarding was enabled/disabled for qg- port, MLDv2 packets are
    sent and that might lead to temportary packets loss as packets to
    FIP were sent to this backup node instead of master one.

    Related-Bug: #1771841

    Change-Id: Ia6b772e91c1f94612ca29d7082eca999372e60d6
    (cherry picked from commit 3e9e2a5b4b2654864bb0e2a7d9b841eb03240147)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.openstack.org/569458
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in neutron:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.