HA routers may fail to send out GARPs when node boots

Bug #1453855 reported by Assaf Muller
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Nir Magnezi
Juno
Fix Released
Undecided
Unassigned
Kilo
New
Undecided
Unassigned

Bug Description

When a node boots, it starts the OVS and L3 agents. As an example, in RDO systemd unit files, these services have no dependency. This means that the L3 agent can start before the OVS agent. It can start configuring routers before the OVS agent finished syncing with the server and starts processing ovsdb monitor updates. The result is that when the L3 agent finishes configuring an HA router, it starts up keepalived, which under certain conditions will transition to master and send our gratuitous ARPs before the OVS agent finishes plugging its ports. This means that the gratuitous ARP will be lost, but with the router acting as master, this can cause black holes.

Possible solutions:
* Introduce systemd dependencies, but this has its set of intricacies and it's hard to solve the above problem comprehensively just with this approach.
* Regardless, it's a good idea to use new keepalived flags:
garp_master_repeat <INTEGER> how often the gratuitous ARP after MASTER state transition should be repeated?
garp_master_refresh <INTEGER> Periodic delay in seconds sending gratuitous ARP while in MASTER state

These will be configurable and have sane defaults.

Assaf Muller (amuller)
description: updated
description: updated
Changed in neutron:
assignee: nobody → Sridhar Gaddam (sridhargaddam)
Revision history for this message
Assaf Muller (amuller) wrote :

Talked to Sridhar and Nir on IRC, Nir will take this over.

Changed in neutron:
assignee: Sridhar Gaddam (sridhargaddam) → Nir Magnezi (nmagnezi)
Assaf Muller (amuller)
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/201128

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/201128
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=141560b4221b9b58fd34405a8f94e826f92f35ff
Submitter: Jenkins
Branch: master

commit 141560b4221b9b58fd34405a8f94e826f92f35ff
Author: nmagnezi <email address hidden>
Date: Mon Jul 13 16:23:28 2015 +0300

    Adds garp_master_repeat and garp_master_refresh to keepalived.conf

    This patch aims to solve an issue when, in some cases, the L3 agent might
    start before the OVS agent during a node boot process.
    At such scenario, the L3 agent will configure the routers before the OVS agent
    done syncing with the server and starts processing ovsdb monitor updates.

    The result:
    1. L3 agent finishes configuring an HA router
    2. Keepalived spawned
    3. In case of transition to master state, it (keepalived) sends gratuitous ARPs
                                                                ^---The issue
    4. The OVS agent has just finished plugging its ports

    Meaning, the gratuitous ARP will be lost, but the router acting as master.
    Closes-Bug: #1453855

    Change-Id: I2cffa7c0ebad20bd50396265aef9fcfa64018744

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/206968

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/juno)

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/206998

Changed in neutron:
milestone: none → liberty-2
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (feature/pecan)

Fix proposed to branch: feature/pecan
Review: https://review.openstack.org/207903

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (feature/pecan)
Download full text (70.7 KiB)

Reviewed: https://review.openstack.org/207903
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9badcd249dab2d3330f0cd608496b59c9f44499a
Submitter: Jenkins
Branch: feature/pecan

commit 991ea00e6c115343eabecc62e86072175823f81f
Author: Moshe Levi <email address hidden>
Date: Thu Jul 30 12:04:15 2015 +0300

    SR-IOV: Fix SR-IOV agent to run ip link commands as root

    Commit https://review.openstack.org/#/c/155523/
    remove the remaining root_helper args, but didn't
    update the SR-IOV agent to execute them as root.

    This patch updates the agent to execute ip link commands
    as root and pass options argument as a list in the
    self._as_root method.

    Closes-Bug: #1479694
    Change-Id: I53cafd61845a69fae3a759fb7526950d655ffa20

commit 5b3bacedf6c014815bef03c2e821b5eb8ef92dcb
Author: Saksham Varma <email address hidden>
Date: Thu Jul 23 22:46:23 2015 -0700

    Moved out cisco n1kv mech driver and db models

    Moving out Cisco N1Kv database models and the mech driver
    to the openstack/networking-cisco repo as part of the second
    phase vendor-core decomposition

    Partial-Bug: #1479123
    Partial-implements: blueprint core-vendor-decomposition

    Change-Id: I65a704b238d8cbe2951a6912fa4f1e8518c6f412

commit 65ac8cd0a80548e115b8ec1d4cfd47a94422fddf
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Jul 29 20:44:01 2015 +0000

    Updated from global requirements

    Change-Id: I3a884a73d51df0f93b5cc88b3e3250d81bb1f455

commit f0e8356d04d04600c072a1c0f0bdf274ff19ab8c
Author: sridhargaddam <email address hidden>
Date: Mon Jul 27 03:46:48 2015 +0000

    Update dhcp agent cache for network:dhcp ports

    When a network with a dhcp_enabled subnet is scheduled on a dhcp
    agent, dhcp driver creates the network:dhcp port for the subnet.
    However, the port info is not updated in dhcp agents internal cache.
    Subsequently if the user deletes the network:dhcp port, the port is
    properly deleted on the server side (i.e., in the database) and when
    the port_delete_end notification is sent to the dhcp agent, it simply
    ignores it as the port entry would be missing in the cache. This patch
    fixes this issue by updating the dhcp agents cache when dhcp driver
    creates the network:dhcp port for the subnets.

    Closes-Bug: #1478426
    Change-Id: I69f5834dd964a4320c606c4e0aa2cdba70416943

commit cb60d0bb4e0cc0cba68f59fdf5f4e89d6ec52950
Author: changzhi <email address hidden>
Date: Thu Jul 16 10:14:16 2015 +0800

    Keep dns nameserver order consistency

    Currently, there is no dns servers prioritization for subnets
    for Neutron.

    Generally speaking, it is useful to keep the order of dns
    nameservers consistent. Add a new column named 'order' in table
    'dnsnameservers' and add nameserver into DB one by one.

    Closes-Bug: #1218629
    Implements: blueprint keep-dns-nameserver-orderconsistency
    Change-Id: Id937aea411397d39370368a4eb45be26c4eefa9e

commit b39e1469e824bc8bc79e1ecafa98825a94811c0b
Author: Salvatore Orlando <email address hidden>
Date: Tue Jun 23 04:54:2...

tags: added: in-feature-pecan
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/206968
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5d38dc59c463aaa86bd0c17ca2d19486b555e2be
Submitter: Jenkins
Branch: stable/kilo

commit 5d38dc59c463aaa86bd0c17ca2d19486b555e2be
Author: nmagnezi <email address hidden>
Date: Mon Jul 13 16:23:28 2015 +0300

    Adds garp_master_repeat and garp_master_refresh to keepalived.conf

    This patch aims to solve an issue when, in some cases, the L3 agent might
    start before the OVS agent during a node boot process.
    At such scenario, the L3 agent will configure the routers before the OVS agent
    done syncing with the server and starts processing ovsdb monitor updates.

    The result:
    1. L3 agent finishes configuring an HA router
    2. Keepalived spawned
    3. In case of transition to master state, it (keepalived) sends gratuitous ARPs
                                                                ^---The issue
    4. The OVS agent has just finished plugging its ports

    Meaning, the gratuitous ARP will be lost, but the router acting as master.
    Closes-Bug: #1453855

    Change-Id: I2cffa7c0ebad20bd50396265aef9fcfa64018744
    (cherry picked from commit 141560b4221b9b58fd34405a8f94e826f92f35ff)

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/juno)

Reviewed: https://review.openstack.org/206998
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=98b6cd8e7f2d482649ce2976ba5f2027a3ce1cbe
Submitter: Jenkins
Branch: stable/juno

commit 98b6cd8e7f2d482649ce2976ba5f2027a3ce1cbe
Author: nmagnezi <email address hidden>
Date: Mon Jul 13 16:23:28 2015 +0300

    Adds garp_master_repeat and garp_master_refresh to keepalived.conf

    This patch aims to solve an issue when, in some cases, the L3 agent might
    start before the OVS agent during a node boot process.
    At such scenario, the L3 agent will configure the routers before the OVS agent
    done syncing with the server and starts processing ovsdb monitor updates.

    The result:
    1. L3 agent finishes configuring an HA router
    2. Keepalived spawned
    3. In case of transition to master state, it (keepalived) sends gratuitous ARPs
                                                                ^---The issue
    4. The OVS agent has just finished plugging its ports

    Meaning, the gratuitous ARP will be lost, but the router acting as master.
    Closes-Bug: #1453855

    Change-Id: I2cffa7c0ebad20bd50396265aef9fcfa64018744
    (cherry picked from commit 141560b4221b9b58fd34405a8f94e826f92f35ff)

tags: added: in-stable-juno
Thierry Carrez (ttx)
Changed in neutron:
milestone: liberty-2 → 7.0.0
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/268348

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/liberty)

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/270652

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/kilo)

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/270653

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/268348
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=303cbc6b5b745aadf026627c4cf2e90e3862b586
Submitter: Jenkins
Branch: master

commit 303cbc6b5b745aadf026627c4cf2e90e3862b586
Author: Assaf Muller <email address hidden>
Date: Fri Jan 15 16:47:39 2016 -0500

    Fix L3 HA with IPv6

    We currently use garp_master_repeat and garp_master_refresh
    to solve bug 1453855. We need to spawn keepalived only after
    all of the qr/qg ports have been wired so that the
    initial GARP will be properly sent. Otherwise you get a routing
    black hole. In lieu of a proper sync method, we used those two keepalived
    options to send GARPs repeatedly:

    a) We did not know it never stops spamming the network
    b) It causes VMs to lose their IPv6 default gateway due to a keepalived
       bug, which has since been fixed, but it would need to be backported
       to every keepalived version on every distro. Here's the patch:
       https://github.com/acassen/keepalived/pull/200

    The solution this patch proposes is to drop the repeat and refresh
    keepalived options. This will fix the IPv6 bug but re-introduce bug
    1453855. So, this patch uses the delay option instead. It turns
    out keepalived sends a GARP when it transitions to MASTER, and then
    it waits a number of seconds determined by the delay option, and
    sends a GARP again. We'll use an aggressive 'delay' setting to make
    sure that when the node boots and the L3/L2 agents start, we'll
    give the L2 agent enough time to wire the ports as a stopgap solution.
    Note that this only affects initial synchronization time, not failover
    times. Failover times will continue to be fast because the ports
    are wired ahead of time, the initial GARP after the state transition
    to MASTER will be sent properly.

    Change-Id: I7a086472b8742828dae08ffd915c45e94fb4b94e
    Closes-Bug: #1520517
    Related-Bug: #1453855

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/270652
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c12bf81e97fcf21ce913339e581e76f657a8b4aa
Submitter: Jenkins
Branch: stable/liberty

commit c12bf81e97fcf21ce913339e581e76f657a8b4aa
Author: Assaf Muller <email address hidden>
Date: Fri Jan 15 16:47:39 2016 -0500

    Fix L3 HA with IPv6

    We currently use garp_master_repeat and garp_master_refresh
    to solve bug 1453855. We need to spawn keepalived only after
    all of the qr/qg ports have been wired so that the
    initial GARP will be properly sent. Otherwise you get a routing
    black hole. In lieu of a proper sync method, we used those two keepalived
    options to send GARPs repeatedly:

    a) We did not know it never stops spamming the network
    b) It causes VMs to lose their IPv6 default gateway due to a keepalived
       bug, which has since been fixed, but it would need to be backported
       to every keepalived version on every distro. Here's the patch:
       https://github.com/acassen/keepalived/pull/200

    The solution this patch proposes is to drop the repeat and refresh
    keepalived options. This will fix the IPv6 bug but re-introduce bug
    1453855. So, this patch uses the delay option instead. It turns
    out keepalived sends a GARP when it transitions to MASTER, and then
    it waits a number of seconds determined by the delay option, and
    sends a GARP again. We'll use an aggressive 'delay' setting to make
    sure that when the node boots and the L3/L2 agents start, we'll
    give the L2 agent enough time to wire the ports as a stopgap solution.
    Note that this only affects initial synchronization time, not failover
    times. Failover times will continue to be fast because the ports
    are wired ahead of time, the initial GARP after the state transition
    to MASTER will be sent properly.

    Conflicts:
     neutron/tests/functional/agent/test_l3_agent.py

    Change-Id: I7a086472b8742828dae08ffd915c45e94fb4b94e
    Closes-Bug: #1520517
    Related-Bug: #1453855
    (cherry picked from commit 303cbc6b5b745aadf026627c4cf2e90e3862b586)

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/270653
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=25ba9bd5873864e018f732a55d17297bcd9bd559
Submitter: Jenkins
Branch: stable/kilo

commit 25ba9bd5873864e018f732a55d17297bcd9bd559
Author: Assaf Muller <email address hidden>
Date: Fri Jan 15 16:47:39 2016 -0500

    Fix L3 HA with IPv6

    We currently use garp_master_repeat and garp_master_refresh
    to solve bug 1453855. We need to spawn keepalived only after
    all of the qr/qg ports have been wired so that the
    initial GARP will be properly sent. Otherwise you get a routing
    black hole. In lieu of a proper sync method, we used those two keepalived
    options to send GARPs repeatedly:

    a) We did not know it never stops spamming the network
    b) It causes VMs to lose their IPv6 default gateway due to a keepalived
       bug, which has since been fixed, but it would need to be backported
       to every keepalived version on every distro. Here's the patch:
       https://github.com/acassen/keepalived/pull/200

    The solution this patch proposes is to drop the repeat and refresh
    keepalived options. This will fix the IPv6 bug but re-introduce bug
    1453855. So, this patch uses the delay option instead. It turns
    out keepalived sends a GARP when it transitions to MASTER, and then
    it waits a number of seconds determined by the delay option, and
    sends a GARP again. We'll use an aggressive 'delay' setting to make
    sure that when the node boots and the L3/L2 agents start, we'll
    give the L2 agent enough time to wire the ports as a stopgap solution.
    Note that this only affects initial synchronization time, not failover
    times. Failover times will continue to be fast because the ports
    are wired ahead of time, the initial GARP after the state transition
    to MASTER will be sent properly.

    Conflicts:
     neutron/tests/functional/agent/test_l3_agent.py

    Change-Id: I7a086472b8742828dae08ffd915c45e94fb4b94e
    Closes-Bug: #1520517
    Related-Bug: #1453855
    (cherry picked from commit 303cbc6b5b745aadf026627c4cf2e90e3862b586)

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.