IPv6 networking broken by garp_master_* keepalived settings when using HA routers

[Tore Anderson]

http://git.openstack.org/cgit/openstack/neutron/commit/?id=5d38dc5 added the "garp_master_repeat 5" and "garp_master_refresh 10". This badly breaks networking to the point where it is completely unsuable:

First of all, this setting causes Keepalived to constantly spam five unsolicited neighbour advertisements every ten seconds, as shown in this tcpdump (the active router is fe80::f816:3eff:feb8:3857):

ubuntu@test:~$ sudo tcpdump -i eth0 host fe80::f816:3eff:feb8:3857
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
10:08:09.459090 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, router advertisement, length 56
10:08:12.566305 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, router advertisement, length 56
10:08:15.638044 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, router advertisement, length 56
10:08:18.039185 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32
10:08:18.039275 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32
10:08:18.039496 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32
10:08:18.039581 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32
10:08:18.039595 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32
10:08:21.952451 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, router advertisement, length 56
10:08:28.046863 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32
10:08:28.046944 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32
10:08:28.047045 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32
10:08:28.047986 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32
10:08:28.048033 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, neighbor advertisement, tgt is fe80::f816:3eff:feb8:3857, length 32
10:08:30.931114 IP6 fe80::f816:3eff:feb8:3857 > ip6-allnodes: ICMP6, router advertisement, length 56

That's not a problem in itself. The problem is however that these neighbour advertisements causes the instance to loose its default route, which stays gone until the next router advertisement packet arrives:

ubuntu@test:~$ sudo ip -6 monitor route | ts
Nov 27 10:08:09 default via fe80::f816:3eff:feb8:3857 dev eth0 proto ra metric 1024
Nov 27 10:08:18 Deleted default via fe80::f816:3eff:feb8:3857 dev eth0 proto ra metric 1024 expires 27sec hoplimit 64
Nov 27 10:08:21 default via fe80::f816:3eff:feb8:3857 dev eth0 proto ra metric 1024
Nov 27 10:08:28 Deleted default via fe80::f816:3eff:feb8:3857 dev eth0 proto ra metric 1024 expires 23sec hoplimit 64
Nov 27 10:08:30 default via fe80::f816:3eff:feb8:3857 dev eth0 proto ra metric 1024

These periods without a default route obviously badly breaks network connectivity for the node, which is easily observable by a simple ping test:

ubuntu@test:~$ ping6 2a02:c0::1 2>&1 | ts
Nov 27 10:08:15 PING 2a02:c0::1(2a02:c0::1) 56 data bytes
Nov 27 10:08:15 64 bytes from 2a02:c0::1: icmp_seq=1 ttl=62 time=0.570 ms
Nov 27 10:08:16 64 bytes from 2a02:c0::1: icmp_seq=2 ttl=62 time=0.873 ms
Nov 27 10:08:17 64 bytes from 2a02:c0::1: icmp_seq=3 ttl=62 time=0.666 ms
Nov 27 10:08:18 ping: sendmsg: Network is unreachable
Nov 27 10:08:19 ping: sendmsg: Network is unreachable
Nov 27 10:08:20 ping: sendmsg: Network is unreachable
Nov 27 10:08:21 ping: sendmsg: Network is unreachable
Nov 27 10:08:22 64 bytes from 2a02:c0::1: icmp_seq=8 ttl=62 time=1.42 ms
Nov 27 10:08:23 64 bytes from 2a02:c0::1: icmp_seq=9 ttl=62 time=0.785 ms
Nov 27 10:08:24 64 bytes from 2a02:c0::1: icmp_seq=10 ttl=62 time=0.712 ms
Nov 27 10:08:25 64 bytes from 2a02:c0::1: icmp_seq=11 ttl=62 time=0.724 ms
Nov 27 10:08:26 64 bytes from 2a02:c0::1: icmp_seq=12 ttl=62 time=0.864 ms
Nov 27 10:08:27 64 bytes from 2a02:c0::1: icmp_seq=13 ttl=62 time=0.652 ms
Nov 27 10:08:28 ping: sendmsg: Network is unreachable
Nov 27 10:08:29 ping: sendmsg: Network is unreachable
Nov 27 10:08:30 ping: sendmsg: Network is unreachable
Nov 27 10:08:31 64 bytes from 2a02:c0::1: icmp_seq=17 ttl=62 time=1.50 ms
Nov 27 10:08:32 64 bytes from 2a02:c0::1: icmp_seq=18 ttl=62 time=0.683 ms
Nov 27 10:08:33 64 bytes from 2a02:c0::1: icmp_seq=19 ttl=62 time=0.677 ms
Nov 27 10:08:34 64 bytes from 2a02:c0::1: icmp_seq=20 ttl=62 time=0.729 ms

Removing the garp_master_* settings from keepalived.conf and HUP-ing it solves the problem and makes the network usable again.

So the question you might be asking yourself at this point is probably "why do the neighbour advertisement packets cause the default route to be removed"? I am 99.9% certain it is because the NAs have the "router" flag set to 0. (I have not verified it 100%, because I do not know how to configure Keepalived to set the "router" flag, if that is even possible.)

ubuntu@test:~$ sudo tshark -i eth0 -f 'icmp6 and ip6[40] == 136 and src host fe80::f816:3eff:feb8:3857' -c 1 -V
tshark: Lua: Error during loading:
 [string "/usr/share/wireshark/init.lua"]:46: dofile has been disabled due to running Wireshark as superuser. See http://wiki.wireshark.org/CaptureSetup/CapturePrivileges for help in running Wireshark as an unprivileged user.
Running as user "root" and group "root". This could be dangerous.
Capturing on 'eth0'
Frame 1: 86 bytes on wire (688 bits), 86 bytes captured (688 bits) on interface 0
    Interface id: 0
    Encapsulation type: Ethernet (1)
    Arrival Time: Nov 27, 2015 10:18:48.182563000 UTC
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1448619528.182563000 seconds
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 86 bytes (688 bits)
    Capture Length: 86 bytes (688 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ipv6:icmpv6]
Ethernet II, Src: fa:16:3e:b8:38:57 (fa:16:3e:b8:38:57), Dst: IPv6mcast_00:00:00:01 (33:33:00:00:00:01)
    Destination: IPv6mcast_00:00:00:01 (33:33:00:00:00:01)
        Address: IPv6mcast_00:00:00:01 (33:33:00:00:00:01)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
    Source: fa:16:3e:b8:38:57 (fa:16:3e:b8:38:57)
        Address: fa:16:3e:b8:38:57 (fa:16:3e:b8:38:57)
        .... ..1. .... .... .... .... = LG bit: Locally administered address (this is NOT the factory default)
        .... ...0 .... .... .... .... = IG bit: Individual address (unicast)
    Type: IPv6 (0x86dd)
Internet Protocol Version 6, Src: fe80::f816:3eff:feb8:3857 (fe80::f816:3eff:feb8:3857), Dst: ff02::1 (ff02::1)
    0110 .... = Version: 6
        [0110 .... = This field makes the filter "ip.version == 6" possible: 6]
    .... 0000 0000 .... .... .... .... .... = Traffic class: 0x00000000
        .... 0000 00.. .... .... .... .... .... = Differentiated Services Field: Default (0x00000000)
        .... .... ..0. .... .... .... .... .... = ECN-Capable Transport (ECT): Not set
        .... .... ...0 .... .... .... .... .... = ECN-CE: Not set
    .... .... .... 0000 0000 0000 0000 0000 = Flowlabel: 0x00000000
    Payload length: 32
    Next header: ICMPv6 (58)
    Hop limit: 255
    Source: fe80::f816:3eff:feb8:3857 (fe80::f816:3eff:feb8:3857)
    Destination: ff02::1 (ff02::1)
    [Source GeoIP: Unknown]
    [Destination GeoIP: Unknown]
Internet Control Message Protocol v6
    Type: Neighbor Advertisement (136)
    Code: 0
    Checksum: 0x0c2b [correct]
    Flags: 0x20000000
        0... .... .... .... .... .... .... .... = Router: Not set <--------------- HERE
        .0.. .... .... .... .... .... .... .... = Solicited: Not set
        ..1. .... .... .... .... .... .... .... = Override: Set
        ...0 0000 0000 0000 0000 0000 0000 0000 = Reserved: 0
    Target Address: fe80::f816:3eff:feb8:3857 (fe80::f816:3eff:feb8:3857)
    ICMPv6 Option (Target link-layer address : fa:16:3e:b8:38:57)
        Type: Target link-layer address (2)
        Length: 1 (8 bytes)
        Link-layer address: fa:16:3e:b8:38:57 (fa:16:3e:b8:38:57)

Tore

[Tore Anderson (toreanderson)]

FYI, I've posted a simple patch to make Keepalived set the router flag here: https://github.com/acassen/keepalived/pull/200

[Assaf Muller (amuller)]

Can we have more information about the bug?

Does it break IPv6 for all address modes (slaac etc), or just some.

Assuming it breaks IPv6 entirely we need to do some thinking. I don't know how to workaround this issue without fixing it in keepalived itself. If that is the case, it could take a while (And
even then we'd have to make sure to get the fix in all relevant
versions of keepalived for all distros for OpenStack Juno/Kilo/Liberty, and that is... Difficult). We could revert: https://review.openstack.org/#/q/I2cffa7c0ebad20bd50396265aef9fcfa64018744,n,z (Note that it was merged to Juno, Kilo and Liberty) and come up with an alternate solution to the bug that it was solving https://bugs.launchpad.net/neutron/+bug/1453855. That's very difficult too.

[Steve Brown (stephen-brown-2)]

I am also being affected by this issue, running on Liberty with a ha router.

symptoms exactly like the ones described by toreanderson (ping fine for a few seconds then network is unreachable) ,
I could see the default route being removed at the same time as a number of neighbor advertisements, the route was put back when the next RA was received. Removing the "garp_master_repeat 5" and "garp_master_refresh 10" from the relevant keepalived config fixed the issue.

the problem was encountered with the following address modes:

--ipv6_ra_mode slaac --ipv6-address-mode slaac
and
--ipv6_ra_mode dhcpv6-stateless --ipv6-address-mode dhcpv6-stateless

[Tore Anderson (toreanderson)]

Hello Assaf, I'm happy to help out with any further information you might need:

The issue does indeed impact all address assignment modes. This is as expected, as address assignment and default route assignment are two orthogonal mechanisms in IPv6, and it is the default route that is continuously removed and re-added. The instance's assigned address is stable and is not impacted.

It does break IPv6 entirely, to the point where the instance's external network connectivity is really completely unusable for any sort of production purpose. The only thing that keeps working reliably is internal traffic between instances on the same subnet. The way I see it, the issue is without question critical - an OpenStack infrastructure where the instances do not have working network connectivity is only marginally more useful than having no OpenStack infrastructure at all.

My one-line patch to Keepalived got quickly applied upstream, see https://github.com/acassen/keepalived/commit/be69d87151325b7d906ade988b519ca35fbb25cf. However, as you rightly point out, this does not help in the short or probably even medium term as the fix is not included in any stable release of Keepalived, much less any binary distro packages.

One could accurately argue that the bug here is really in Keepalived, and not in Neutron. However, before commit 5d38dc5 it had virtually no impact, as the problematic (default-route-removing) unsolicited Neighbour Advertisements got sent only for a few seconds after a Neutron router failover event. This prolonged the network downtime/convergence time following such a failure, but only with a few seconds. That was probably not even noticeable amidst all the other transient connectivity issues an instance would likely experience during a router failover. After commit 5d38dc5 however, the network became perpetually broken. Therefore I do think that this urgently needs to be fixed/reverted in Neutron.

A final comment: The solution for bug #1453855 in commit 5d38dc5 is far from ideal. Even if didn't cause the issues described in this report, I'd still suggest you reconsider that approach. The reason for that is that it causes a constant and perpetual 0.5pps stream of multicast IPv6 traffic per subnet (not including the IPv4 gratuitous ARP packets that also occur). I believe most operators are interested in limiting the amount of so-called "BUM" traffic (Broadcast, Unknown Unicast, and Multicast) in their networks because this traffic is much more expensive/difficult to deliver than regular unicast traffic. Commit 5d38dc5 goes the exact opposite way - endlessly spamming the network with tons of BUM traffic in order to solve a transient and short-lived service ordering/dependency issue occurring (as I understand it) only when a network node running the Neutron L3 agent boots up. It's an extremely blunt approach.

Anyway, please do not hesitate to ask again if you need any further information from me - I'm very interested in helping out any way I can in getting this issue promptly resolved.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/268348

[Tore Anderson (toreanderson)]

I tested the fix proposed in https://review.openstack.org/268348 by manually editing the /var/lib/neutron/ha_confs/*/keepalived.conf files and then SIGHUP-ing Keepalived.

It works - the change successfully stabilised the network, there is no more packet loss or other connectivity issues.

For what it's worth, the version of neutron-l3-agent I'm running is 1:2015.1.2-0ubuntu1~cloud0.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/270652

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/270653

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/268348
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=303cbc6b5b745aadf026627c4cf2e90e3862b586
Submitter: Jenkins
Branch: master

commit 303cbc6b5b745aadf026627c4cf2e90e3862b586
Author: Assaf Muller <email address hidden>
Date: Fri Jan 15 16:47:39 2016 -0500

    Fix L3 HA with IPv6

    We currently use garp_master_repeat and garp_master_refresh
    to solve bug 1453855. We need to spawn keepalived only after
    all of the qr/qg ports have been wired so that the
    initial GARP will be properly sent. Otherwise you get a routing
    black hole. In lieu of a proper sync method, we used those two keepalived
    options to send GARPs repeatedly:

    a) We did not know it never stops spamming the network
    b) It causes VMs to lose their IPv6 default gateway due to a keepalived
       bug, which has since been fixed, but it would need to be backported
       to every keepalived version on every distro. Here's the patch:
       https://github.com/acassen/keepalived/pull/200

    The solution this patch proposes is to drop the repeat and refresh
    keepalived options. This will fix the IPv6 bug but re-introduce bug
    1453855. So, this patch uses the delay option instead. It turns
    out keepalived sends a GARP when it transitions to MASTER, and then
    it waits a number of seconds determined by the delay option, and
    sends a GARP again. We'll use an aggressive 'delay' setting to make
    sure that when the node boots and the L3/L2 agents start, we'll
    give the L2 agent enough time to wire the ports as a stopgap solution.
    Note that this only affects initial synchronization time, not failover
    times. Failover times will continue to be fast because the ports
    are wired ahead of time, the initial GARP after the state transition
    to MASTER will be sent properly.

    Change-Id: I7a086472b8742828dae08ffd915c45e94fb4b94e
    Closes-Bug: #1520517
    Related-Bug: #1453855

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270652
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c12bf81e97fcf21ce913339e581e76f657a8b4aa
Submitter: Jenkins
Branch: stable/liberty

commit c12bf81e97fcf21ce913339e581e76f657a8b4aa
Author: Assaf Muller <email address hidden>
Date: Fri Jan 15 16:47:39 2016 -0500

    Fix L3 HA with IPv6

    We currently use garp_master_repeat and garp_master_refresh
    to solve bug 1453855. We need to spawn keepalived only after
    all of the qr/qg ports have been wired so that the
    initial GARP will be properly sent. Otherwise you get a routing
    black hole. In lieu of a proper sync method, we used those two keepalived
    options to send GARPs repeatedly:

    a) We did not know it never stops spamming the network
    b) It causes VMs to lose their IPv6 default gateway due to a keepalived
       bug, which has since been fixed, but it would need to be backported
       to every keepalived version on every distro. Here's the patch:
       https://github.com/acassen/keepalived/pull/200

    The solution this patch proposes is to drop the repeat and refresh
    keepalived options. This will fix the IPv6 bug but re-introduce bug
    1453855. So, this patch uses the delay option instead. It turns
    out keepalived sends a GARP when it transitions to MASTER, and then
    it waits a number of seconds determined by the delay option, and
    sends a GARP again. We'll use an aggressive 'delay' setting to make
    sure that when the node boots and the L3/L2 agents start, we'll
    give the L2 agent enough time to wire the ports as a stopgap solution.
    Note that this only affects initial synchronization time, not failover
    times. Failover times will continue to be fast because the ports
    are wired ahead of time, the initial GARP after the state transition
    to MASTER will be sent properly.

    Conflicts:
     neutron/tests/functional/agent/test_l3_agent.py

    Change-Id: I7a086472b8742828dae08ffd915c45e94fb4b94e
    Closes-Bug: #1520517
    Related-Bug: #1453855
    (cherry picked from commit 303cbc6b5b745aadf026627c4cf2e90e3862b586)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 7.0.3 release.

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/neutron 8.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270653
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=25ba9bd5873864e018f732a55d17297bcd9bd559
Submitter: Jenkins
Branch: stable/kilo

commit 25ba9bd5873864e018f732a55d17297bcd9bd559
Author: Assaf Muller <email address hidden>
Date: Fri Jan 15 16:47:39 2016 -0500

    Fix L3 HA with IPv6

    We currently use garp_master_repeat and garp_master_refresh
    to solve bug 1453855. We need to spawn keepalived only after
    all of the qr/qg ports have been wired so that the
    initial GARP will be properly sent. Otherwise you get a routing
    black hole. In lieu of a proper sync method, we used those two keepalived
    options to send GARPs repeatedly:

    a) We did not know it never stops spamming the network
    b) It causes VMs to lose their IPv6 default gateway due to a keepalived
       bug, which has since been fixed, but it would need to be backported
       to every keepalived version on every distro. Here's the patch:
       https://github.com/acassen/keepalived/pull/200

    The solution this patch proposes is to drop the repeat and refresh
    keepalived options. This will fix the IPv6 bug but re-introduce bug
    1453855. So, this patch uses the delay option instead. It turns
    out keepalived sends a GARP when it transitions to MASTER, and then
    it waits a number of seconds determined by the delay option, and
    sends a GARP again. We'll use an aggressive 'delay' setting to make
    sure that when the node boots and the L3/L2 agents start, we'll
    give the L2 agent enough time to wire the ports as a stopgap solution.
    Note that this only affects initial synchronization time, not failover
    times. Failover times will continue to be fast because the ports
    are wired ahead of time, the initial GARP after the state transition
    to MASTER will be sent properly.

    Conflicts:
     neutron/tests/functional/agent/test_l3_agent.py

    Change-Id: I7a086472b8742828dae08ffd915c45e94fb4b94e
    Closes-Bug: #1520517
    Related-Bug: #1453855
    (cherry picked from commit 303cbc6b5b745aadf026627c4cf2e90e3862b586)

[Dr. Jens Harbott (j-harbott)]

With this patch in (python-neutron=2:8.0.0~b3-0ubuntu1 on Ubuntu Xenial), I'm still seeing issues:

After doing a restart of the L3 agent hosting the active router, failover works fine within some seconds. But 60 seconds later traffic is interrupted for as much as 30-40 seconds. Looking at tcpdump it seems like the restarted keepalived (which will stay in backup state) still sends out another set of gratitous neigbor advertisements at that time.

[Assaf Muller (amuller)]

> After doing a restart of the L3 agent hosting the active router ...

Do you mean restart the node, or restart the L3 agent process?

Also, what keepalived version are you running?

[Dr. Jens Harbott (j-harbott)]

Just did some more test, the behaviour is the same whether I restart the agent, just stop it, or restart the whole node, so the bad behaviour in fact seems to come from the other agent that gets promoted from backup to master.

keepalived is version 1:1.2.19-1, it seems like there hasn't been a release yet with the workaround of setting the router flag for NAs.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 2015.1.4 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 2015.1.4 release.