L2pop flows are lost after OVS agent restart

[Arjun Baindur]

In OVS agent, there is a race condition between l2pop's add_fdb_entries notification and provision_local_vlan when we create a vlanmanager mapping. This results in either unicast, flooding, or both entries not being populated on the host. Without the flooding entries, connectivity is lost.

They are lost semi-permanently after this as l2Pop mechanism driver only sends full list of fdb entries after a port_update_up, but only on 1st agent port, or after OVS reboot (where we again hit same race condition, or it partially fixed flows).

Legacy testbed w/ 3 nodes. 4 tenant networks:

1. The add_fdb_entries code path will create the tunnel port(s) in add_fdb_tun, then invoke add_fdb_flow to add the BC/UC l2pop flows and - but only if it can get a Vlanmanager mapping:

    def fdb_add(self, context, fdb_entries):
        LOG.info("2. fdb_add received")
        for lvm, agent_ports in self.get_agent_ports(fdb_entries):
            agent_ports.pop(self.local_ip, None)
            LOG.info("2. fdb_add: agent_ports = %s", agent_ports)
            LOG.info("2. fdb_add: lvm = %s", lvm)
            if len(agent_ports):
                if not self.enable_distributed_routing:
                    with self.tun_br.deferred() as deferred_br:
                        LOG.info("2. fdb_add: about to call fdb_add_tun w/ lvm = %s", lvm)
                        self.fdb_add_tun(context, deferred_br, lvm,
                                         agent_ports, self._tunnel_port_lookup)
                else:
                    self.fdb_add_tun(context, self.tun_br, lvm,
                                     agent_ports, self._tunnel_port_lookup)

    def get_agent_ports(self, fdb_entries, local_vlan_map=None):
        """Generator to yield port info.

        For each known (i.e found in VLAN manager) network in
        fdb_entries, yield (lvm, fdb_entries[network_id]['ports']) pair.

        :param fdb_entries: l2pop fdb entries
        :param local_vlan_map: Deprecated.
        """
        lvm_getter = self._get_lvm_getter(local_vlan_map)
        for network_id, values in fdb_entries.items():
            try:
                lvm = lvm_getter(network_id, local_vlan_map)
            except vlanmanager.MappingNotFound:
                LOG.info("get_agent_ports: vlanmanager.MappingNotFound EXCEPTION! netid = %s, local_vlan_map = %s", network_id, local_vlan_map)
                continue
            agent_ports = values.get('ports')
            LOG.info("get_agent_ports: got lvm= %s", lvm)
            yield (lvm, agent_ports)

2. If the vlan mapping isn't found, the tunnel port creation is skipped, as are flows.

3. When we create VLAN mapping in provision_local_vlan(), the install_flood_to_tun however is skipped if there are currently no tunnel ports created:

    def provision_local_vlan(self, net_uuid, network_type, physical_network,
                             segmentation_id):
...

        if network_type in constants.TUNNEL_NETWORK_TYPES:
            LOG.info("ARJUN: network_type = %s", network_type)
            if self.enable_tunneling:
                # outbound broadcast/multicast
                ofports = list(self.tun_br_ofports[network_type].values())
                LOG.info("ARJUN: provision_local_vlan: ofports = %s enable_tunneling = %s", ofports, self.enable_tunneling)
                if ofports:
                    LOG.info("ARJUN: installing FLOODING_ENTRY: lvid = %s segment_id = %s", lvid, segmentation_id)
                    self.tun_br.install_flood_to_tun(lvid,
                                                     segmentation_id,
                                                     ofports)
                # inbound from tunnels: set lvid in the right table
                # and resubmit to Table LEARN_FROM_TUN for mac learning

4. Finally, the cleanup stale flows logic removes all old flows. At this point br-tun is left with missing flooding and/or unicast flows.

5. If #3 always happens first for all networks, we are good. Otherwise flows are lost:

Unicast only flows missing if (but flood added):

 - Network Vlanmanager mapping is allocated *after* it's add_fdb_entries, but some other network sets up tunnel ports on br-tun

Broadcast AND UC flows missing if:

 - A network tries to add fdb flows before vlanmanager allocated, and no other network has created the tunnel ports/ofports on br-tun yet.

Example with 3 tenant networks:

1. add_fdb_entries for network 1 and 2 - no LVM yet, so flow and tunnel ports not created yet
2. LVM created for network 2, but flood not installed because no ofports
3. LVM created for networks 3
4. add_fdb_entries for network 3, here it properly finds the LVM, and creates tunnel ports/flows
5. LVM created for network 1, tunnel ofports created, so flood installed - but unicast missing

After this point, network 3 would be fine, network 2 would me missing all flows, network 1 would have flood but not unicast.

The ordering seems to vary wildly depending on # of tunnel ports, # of networks, ports per network, how ports are distributed, network speed, etc...

[Arjun Baindur (abaindur)]

Here 4 networks. A VM from each network on both 2 compute nodes. This was OVS agent restart on network node (legacy deployment here). Tunnel 0x75b34 missing unicast flows. Tunnel 0x75b1d missing both.

BEFORE:
[root@123-1449 ~]# ovs-ofctl dump-flows br-tun
NXST_FLOW reply (xid=0x4):
 cookie=0x9f710356180d5375, duration=872.577s, table=0, n_packets=262, n_bytes=27320, idle_age=83, priority=1,in_port=1 actions=resubmit(,2)
 cookie=0x9f710356180d5375, duration=526.306s, table=0, n_packets=231, n_bytes=22502, idle_age=180, priority=1,in_port=7 actions=resubmit(,4)
 cookie=0x9f710356180d5375, duration=422.125s, table=0, n_packets=106, n_bytes=10221, idle_age=140, priority=1,in_port=8 actions=resubmit(,4)
 cookie=0x9f710356180d5375, duration=872.574s, table=0, n_packets=0, n_bytes=0, idle_age=65534, priority=0 actions=drop
 cookie=0x9f710356180d5375, duration=872.573s, table=2, n_packets=195, n_bytes=22750, idle_age=122, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20)
 cookie=0x9f710356180d5375, duration=872.572s, table=2, n_packets=67, n_bytes=4570, idle_age=83, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22)
 cookie=0x9f710356180d5375, duration=872.569s, table=3, n_packets=0, n_bytes=0, idle_age=65534, priority=0 actions=drop
 cookie=0x9f710356180d5375, duration=863.798s, table=4, n_packets=27, n_bytes=4311, idle_age=197, priority=1,tun_id=0x75ae7 actions=mod_vlan_vid:2,resubmit(,10)
 cookie=0x9f710356180d5375, duration=861.078s, table=4, n_packets=202, n_bytes=18087, idle_age=140, priority=1,tun_id=0x75b10 actions=mod_vlan_vid:1,resubmit(,10)
 cookie=0x9f710356180d5375, duration=770.920s, table=4, n_packets=18, n_bytes=2874, idle_age=282, priority=1,tun_id=0x75b1d actions=mod_vlan_vid:4,resubmit(,10)
 cookie=0x9f710356180d5375, duration=756.222s, table=4, n_packets=90, n_bytes=7451, idle_age=346, priority=1,tun_id=0x75b34 actions=mod_vlan_vid:5,resubmit(,10)
 cookie=0x9f710356180d5375, duration=872.568s, table=4, n_packets=0, n_bytes=0, idle_age=65534, priority=0 actions=drop
 cookie=0x9f710356180d5375, duration=872.567s, table=6, n_packets=0, n_bytes=0, idle_age=65534, priority=0 actions=drop
 cookie=0x9f710356180d5375, duration=872.566s, table=10, n_packets=337, n_bytes=32723, idle_age=140, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0x9f710356180d5375,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:NXM_OF_IN_PORT[]),output:1
 cookie=0x9f710356180d5375, duration=526.301s, table=20, n_packets=31, n_bytes=3520, idle_age=458, priority=2,dl_vlan=5,dl_dst=fa:16:3e:5f:25:2e actions=strip_vlan,set_tunnel:0x75b34,output:7
 cookie=0x9f710356180d5375, duration=513.346s, table=20, n_packets=0, n_bytes=0, idle_age=513, priority=2,dl_vlan=4,dl_dst=fa:16:3e:51:92:0d actions=strip_vlan,set_tunnel:0x75b1d,output:7
 cookie=0x9f710356180d5375, duration=422.479s, table=20, n_packets=35, n_bytes=3964, idle_age=346, priority=2,dl_vlan=5,dl_dst=fa:16:3e:39:05:d8 actions=strip_vlan,set_tunnel:0x75b34,output:8
 cookie=0x9f710356180d5375, duration=422.125s, table=20, n_packets=0, n_bytes=0, idle_age=422, prior...

[Trevor McCasland (twm2016)]

Can you provide the configuration you have for ovs agent and l2pop driver? I suspect the agent_boot_time configuration option for the l2pop driver is sensitive to this bug. "Delay within which agent is expected to update existing ports when it restarts"

[Arjun Baindur (abaindur)]

The agent_boot_time is untouched, so default of 180. This is well more than the duration of OVS agent boot to stabilize/all ports to be re-updated.

I think the problem here is, there are not too many overall ports on this system, but there are at least a couple of networks. This node is not a hypervisor - just used for Glance image lib and network node services (l3 agent, dhcp/dnsmasq).

Therefore the only port updates received after OVS restart are for router ports and dhcp ports. We also see this regularly on compute hosts where there are not too many VMs, but more than 1 or 2 networks (4 networks, around 1-2 VMs per network). Some networks get no flows, some get only UC flows as per examples above.

Increasing agent_boot_time to something obscenely large doesn't seem like a proper fix.

[Arjun Baindur (abaindur)]

We have touched minimal config. Basic testbed w/ VXLAN tenant networks only, VLAN external network.

ML2/OVS config on hypervisor/network node:

[ovs]
enable_tunneling = True
ovsdb_interface = vsctl
bridge_mappings = physnet:br-vlan
local_ip = 192.168.172.4
of_interface = ovs-ofctl

[ml2_type_vxlan]

[ml2_type_vlan]

[ml2_type_flat]

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver

[ml2_type_gre]

[ml2]

[agent]
tunnel_types = vxlan
enable_distributed_routing = False
root_helper = sudo /opt/neutron/bin/neutron-rootwrap /opt/etc/neutron/rootwrap.conf
l2_population = True

ML2 config on neutron controller:

[ml2]
type_drivers = flat,vxlan,vlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch,linuxbridge,l2population

extension_drivers = dns

[ml2_type_flat]
flat_networks = *

[ml2_type_vlan]
network_vlan_ranges = external,physnet

[ml2_type_gre]

[ml2_type_vxlan]
vni_ranges = 482000:500000

[securitygroup]

[l2pop]
agent_boot_time = 180

[yangjianfeng (1912294464-o)]

I also encountered the similar problem.
I found a logic error after browse neutron code:

    def notify_l2pop_port_wiring(self, port_id, rpc_context,
                                 status, host):
        ......
        # NOTE: DVR ports are already handled and updated through l2pop
        # and so we don't need to update it again here
        if port['device_owner'] == n_const.DEVICE_OWNER_DVR_INTERFACE:
            return
        ......
But the DVR port did not handle and updated when I restart neutron-openvswitch-agent

    def _update_individual_port_db_status(self, context, port, status, host):
        ......
        if updated:
            self.mechanism_manager.update_port_postcommit(mech_context)
        ......
Beacuse the "updated" value is "false" when restart neutron-openvswitch-agent.So,the "update_port_postcommit" and "update_port_up" will not be called.So In compute node when there is only DVR port on one network the flood flow of the network will not be set.

Before I restart neutron-openvswitch-agent:
 cookie=0x3c125494667e2cef, duration=4931.840s, table=21, n_packets=8895, n_bytes=373590, idle_age=3, priority=0 actions=resubmit(,22)
 cookie=0x3c125494667e2cef, duration=4928.110s, table=22, n_packets=0, n_bytes=0, idle_age=65534, priority=1,dl_vlan=4 actions=strip_vlan,load:0x44->NXM_NX_TUN_ID[],output:6,output:7
 cookie=0x3c125494667e2cef, duration=4926.851s, table=22, n_packets=0, n_bytes=0, idle_age=65534, priority=1,dl_vlan=3 actions=strip_vlan,load:0x35->NXM_NX_TUN_ID[],output:6,output:7
 cookie=0x3c125494667e2cef, duration=6.136s, table=22, n_packets=8, n_bytes=608, idle_age=3, priority=1,dl_vlan=1 actions=strip_vlan,load:0x20->NXM_NX_TUN_ID[],output:6,output:7
 cookie=0x3c125494667e2cef, duration=4931.838s, table=22, n_packets=162, n_bytes=13348, idle_age=7, priority=0 actions=drop

After I restart neutron-openvswitch-agent(The flood flow of "dl_vlan=1" is missing):
 cookie=0xf0733c083d3dceb, duration=14.084s, table=21, n_packets=8895, n_bytes=373590, idle_age=202, priority=0 actions=resubmit(,22)
 cookie=0xf0733c083d3dceb, duration=10.353s, table=22, n_packets=0, n_bytes=0, idle_age=65534, priority=1,dl_vlan=4 actions=strip_vlan,load:0x44->NXM_NX_TUN_ID[],output:6,output:7
 cookie=0xf0733c083d3dceb, duration=9.345s, table=22, n_packets=0, n_bytes=0, idle_age=65534, priority=1,dl_vlan=3 actions=strip_vlan,load:0x35->NXM_NX_TUN_ID[],output:6,output:7
 cookie=0xf0733c083d3dceb, duration=14.082s, table=22, n_packets=162, n_bytes=13348, idle_age=206, priority=0 actions=drop

Here is my environment information:
[root@yjfnode04 ~]# ip netns exec qrouter-2659785b-b619-41c6-be13-f95cdd5e62b0 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
39: qr-33133255-e2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:5c:cb:37 brd ff:ff:ff:ff:ff:ff
    inet 10.10.30.1/24 brd 10.10.30.255 scope global qr-33133255-e2
       valid_lft fo...

[Wenran Xiao (wenran)]

Hi, Arjun Baindur. Has this problem been solved?

[Swaminathan Vasudevan (swaminathan-vasudevan)]

This seems to be a duplicate of #1773286

[Swaminathan Vasudevan (swaminathan-vasudevan)]

See this patch.
https://review.openstack.org/#/c/571920/