Update permanent ARP entries for allowed_address_pair IPs in DVR Routers

Bug #1774459 reported by Swaminathan Vasudevan on 2018-05-31
36
This bug affects 4 people
Affects Status Importance Assigned to Milestone
neutron
High
Swaminathan Vasudevan

Bug Description

We have a long term issue with Allowed_address_pairs IP which associated with unbound ports and DVR routers.
The ARP entry for the allowed_address_pair IP does not change based on the GARP issued by any keepalived instance.

Since DVR does the ARP table update through the control plane, and does not allow any ARP entry to get out of the node to prevent the router IP/MAC from polluting the network, there has been always an issue with this.

A recent patch in master https://review.openstack.org/#/c/550676/ to address this issue was not successful.

This patch helped in updating the ARP entry dynamically from the GARP message. But the entry has to be Temporary(NUD - reachable). Only if it is set to 'reachable' we were able to update it on the fly from the GARP message, without using any external tools.

But the problem here is, when we have VMs residing in two different subnets (Subnet A and Subnet B) and if a VM from the Subnet B which is on a different isolated node and is trying to ping the VRRP IP in the Subnet A, the packet from the VM comes to the router namespace where the ARP entry for the VRRP IP is available as reachable. While it is reachable the VM is able to send couple of pings, and later within in 15 sec, the pings timeout.

The reason is that the Router is in turn trying to make sure that if the IP/MAC combination for the VRRP IP is still valid or not, since the entry in the ARP table is "REACHABLE" and not "PERMANENT".
When it tries to re-ARP for the IP, the ARP entries are blocked by the DVR flow rules in the br-tun and so the ARP timesout and the ARP entry in the Router Namespace becomes incomplete.

Option A:
So the way to address this situation is to make use of some GARP sniffer tool/utility that would be running in the router namespace to sniff a GARP packet with a specific IP as a filter. If that IP is seen in the GARP message, the tool/utility should in-turn try to reset the ARP entry for the VRRP IP as permanent. ( This is one option ). This is very performance intensive and so not sure if it would be helpful. So we should probably make it configurable, so that people can use it if required.

Option B:
The other option is, instead of running it on all nodes and in all router-namespace, we can probably just run it on the network_node router_namespace, or in the network node host, and then send a message to the neutron that there was a change in IP/MAC somehow and then neutron will then communicate to all the hosts to do an ARP update for the given IP/MAC. ( Just an idea not sure how simple it is when compared to the former)

Any ideas or thoughts would be helpful.

Boden R (boden) on 2018-06-01
tags: added: rfe
Miguel Lavalle (minsel) on 2018-06-08
tags: added: rfe-triaged
removed: rfe
Changed in neutron:
importance: Undecided → Wishlist
Brian Haley (brian-haley) wrote :

I'm not a big fan of a process running that is snooping on traffic, it's most likely going to cause a performance issue.

Can doing this like the keepalived_state_change code work? It uses "ip monitor" to watch for events and triggers action, and could be modified to look for "neigh" events.

Hi Brian, yes I can take a look at the keepalived_state_change code and see how it works and how it can be used in our case.
But not sure if it can be as such used in our case, since the keepalived in our case is running in side the VM and not in the Namespace.

Changed in neutron:
status: New → Confirmed
Miguel Lavalle (minsel) on 2018-06-14
Changed in neutron:
importance: Wishlist → Critical
importance: Critical → High
tags: removed: rfe-triaged
Miguel Lavalle (minsel) on 2018-06-21
summary: - RFE: Update permanent ARP entries for allowed_address_pair IPs in DVR
- Routers
+ Update permanent ARP entries for allowed_address_pair IPs in DVR Routers

I don't think that we can use the IP-Monitor for our purpose.
We should definitely come up with a GARP-sniff tool similar to IP-Monitor and then use if for our purpose.
If we think that the performance will be an issue. We should probably come up with a dedicated node, that is doing the sniff and reporting it back to Neutron Server.
That way neutron server can then do an rpc update to all agents to add a permanent entry.

Adolfo Duarte (adolfo-duarte) wrote :

Is there any other way to collect the same information that would be collected by garp-sniffing?
What info needs to be propagated?

Also what about putting a flow rule into the openflow tables that route certain packets to all other compute nodes?

For example, a rule could be added to forward GARP packets to all other "members" of dvr group?

And then you can hang a smarter process off each router namespace and process them however is required. Or make it part of the dvr code.

Adolfo Duarte (adolfo-duarte) wrote :

Also on the performance aspect of sniffing garp packets, the reason a system would suffer performance is if it is processing ALL packets, what if we can filter to only garp packets *before* the monitoring tool gets them.
Its basically the same thing as what a host must do. All hosts have to listen to garp packets, so I am not sure you would get any more traffic than what you would get already.
its just instead of dropping the packets on the floor, you pull them into user space for processing.
Perhaps the term "sniffing" is what we need to avoid.
Again we could put a flow rule that pulls specifically garp packets from an ip address/mac combo and give those packets to a process in user space.

One other simple solution would be to forward a packet ( GARP ) for the MAC's that are configured for Allowed_address_pair to the Ryu controller. The controller can process the packet under the 'packet_in_handler' and then try to create an ARP response entry in the ARP Responder for the MAC and IP.

Before writing the entry probably we should check for the current flows configured in the ARP Responder table and if there is a duplicate entry for the IP/MAC combination, then delete it and rewrite the flows for the GARP packet.

Today we don't have a packet_in_handler for the Ryu controller app.
Similar to this.
@handler.set_ev_cls(ofp_event.EventOFPPacketIn, handler.MAIN_DISPATCHER)
def packet_in_handler(self, ev):
    pkt = packet.Packet(array.array('B', ev.msg.data))
    for p in pkt:
        print p.protocol_name, p

My knowledg is pretty limited in L2 openflow, so if there are any L2 Openflow experts could comment on this, if this would work or not.

Rossella Sblendido (rossella-o) wrote :

Swami I am not sure this last proposal is better in terms of performance than sniffing garp. I don't have all the details and I am new to this problem but have you ever tried instead of setting the NUD to reachable to change the timeout so that arp entries become stale pretty quickly and the GARP can update them? A combination between frequent GARP and entries that get stale quickly might fix this.

Rossella thanks for your feedback. The issue is we are seeing the garp updates the arp entry.But when there is ping to that IP, the router tries to re-arp to confirm the IP and that is were it fails.

Not sure if we can do something similar to ARP Responder to add a dynamic ARP Responder rule to send an ARP reply for the GARP'd MAC.

Adding a rule similar to ARP Responder may not be possible. So the best bet is to forward the GARP packets to an output port (tap port created for this purpose). Then a separate process can listen on the tap port and then parse the packet and provide info to the Network node ( neutron-server) to update the ARP entry on all nodes.

Please provide me your thoughts on this.

If we feel that this process running in the compute node will reduce the performance, then what we should do is probably have a new agent type running this process and then communicating to the Network node. That way we don't need to run this process on all compute nodes.

Adding a rule dynamically to the ARP responder is only possible if we can forward the GARP packet to the openflow controller where the openflow controller can process the IN_PACKETS and then make a decision on creating a flow in the ARP responder table.
Is this possible in neutron openflow native drivers today. Any l2 openflow experts can comment on it.
This would be the simplest approach otherwise, we need to forward it to the user space where we process the packet and then send it to Neutron server to take necessary action.

After talking with Miguel Ajo and Daniel Alvarez, plan is to intecept GARP packets and forward to local controller for processing
Swami had a WIP patch that inserted an OF rule to intercept, would need update to send to controller
https://review.openstack.org/#/c/601336/
For ryu look here:
 https://ryu.readthedocs.io/en/latest/ofproto_v1_0_ref.html#packet-in-message
https://ryu.readthedocs.io/en/latest/writing_ryu_app.html?highlight=packet%20in
ovn-controller handles incoming packets (packet-in) as controller here:
https://github.com/openvswitch/ovs/blob/769e6223daf3d6e51963dc3ee938a01fdc71a0d0/ovn/controller/pinctrl.c#L1221
https://github.com/openvswitch/ovs/blob/769e6223daf3d6e51963dc3ee938a01fdc71a0d0/ovn/controller/pinctrl.c#L1121
ODL is tracking this on https://jira.opendaylight.org/browse/NETVIRT-1402
Basically flows will be programmed dynamically when a GARP is recognized by the controller.

Miguel Lavalle (minsel) on 2018-10-04
Changed in neutron:
assignee: nobody → Swaminathan Vasudevan (swaminathan-vasudevan)

Related fix proposed to branch: master
Review: https://review.openstack.org/616272

Related fix proposed to branch: master
Review: https://review.openstack.org/628027

Related fix proposed to branch: master
Review: https://review.openstack.org/651905

Change abandoned by Swaminathan Vasudevan (<email address hidden>) on branch: master
Review: https://review.openstack.org/628027
Reason: Abandoning this patch since I have a better one to address this issue.
https://review.openstack.org/#/c/651905/

Reviewed: https://review.opendev.org/651905
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=52b537ca22b2d7d81a84b2f75de577d8dffee94c
Submitter: Zuul
Branch: master

commit 52b537ca22b2d7d81a84b2f75de577d8dffee94c
Author: Swaminathan Vasudevan <email address hidden>
Date: Thu Apr 11 11:12:24 2019 -0700

    DVR: Modify DVR flows to allow ARP requests to hit ARP Responder table

    DVR does the ARP table update through the control plane, and does not
    allow any ARP requests to get out of the node.

    In order to address the allowed address pair VRRP IP issue with DVR,
    we need to add an ARP entry into the ARP Responder table for the
    allowed address pair IP ( which is taken care by the patch in [1])

    This patch adds a rule in the br-int to redirect the packet
    destinated to the router to the actual router-port and also moves
    the arp filtering rule to the tunnel or the physical port based on the
    configuration.

    By adding the above rule it allows the ARP requests to reach the
    ARP Responder table and filters the ARP requests before it reaches
    the physical network or the tunnel.

    [1] https://review.opendev.org/#/c/601336/
    Related-Bug: #1774459

    Change-Id: I3905ea56ca0ff35bdd96c818719e6d63a3eb5a72

Changed in neutron:
status: Confirmed → In Progress
tags: added: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers