DVR router ARP traffic broken for networks containing multiple subnets
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Fix Released
|
Medium
|
LIU Yulong |
Bug Description
Hi,
I am running openstack ussuri with ovs and DVR routers.
When there are multiple subnets in one network, Neutron does not consider the possibility that the subnets could be connected to different routers. This is a problem when a DVR router is expecting to receive an ARP reply. In OVS br-int, table 3 contains only one rule per network which applies to traffic destined to the DVR MAC. This rule translates the DVR MAC to the MAC of the newest router on the network but does not take into consideration that a network could have multiple subnets connected to different routers.
The use case where I am facing this issue is with manila. Manila defines one network object in the service project but each time a user creates a new "Share Network", the Manila service creates a new subnet within the network. So you can end up with many subnets and routers within a network.
It is a bit confusing so below are more details taking my use case with manila as an example.
Manila has a network called manila_
In manila.conf a CIDR and mask is configured and the subnets that will be created by manila service are configured within the CIDR defined and using the mask that is defined.
On a user project I create NetworkX/SubnetX (172.20.20.0/24) and connect it to routerX. I also have instanceX on this network.
Then I create a Share network. This creates a subnet within network manila_
Manila_subnet1 (10.128.16.0/20) is connected to routerX which is already connected to SubnetX (172.20.20.0/24).
A ShareInstance is created on the manila_subnet1 and has IP 10.128.19.189.
It is important that the ShareInstance and InstanceX be located on different computes.
Now communication between InstanceX and the ShareInstance should work but it does not. Here's why.
InstanceX wants to communicate to the ShareInstance so it sends a packet to it's gateway RouterX.
RouterX needs to route the packet to the ShareInstance but it does not have the MAC address in its ARP table.
Router X sends an ARP request -> ARP, Request who-has 10.128.19.189 tell 10.128.16.1, length 28
RouterX never receives an ARP reply.
I followed the flows in br-int and br-tun.
Since traffic is coming from a DVR router, OVS br-tun changes the router's source MAC to the computes's DVR MAC. fa:16:3e:80:4c:3a is the MAC of the router with IP 10.128.16.1
cookie=
Then the packet reaches the ARP responder table 21. There is an entry in table 21 for the ShareInstance MAC so it modifies the packet and sends it back to br-int.
cookie=
But remember that the router source MAC became the DVR MAC and, because of table 21, it is now the destination MAC.
This br-int table 1 rules sends us to table 3 because of the destination MAC being the DVR MAC.
cookie=
In table 3 there is a rule that changes the destination MAC from the DVR MAC to the router MAC based on the VLAN (network). In our case vlan 31.
cookie=
You can see that the mod_dl_dst MAC (fa:16:3e:4d:d0:f9) is not the original source MAC of my router (fa:16:
Why?
Because there are multiple subnets in the network manila_
fa:16:3e:4d:d0:f9 belongs to a router connected to Manila_subnet2 (10.128.48.0/20) which is within manila_
This means the ARP reply is sent to the wrong router.
All the subnets in manila_
In my use case there are 6 subnets in manila_
You dont need to use manila to recreate the problem.
You need networkA with subnetA and network1 with subnet1, subnet2, subnet3, etc...
Connect subnetA and subnet1 to the same router and create a couple instances on subnetA and subnet1 (they need to be on different computes).
Then connect subnet2 to a new router and subnet3 to another new router.
You should see that the instances on subnetA and subnet1 wont be able to communicate as the traffic will stop on the router.
Table 3 is a fairly new addition to neutron/ovs so I believe that the possibility of having multiple subnets in one network was not considered when writing the code.
I think it is related to this commit
https:/
Though it is rare to have multiple subnets in one network, the manila service uses this architecture, making it important that it works correctly.
Changed in neutron: | |
assignee: | nobody → LIU Yulong (dragon889) |
Changed in neutron: | |
status: | Confirmed → Fix Released |
Cloud not reproduce this on Rocky deployment, maybe there is an aggression on the ussuri.
Here is my test toplogy:
Router1 is connect to subnet-1(from network-1), Router-1 is also connected to subnet-2 (from network-2).
VM-1 from network-1 on host-1 can ping VM-2 from network-2 on host-2.
This should be same to your case "instanceX from NetworkX/SubnetX " and "ShareInstance from manila_ service_ network/ Manila_ subnet1" , right?