Instances on same flat/vlan network but different nodes cannot communicate
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
ovn-bgp-agent |
New
|
Undecided
|
Unassigned |
Bug Description
Using vagrant I have set up devstack on two nodes, a controller/compute node and a compute node. Devstack created a shared external provider flat network named "public" with subnet 172.24.4.0/24. By default this network is not associated with any physical interfaces as br-ex is not associated with any physical interfaces.
I use a virtual cumulus linux switch running BGP to connect the two nodes together. Each node has a single interface, e.g. eth1, connected to the leaf and configured with a p2p address. If I start an instance on the public network on each node, ovn-bgp-agent will see the event, add the IP to bgp-nic interface and frr will advertise the IP to the leaf. Connectivity from an external host works as expected- for example from either node I can ssh to the instance that is hosted on the local or remote node.
However, if I log into either instance, the instance itself is unable to communicate with the other instance on remote node in the same network. I believe this is because the "public" provider network devstack creates is a flat network which means that neutron is expecting the underlay to be providing L2 adjacency to the nodes. But this is not the case as we are trying to use ovn-bgp-agent to eliminate the need for L2 adjacency.
What is the supported configuration that will allow instances on the same provider network (not tenant network) to communicate with each other while only having L3 connectivity to the nodes?
Hello,
We cover some cases like this on our downstream environments and they don't fail.
I'd like to understand what is different between our setups.
The tests consists in two VMs connected to the external provider flat network running on two different computes, connected between them with routers, advertising routes via BGP.
I am wondering if the problem in your setup is the existence of any route to the public subnet 172.24.4.0/24 on the computes.
In my case, the public subnet is 172.16.100.0/24. One of the VMs sends a ping to another one. That ping goes through this path when it leaves the source compute:
- tap interface (src and dest MACs correspond with the src and dest VM MAC addresses)
- br-ex (dest MAC modified to br-ex MAC due to br-ex flows)
- enp2s0 nic (src MAC is the enp2s0 MAC, dest MAC is the router's MAC)
That compute doesn't have any route to any IP within 172.16.100.0/24. It uses the default route that it obtains via BGP to send the traffic to the destination compute.
$ ip r
default nhid 34 proto bgp src 172.30.1.3 metric 20
nexthop via 100.64.0.9 dev enp3s0 weight 1
nexthop via 100.65.1.9 dev enp2s0 weight 1
100.64.0.8/30 dev enp3s0 proto kernel scope link src 100.64.0.10
100.65.1.8/30 dev enp2s0 proto kernel scope link src 100.65.1.10
192.168.1.0/24 dev enp1s0 proto kernel scope link src 192.168.1.152
192.168.2.0/24 via 192.168.1.1 dev enp1s0
192.168.3.0/24 via 192.168.1.1 dev enp1s0
192.168.4.0/24 via 192.168.1.1 dev enp1s0
The intermediate routers obtained the route to the destination compute via BGP.
>What is the supported configuration that will allow instances on the same provider network (not tenant network) to communicate with each other while only having L3 connectivity to the nodes?
So, answering your question, from the compute's perspective, this is pure L3 routing. The compute doesn't know about the external subnet at all.