Connection between two virtual routers does not work with DVR

[Brian Haley]

In DVR environment, connection over two virtual routers is not reachable if VMs are on different computes but works if they are on the same compute. Here is a reproducer:

    $ openstack network create AP1Site
    $ openstack network create AP2Site
    $ openstack network create Wan
    $ openstack subnet create --gateway=10.10.210.254 --network AP1Site --subnet-range 10.10.210.0/24 AP1_sub
    $ openstack subnet create --gateway=10.10.220.254 --network AP2Site --subnet-range 10.10.220.0/24 AP2_sub
    $ openstack subnet create --gateway=10.10.200.254 --network Wan --subnet-range 10.10.200.0/24 Wan_sub
    $ openstack router create AP2-RT
    $ openstack router create AP1-RT
    $ openstack port create --network AP1Site --fixed-ip subnet=AP1_sub,ip-address=10.10.210.10 AP1Port-AP1Internal
    $ openstack port create --network AP2Site --fixed-ip subnet=AP2_sub,ip-address=10.10.220.10 AP2Port-AP2Internal
    $ openstack port create --network Wan --fixed-ip subnet=Wan_sub,ip-address=10.10.200.10 AP1-WANPort
    $ openstack port create --network Wan --fixed-ip subnet=Wan_sub,ip-address=10.10.200.20 AP2-WANPort
    $ nova boot --flavor <flavor> --image <image name> --nic port-id=<AP1Port-AP1Internal port id> AP1
    $ nova boot --flavor <flavor> --image <image name> --nic port-id=<AP2Port-AP2Internal port id> AP2
    $ openstack router add port AP2-RT AP2-WANPort
    $ openstack router add port AP1-RT AP1-WANPort
    $ openstack router add subnet AP2-RT AP2_sub
    $ openstack router add subnet AP1-RT AP1_sub
    $ openstack router set --route destination=10.10.220.0/24,gateway=10.10.200.20 AP1-RT
    $ openstack router set --route destination=10.10.210.0/24,gateway=10.10.200.10 AP2-RT

  Set default routing on AP1.
    $ ip route add default via 10.10.210.254 dev eth0

VMs can't ping each other and can't ping the gateways either.

This is not using a shared network as another related bug mentions.

I want to discuss this at the l3 team meeting to make sure this isn't just a limitation in a DVR environment, but I would think in the case where the two VMs on different subnets were on different computes, the traffic should be going via the centralized router, but would like to confirm with others.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Even if two VMs resides on two different compute nodes belonging to two different subnets, the traffic should be going through the distributed router and not through the centralized router.

[LIU Yulong (dragon889)]

I tested this in stable/queens with dvr only L3 (no ha).
The testing VMs can ping each other and the gateway. But I can see some packet loss in the VM, but such loss does not show very frequently.
Something more: In HOST-1 dvr local namespace I can't ping HOST-2 vm's IP, vice versa.
Such behavior seems like the bug.
some testing:
==========HOST -1===============
[root@10e129e168e34 ~]# ip netns exec qrouter-bd6fa891-eaa2-4c0b-abfb-67cfafe3df1a ping -c 3 10.10.210.10
PING 10.10.210.10 (10.10.210.10) 56(84) bytes of data.

--- 10.10.210.10 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2000ms

[root@10e129e168e34 ~]# ip netns exec qrouter-39dc3eff-d074-470f-afd2-8fa66098fe42 ping -c 3 10.10.210.10
PING 10.10.210.10 (10.10.210.10) 56(84) bytes of data.

--- 10.10.210.10 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms

[root@10e129e168e34 ~]# ip netns exec qrouter-39dc3eff-d074-470f-afd2-8fa66098fe42 ping -c 3 10.10.220.10
PING 10.10.220.10 (10.10.220.10) 56(84) bytes of data.
64 bytes from 10.10.220.10: icmp_seq=1 ttl=63 time=0.446 ms
64 bytes from 10.10.220.10: icmp_seq=2 ttl=63 time=0.389 ms
64 bytes from 10.10.220.10: icmp_seq=3 ttl=63 time=0.255 ms

--- 10.10.220.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.255/0.363/0.446/0.081 ms
[root@10e129e168e34 ~]# ip netns exec qrouter-bd6fa891-eaa2-4c0b-abfb-67cfafe3df1a ping -c 3 10.10.220.10
PING 10.10.220.10 (10.10.220.10) 56(84) bytes of data.
64 bytes from 10.10.220.10: icmp_seq=1 ttl=64 time=0.238 ms
64 bytes from 10.10.220.10: icmp_seq=2 ttl=64 time=0.237 ms
64 bytes from 10.10.220.10: icmp_seq=3 ttl=64 time=0.181 ms

--- 10.10.220.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.181/0.218/0.238/0.031 ms
=================================================

==============HOST 2=============
[root@10e129e168e36 ~]# ip netns exec qrouter-bd6fa891-eaa2-4c0b-abfb-67cfafe3df1a ping -c 3 10.10.210.10
PING 10.10.210.10 (10.10.210.10) 56(84) bytes of data.
64 bytes from 10.10.210.10: icmp_seq=1 ttl=63 time=0.482 ms
64 bytes from 10.10.210.10: icmp_seq=2 ttl=63 time=0.419 ms
64 bytes from 10.10.210.10: icmp_seq=3 ttl=63 time=0.280 ms

--- 10.10.210.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.280/0.393/0.482/0.087 ms
[root@10e129e168e36 ~]# ip netns exec qrouter-39dc3eff-d074-470f-afd2-8fa66098fe42 ping -c 3 10.10.210.10
PING 10.10.210.10 (10.10.210.10) 56(84) bytes of data.
64 bytes from 10.10.210.10: icmp_seq=1 ttl=64 time=0.178 ms
64 bytes from 10.10.210.10: icmp_seq=2 ttl=64 time=0.196 ms
64 bytes from 10.10.210.10: icmp_seq=3 ttl=64 time=0.213 ms

--- 10.10.210.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 0.178/0.195/0.213/0.021 ms
[root@10e129e168e36 ~]# ip netns exec qrouter-39dc3eff-d074-470f-afd2-8fa66098fe42 ping -c 3 10.10.220.10
PING 10.10.220.10 (10.10.220.10) 56(84) bytes of data.

--- 10.10.220.10 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, ...

[Brian Haley (brian-haley)]

Swami - so it's two different networks each with their own subnet, and each neutron router is only connected to one tenant network.

So when each is on the same compute, the traffic would go to the local router, get forwarded onto the integration bridge, and into the other local router (at least that seems the logical flow.

When each is on a separate compute, the traffic would get to the local router, then get forwarded onto the integration bridge, but then what is done with it? I'd really need to see a running (failed) config to verify, but mine just isn't running at the moment.

But I'm still thinking it should work.

[Slawek Kaplonski (slaweq)]

I was looking at this issue a bit. Here is what I found so far.

This can be reproduced by creating only 2 dvr routers and connects them to same network (like WAN in this example). In such case if centralized routers are hosted on same L3 agent then traffic between them is going fine. If both are on different L3 agents, then traffic is not going properly.

So basically problem in described scenario is strictly in connection between two routers and it happens if dvr routers don't have "qrouter-XXXX" namespaces on same host.

In such case traffic is going out from hostA and in br-tun dl_src for packets is change d to dvr_mac_address:

│ cookie=0x9524e9e67b5544ee, duration=8540.374s, table=1, n_packets=5466, n_bytes=229572, idle_age=375, priority=1,dl_vlan=2,dl_src=fa:16:3e:ec:e9:66 actions=mod_dl_src:fa:16:3f:4d:f8:98,resubmit(,2)

Such packet is then sent with tunnel (or vlan) to other hosts. There it comes through br-tun bridge properly and is dropped in br-int bridge by rule:

[root@controller-2 heat-admin]# ovs-ofctl dump-flows br-int | grep fa:16:3f:4d:f8:98
 cookie=0xbba8f62152b34c4c, duration=42321.535s, table=0, n_packets=0, n_bytes=0, idle_age=42547, priority=4,in_port=1,dl_src=fa:16:3f:4d:f8:98 actions=resubmit(,2)
 cookie=0xbba8f62152b34c4c, duration=42321.521s, table=0, n_packets=0, n_bytes=0, idle_age=42547, priority=4,in_port=2,dl_src=fa:16:3f:4d:f8:98 actions=resubmit(,2)
 cookie=0xbba8f62152b34c4c, duration=42321.507s, table=0, n_packets=5468, n_bytes=229768, idle_age=489, priority=2,in_port=3,dl_src=fa:16:3f:4d:f8:98 actions=resubmit(,1)

If both routers have got some ports on same host, qrouter-XXX namespace for both exists on same host and traffic between them is going only via br-int. So there is no this change dl_src address to dvr_mac_address and packets are not dropped in br-int.

I tested it in old Newton branch but it looks that this code which created those OF rules wasn't changed since then so same issue is probably valid also for master branch.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

I also looked into this issue.
When there are two routers on the connected to two different subnets belonging to the same network, the communication between the routers may not work.
You need a third subnet to connect between the routers that is common to both routers.
Then in that case you can have static routes mapping to each subnet through the common subnet and that would work.
So this behavior is as per design and has not changed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/597567

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Ok, there are two reasons why the ping is not working between the VMs when the routers are inter connected.

1. First we provide a static route in the router namespace for a remote NextHop IP address.( In this case 10.10.200.20).
2. The router-namespace does not have a prepopulated ARP entry for that Router Interface. It tries to ARP, but since DVR openflow rules in tunnel bridge filters the ARP we may not receive the ARP reply.
3. If we manually add the ARP entry in the two router namespaces, since the ping does not go through because, the packet actually reaches the destination node 'br-tun', but it gets dropped since, there is no corresponding table=4 rule to process the packets that are coming in from br-tun with the 'vxlan' tunnel-id corresponding the remote ip.

4. Also the return traffic rule in br-tun should be also configured for the remote vxlan tunnel.

So in order for this to work the ARP entries should be updated and the vxlan tun related Input flow rule and output flow rule should be modified.

[Swaminathan Vasudevan (swaminathan-vasudevan)]

Output from Compute Node 1:
stack@ubuntu-18-compute1:~/devstack$ sudo ovs-ofctl dump-ports-desc br-tun
OFPST_PORT_DESC reply (xid=0x2):
 1(patch-int): addr:7a:cc:15:42:34:a7
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
 2(vxlan-c0a87a57): addr:12:9f:b1:e2:db:40
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
 3(vxlan-c0a87a10): addr:36:82:ba:64:2b:31
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-tun): addr:de:d4:5a:38:b7:45
     config: PORT_DOWN
     state: LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max
stack@ubuntu-18-compute1:~/devstack$ sudo ovs-ofctl dump-flows br-tun | grep table=4
 cookie=0x7d728e79bf46779d, duration=1368.634s, table=4, n_packets=2, n_bytes=84, idle_age=1353, priority=1,tun_id=0x3f9 actions=mod_vlan_vid:1,resubmit(,9)
 cookie=0x7d728e79bf46779d, duration=1368.405s, table=4, n_packets=4, n_bytes=843, idle_age=1343, priority=1,tun_id=0x3e9 actions=mod_vlan_vid:2,resubmit(,9)
 cookie=0x7d728e79bf46779d, duration=1408.377s, table=4, n_packets=6, n_bytes=964, idle_age=1337, priority=0 actions=drop
stack@ubuntu-18-compute1:~/devstack$
stack@ubuntu-18-compute1:~/devstack$
stack@ubuntu-18-compute1:~/devstack$ sudo ovs-ofctl dump-flows br-tun | grep table=22
 cookie=0x7d728e79bf46779d, duration=1377.570s, table=22, n_packets=6, n_bytes=964, idle_age=1360, priority=1,dl_vlan=2 actions=strip_vlan,load:0x3e9->NXM_NX_TUN_ID[],output:2,output:3
 cookie=0x7d728e79bf46779d, duration=1363.900s, table=22, n_packets=4, n_bytes=168, idle_age=1379, priority=1,dl_vlan=1 actions=strip_vlan,load:0x3f9->NXM_NX_TUN_ID[],output:2,output:3
 cookie=0x7d728e79bf46779d, duration=1422.023s, table=22, n_packets=18, n_bytes=1660, idle_age=1381, priority=0 actions=drop
stack@ubuntu-18-compute1:~/devstack$

Output from Compute Node 2:
stack@ubuntu-18-compute2:~/devstack$ sudo ovs-ofctl dump-ports-desc br-tun
OFPST_PORT_DESC reply (xid=0x2):
 1(patch-int): addr:aa:ff:2f:8b:27:3d
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
 2(vxlan-c0a87a57): addr:06:4c:c9:43:62:9d
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
 3(vxlan-c0a87af5): addr:aa:12:78:97:df:7f
     config: 0
     state: 0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(br-tun): addr:4e:44:55:48:1a:41
     config: PORT_DOWN
     state: LINK_DOWN
     speed: 0 Mbps now, 0 Mbps max

stack@ubuntu-18-compute2:~/devstack$
stack@ubuntu-18-compute2:~/devstack$ sudo ovs-ofctl dump-flows br-tun | grep table=4
 cookie=0xbdf7cd73c70221e, duration=1377.554s, table=4, n_packets=0, n_bytes=0, idle_age=1377, priority=1,tun_id=0x3f9 actions=mod_vlan_vid:1,resubmit(,9)
 cookie=0xbdf7cd73c70221e, duration=1377.346s, table=4, n_packets=3, n_bytes=800, idle_age=1363, priority=1,tun_id=0x3ef actions=mod_vlan_vid:2,resubmit(,9)
 cookie=0xbdf7cd73c70221e, duration=1425.496s, table=4, n_packets=2, n_bytes=140, idle_age=1368, priority=0 actions=drop
stack@ubuntu-18-compute2:~/devstack$ sudo ovs-ofctl dump-flows br-tun | grep table=22
 cookie=0xbdf7cd73c70221e, duration=1373.851s, table=22, n_packets=2, n_bytes=84, idle_age=1377, priority=1,dl_vlan=1 action...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/597567
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5018d702415a6f056ff54d95447c5031e7e7d9b9
Submitter: Zuul
Branch: master

commit 5018d702415a6f056ff54d95447c5031e7e7d9b9
Author: Slawek Kaplonski <email address hidden>
Date: Wed Aug 29 14:19:39 2018 +0200

    Fix connection between 2 dvr routers

    In case when 2 dvr routers are connected to each other with
    tenant network, those routers needs to be always deployed
    on same compute nodes.
    So this patch changes dvr routers scheduler that it will create
    dvr router on each host on which there are vms or other dvr routers
    connected to same subnets.

    Co-Authored-By: Swaminathan Vasudevan <email address hidden>

    Closes-Bug: #1786272

    Change-Id: I579c2522f8aed2b4388afacba34d9ffdc26708e3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/616508

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/616514

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/616515

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/598676
Committed: https://git.openstack.org/cgit/openstack/neutron-tempest-plugin/commit/?id=b1a3289fe4b98d1270844aa81f60604e64ae50cf
Submitter: Zuul
Branch: master

commit b1a3289fe4b98d1270844aa81f60604e64ae50cf
Author: Slawek Kaplonski <email address hidden>
Date: Mon Aug 27 07:24:00 2018 +0200

    Add connectivity test via 2 routers and tenant networks

    This patch adds test to check if connectivity through 2 routers
    and internal network between routers works fine.

    When at least two compute nodes are available for tempest and
    DifferentHostFilter is enabled in nova-scheduler VM1 and VM2 will be
    spawned on two different compute nodes.

    Connectivity scenario in test is like on picture below:

         +-----+
         | VM 1|
         +--+--+
            |
            | Network 1 (tenant network)
            |
          +-+-+
          | R1|
          +-+-+
            |
            | Network 3 (tenant network)
            |
          +-+-+
          | R2|
          +-+-+
            |
            | Network 2 (tenant network)
            |
         +--+--+
         | VM 2|
         +-----+

    Depends-On: https://review.openstack.org/597567
    Change-Id: Ie9903b7078d6fcff307c5d76770d92df1152d473
    Related-Bug: #1786272

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/616508
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=35033083b988ab49d112d8260380a5a986ce0e85
Submitter: Zuul
Branch: stable/rocky

commit 35033083b988ab49d112d8260380a5a986ce0e85
Author: Slawek Kaplonski <email address hidden>
Date: Wed Aug 29 14:19:39 2018 +0200

    Fix connection between 2 dvr routers

    In case when 2 dvr routers are connected to each other with
    tenant network, those routers needs to be always deployed
    on same compute nodes.
    So this patch changes dvr routers scheduler that it will create
    dvr router on each host on which there are vms or other dvr routers
    connected to same subnets.

    Co-Authored-By: Swaminathan Vasudevan <email address hidden>

    Closes-Bug: #1786272

    Change-Id: I579c2522f8aed2b4388afacba34d9ffdc26708e3
    (cherry picked from commit 5018d702415a6f056ff54d95447c5031e7e7d9b9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/616514
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c801dd8ea1b58c56d36728c4212c2a2b6436a30d
Submitter: Zuul
Branch: stable/queens

commit c801dd8ea1b58c56d36728c4212c2a2b6436a30d
Author: Slawek Kaplonski <email address hidden>
Date: Wed Aug 29 14:19:39 2018 +0200

    Fix connection between 2 dvr routers

    In case when 2 dvr routers are connected to each other with
    tenant network, those routers needs to be always deployed
    on same compute nodes.
    So this patch changes dvr routers scheduler that it will create
    dvr router on each host on which there are vms or other dvr routers
    connected to same subnets.

    Co-Authored-By: Swaminathan Vasudevan <email address hidden>

    Closes-Bug: #1786272

    Conflicts:
        neutron/agent/l3/agent.py
        neutron/db/l3_dvr_db.py
        neutron/tests/unit/agent/l3/test_agent.py

    Change-Id: I579c2522f8aed2b4388afacba34d9ffdc26708e3
    (cherry picked from commit 5018d702415a6f056ff54d95447c5031e7e7d9b9)
    (cherry picked from commit b127433f38d69fee48f89e9fc6b4a14dba8e9352)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/616515
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c08f99c7e072e349632684d565bcac357ce93b6d
Submitter: Zuul
Branch: stable/pike

commit c08f99c7e072e349632684d565bcac357ce93b6d
Author: Slawek Kaplonski <email address hidden>
Date: Wed Aug 29 14:19:39 2018 +0200

    Fix connection between 2 dvr routers

    In case when 2 dvr routers are connected to each other with
    tenant network, those routers needs to be always deployed
    on same compute nodes.
    So this patch changes dvr routers scheduler that it will create
    dvr router on each host on which there are vms or other dvr routers
    connected to same subnets.

    Co-Authored-By: Swaminathan Vasudevan <email address hidden>

    Closes-Bug: #1786272

    Conflicts:
        neutron/agent/l3/agent.py
        neutron/db/l3_dvr_db.py
        neutron/tests/unit/agent/l3/test_agent.py

    Change-Id: I579c2522f8aed2b4388afacba34d9ffdc26708e3
    (cherry picked from commit 5018d702415a6f056ff54d95447c5031e7e7d9b9)
    (cherry picked from commit b127433f38d69fee48f89e9fc6b4a14dba8e9352)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.6 release.