networking-ovn

Ping through 2 routers is not working

Bug #1543795 reported by Albert Fishman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
networking-ovn
Incomplete
Undecided
Unassigned

Bug Description

I have tried with the latest code pulled 2/4 and also with the commit at
https://review.openstack.org/#/c/275439/

I am trying to experiment with a topology when a packet may traverse through more than one router.
For debug purposes, since OpenStack does not allow chaining two routers directly, I have simplified the topology (created using OpenStack Horizon) to:
VM1 10.1.0.3 @ Network 1
VM2 10.2.0.4 @ Network 2
VM3 10.2.0.4 @ Network 3
Router 1 connects Network 1 & Network 2.
Router 2 connects Network 2 & Network 3.

I can successfully ping between the VM 1& VM2, but can't ping between VM2 and VM3 and the desirable VM1 & VM3.
Adding static routes did not resolve the problem too, it looks like once a network is connected to 2 routers, the packets will always pass to the gateway port towards one of the routers and will never be sent to the other router port.

Here are some dumps:
Only pinging from VM3 to VM2:
sudo ovs-dpctl dump-flows
recirc_id(0),in_port(7),eth(src=fa:16:3e:6e:72:de),eth_type(0x0800),ipv4(frag=no), packets:143, bytes:14014, used:3.912s, actions:ct(zone=6),recirc(0x7d)
recirc_id(0x7d),in_port(7),ct_state(+new-est-rel-inv+trk),eth(src=00:00:00:00:00:00/01:00:00:00:00:00,dst=fa:16:3e:86:be:88),eth_type(0x0800),ipv4(src=10.3.0.2/255.255.255.254,dst=10.2.0.4,ttl=64,frag=no), packets:143, bytes:14014, used:3.912s, actions:ct(commit,zone=6),ct(commit,zone=6)

Only pinging from VM2 to VM1:
recirc_id(0),in_port(6),eth(src=fa:16:3e:cb:bf:af),eth_type(0x0800),ipv4(frag=no), packets:14, bytes:1372, used:0.564s, actions:ct(zone=5),recirc(0x7e)
recirc_id(0x7e),in_port(6),ct_state(+new-est-rel-inv+trk),eth(src=00:00:00:00:00:00/01:00:00:00:00:00,dst=fa:16:3e:31:15:e2),eth_type(0x0800),ipv4(src=10.2.0.4,dst=10.3.0.2/255.255.255.254,ttl=64,frag=no), packets:14, bytes:1372, used:0.564s, actions:ct(commit,zone=5),ct(commit,zone=5)

Successful ping between VM1 & VM2:
recirc_id(0x81),in_port(6),ct_state(-new+est-rel-inv+trk),eth(src=fa:16:3e:cb:bf:af,dst=fa:16:3e:31:15:e2),eth_type(0x0800),ipv4(src=10.2.0.4,dst=10.1.0.3,ttl=64,frag=no), packets:162, bytes:15876, used:0.188s, actions:set(eth(src=fa:16:3e:47:ac:e9,dst=fa:16:3e:17:99:1d)),set(ipv4(src=10.2.0.4,dst=10.1.0.3,ttl=63)),ct(commit),ct(zone=4),recirc(0x82)
recirc_id(0),in_port(5),eth(src=fa:16:3e:17:99:1d),eth_type(0x0800),ipv4(frag=no), packets:163, bytes:15974, used:0.188s, actions:ct(zone=4),recirc(0x7f)
recirc_id(0),in_port(6),eth(src=fa:16:3e:cb:bf:af),eth_type(0x0800),ipv4(frag=no), packets:163, bytes:15974, used:0.188s, actions:ct(zone=5),recirc(0x81)
recirc_id(0x80),in_port(5),ct_state(-new+est-rel-inv+trk),eth(dst=fa:16:3e:cb:bf:af),eth_type(0x0800),ipv4(frag=no), packets:162, bytes:15876, used:0.188s, actions:6
recirc_id(0x7f),in_port(5),ct_state(-new+est-rel-inv+trk),eth(src=fa:16:3e:17:99:1d,dst=fa:16:3e:47:ac:e9),eth_type(0x0800),ipv4(src=10.1.0.2/255.255.255.254,dst=10.2.0.4,ttl=64,frag=no), packets:162, bytes:15876, used:0.188s, actions:set(eth(src=fa:16:3e:31:15:e2,dst=fa:16:3e:cb:bf:af)),set(ipv4(src=10.1.0.2/255.255.255.254,dst=10.2.0.4,ttl=63)),ct(commit),ct(zone=5),recirc(0x80)
recirc_id(0x82),in_port(6),ct_state(-new+est-rel-inv+trk),eth(dst=fa:16:3e:17:99:1d),eth_type(0x0800),ipv4(frag=no), packets:162, bytes:15876, used:0.188s, actions:5

Some OVN dumps:
sudo ovs-vsctl show
86b17641-06b1-4cde-b82d-2490450c512c
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "patch-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-ee1c2b34-b70d-491c-8db9-f29a7fe35131"
            Interface "patch-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-ee1c2b34-b70d-491c-8db9-f29a7fe35131"
                type: patch
                options: {peer="patch-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131"}
        Port "tap0bad0160-ef"
            Interface "tap0bad0160-ef"
                type: internal
        Port "patch-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-143ea76d-691f-4440-aadf-90e9e0a03fcd"
            Interface "patch-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-143ea76d-691f-4440-aadf-90e9e0a03fcd"
                type: patch
                options: {peer="patch-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd"}
        Port "tap211d732f-8f"
            Interface "tap211d732f-8f"
        Port "tap005a356d-55"
            Interface "tap005a356d-55"
                type: internal
        Port "patch-lrp-be975971-f023-4824-8a9a-f49deac4c48e-to-be975971-f023-4824-8a9a-f49deac4c48e"
            Interface "patch-lrp-be975971-f023-4824-8a9a-f49deac4c48e-to-be975971-f023-4824-8a9a-f49deac4c48e"
                type: patch
                options: {peer="patch-be975971-f023-4824-8a9a-f49deac4c48e-to-lrp-be975971-f023-4824-8a9a-f49deac4c48e"}
        Port "tap494da741-31"
            Interface "tap494da741-31"
                type: internal
        Port "patch-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11"
            Interface "patch-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11"
                type: patch
                options: {peer="patch-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-376d7039-66dc-40e6-9ebb-ca75ba986f11"}
        Port "tapc9d18a59-8b"
            Interface "tapc9d18a59-8b"
        Port "patch-be975971-f023-4824-8a9a-f49deac4c48e-to-lrp-be975971-f023-4824-8a9a-f49deac4c48e"
            Interface "patch-be975971-f023-4824-8a9a-f49deac4c48e-to-lrp-be975971-f023-4824-8a9a-f49deac4c48e"
                type: patch
                options: {peer="patch-lrp-be975971-f023-4824-8a9a-f49deac4c48e-to-be975971-f023-4824-8a9a-f49deac4c48e"}
        Port "patch-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd"
            Interface "patch-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd"
                type: patch
                options: {peer="patch-lrp-143ea76d-691f-4440-aadf-90e9e0a03fcd-to-143ea76d-691f-4440-aadf-90e9e0a03fcd"}
        Port "patch-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-376d7039-66dc-40e6-9ebb-ca75ba986f11"
            Interface "patch-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-376d7039-66dc-40e6-9ebb-ca75ba986f11"
                type: patch
                options: {peer="patch-376d7039-66dc-40e6-9ebb-ca75ba986f11-to-lrp-376d7039-66dc-40e6-9ebb-ca75ba986f11"}
        Port "patch-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131"
            Interface "patch-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131"
                type: patch
                options: {peer="patch-lrp-ee1c2b34-b70d-491c-8db9-f29a7fe35131-to-ee1c2b34-b70d-491c-8db9-f29a7fe35131"}
        Port "tap6dfb3d54-ea"
            Interface "tap6dfb3d54-ea"

Please advice...

Revision history for this message
Albert Fishman (albert-fishman) wrote :
Download full text (48.3 KiB)

sudo ovs-ofctl -O OpenFlow13 dump-flows br-int
OFPST_FLOW reply (OF1.3) (xid=0x2):
 cookie=0x0, duration=52931.929s, table=0, n_packets=18, n_bytes=2314, priority=100,in_port=2 actions=set_field:0x1->reg5,set_field:0x2->metadata,set_field:0x1->reg6,resubmit(,16)
 cookie=0x0, duration=52911.922s, table=0, n_packets=26, n_bytes=3569, priority=100,in_port=3 actions=set_field:0x2->reg5,set_field:0x3->metadata,set_field:0x1->reg6,resubmit(,16)
 cookie=0x0, duration=52871.156s, table=0, n_packets=57163, n_bytes=5607608, priority=100,in_port=6 actions=set_field:0x2->metadata,set_field:0x2->reg6,resubmit(,16)
 cookie=0x0, duration=52871.156s, table=0, n_packets=97551, n_bytes=9558462, priority=100,in_port=5 actions=set_field:0x5->metadata,set_field:0x1->reg6,resubmit(,16)
 cookie=0x0, duration=52859.599s, table=0, n_packets=17248, n_bytes=1689358, priority=100,in_port=7 actions=set_field:0x3->metadata,set_field:0x2->reg6,resubmit(,16)
 cookie=0x0, duration=52859.599s, table=0, n_packets=59056, n_bytes=5793998, priority=100,in_port=8 actions=set_field:0x5->metadata,set_field:0x2->reg6,resubmit(,16)
 cookie=0x0, duration=52753.744s, table=0, n_packets=99051, n_bytes=9622439, priority=100,in_port=13 actions=set_field:0x4->reg5,set_field:0x2->metadata,set_field:0x3->reg6,resubmit(,16)
 cookie=0x0, duration=2581.858s, table=0, n_packets=3200, n_bytes=310219, priority=100,in_port=31 actions=set_field:0x5->reg5,set_field:0x3->metadata,set_field:0x6->reg6,resubmit(,16)
 cookie=0x0, duration=1896.264s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=34 actions=set_field:0x3->metadata,set_field:0x7->reg6,resubmit(,16)
 cookie=0x0, duration=1896.264s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=35 actions=set_field:0x6->metadata,set_field:0x1->reg6,resubmit(,16)
 cookie=0x0, duration=1830.586s, table=0, n_packets=16, n_bytes=1903, priority=100,in_port=36 actions=set_field:0x3->reg5,set_field:0x8->metadata,set_field:0x1->reg6,resubmit(,16)
 cookie=0x0, duration=1695.314s, table=0, n_packets=189, n_bytes=18445, priority=100,in_port=39 actions=set_field:0x6->reg5,set_field:0x8->metadata,set_field:0x3->reg6,resubmit(,16)
 cookie=0x0, duration=869.925s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=41 actions=set_field:0x8->metadata,set_field:0x4->reg6,resubmit(,16)
 cookie=0x0, duration=869.925s, table=0, n_packets=160, n_bytes=15680, priority=100,in_port=42 actions=set_field:0x6->metadata,set_field:0x2->reg6,resubmit(,16)
 cookie=0x0, duration=52932.057s, table=16, n_packets=0, n_bytes=0, priority=100,metadata=0x2,vlan_tci=0x1000/0x1000 actions=drop
 cookie=0x0, duration=52911.961s, table=16, n_packets=0, n_bytes=0, priority=100,metadata=0x3,vlan_tci=0x1000/0x1000 actions=drop
 cookie=0x0, duration=52871.158s, table=16, n_packets=0, n_bytes=0, priority=100,metadata=0x5,vlan_tci=0x1000/0x1000 actions=drop
 cookie=0x0, duration=1896.271s, table=16, n_packets=0, n_bytes=0, priority=100,metadata=0x6,vlan_tci=0x1000/0x1000 actions=drop
 cookie=0x0, duration=1830.720s, table=16, n_packets=0, n_bytes=0, priority=100,metadata=0x8,vlan_tci=0x1000/0x1000 actions=drop
 cookie=0x0, duration=52932.057s, table=16, n_packets=0, n_bytes=0, priority=100,met...

Revision history for this message
Chandra Sekhar Vejendla (csvejend) wrote :

This does not look like a valid configuration. The default gateway of network 2 is connected to router 1. How can network 2 be connected to router 2 when it is already connected to router 1 ?

Russell Bryant (russellb)
Changed in networking-ovn:
status: New → Incomplete
Revision history for this message
Albert Fishman (albert-fishman) wrote :

Thanks Chandra and Russell for your response,
I am trying to create a topology that involves traversing a packet through multiple routers,
imagine a logical network that mimics the physical infrastructure arch of a leaf (switch) and Fabric + Spine (routers),
since I can't chain in OpenStack 2 routers directly, I had to place a network element in between them, thats why there is a network that is connected to 2 different router ports.

I believe my issue is similar to: https://bugs.launchpad.net/networking-ovn/+bug/1545140
and I am ok with adopting the same topology once the issue is resolved.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.