R4.1 build 15 Netronome 47: Gwless forwarding not working

[Ankit Jain]

Seeing the issue on Netronome testbed with acceleration enabled. Without acceleration, the issue is not observed.

Steps to recreate the issue:

1) Create a VN vn1
2) enable ip fabric forwarding in vn1

Subnet(s)
CIDR Gateway Service Address DNS DHCP Allocation Pools
30.238.247.0/24 30.238.247.1 30.238.247.2 Enabled Enabled -

IP Fabric Forwarding Enabled

2) Launch 2 VMs in VN1

Ping from vm1 to vm2 and expect ping to work.

Observations:

1) Ping not working when gwless (ip fabric forwarding ) and netronome acceleration is enabled

^C
--- 30.238.247.5 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
$ ifconfig
eth0 Link encap:Ethernet HWaddr 02:04:F9:A4:C9:3B
          inet addr:30.238.247.6 Bcast:30.238.247.255 Mask:255.255.255.0
          inet6 addr: fe80::4:f9ff:fea4:c93b/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:924 errors:0 dropped:0 overruns:0 frame:0
          TX packets:962 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:87628 (85.5 KiB) TX bytes:92650 (90.4 KiB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

$ ping 30.238.247.5
PING 30.238.247.5 (30.238.247.5): 56 data bytes

2) Ping starts working on disabling ip fabric forwarding ( Netronome acceleration is still enabled)

$ ping 30.238.247.5
PING 30.238.247.5 (30.238.247.5): 56 data bytes
64 bytes from 30.238.247.5: seq=0 ttl=63 time=1.622 ms

64 bytes from 30.238.247.5: seq=1 ttl=63 time=0.248 ms
64 bytes from 30.238.247.5: seq=2 ttl=63 time=0.283 ms
64 bytes from 30.238.247.5: seq=3 ttl=63 time=0.242 ms
64 bytes from 30.238.247.5: seq=4 ttl=63 time=0.243 ms
^C
--- 30.238.247.5 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.242/0.527/1.622 ms
$

3) Ping works on disabling Netronome acceleration ( ip fabric forwarding is still enabled)

  TEST : test_gw_less_fwd_single_vn_ip_fab

            Test Gateway less forwarding with single VN and IP Fabric provider
            network. IP Fbaric network is configured as provider network
            over vn1. Multiple VMs will be launched across compute nodes
            and ping between VMs should be successful and happen through
            underlay. Ping between vhosts to VMs should be successful and happen
            through underlay. Also, ping between VMs to vhosts also should be
            successful and should go through underlay.

2018-10-11 10:08:25,562 - INFO - Created BGP router: hooper
n2018-10-11 10:08:25,959 - INFO - Using existing project [u'default-domain', u'ctest-TestGWLessFWD-86069364'](fa89b6cb-4530-499c-83b4-559b6d9cb7b6)
2018-10-11 10:08:25,962 - INFO - Default SG to be edited for allow all on project: ctest-TestGWLessFWD-86069364
2018-10-11 10:08:26,037 - INFO - Adding rules to the default security group in Project ctest-TestGWLessFWD-86069364
> /contrail-test/common/gw_less_fwd/base.py(333)setup_gw_less_fwd()
-> vn_fixtures = self.setup_vns(vn)
2018-10-11 10:08:28,366 - INFO - Created VN vn1
2018-10-11 10:08:28,656 - INFO - Configuring IP Fabric forwarding on VN: vn1
11 10:08:33,292 - INFO - VMI 6b67ae72-9823-445c-ab4a-2f6f1c469ad9 verification in API Server passed
2018-10-11 10:08:35,759 - INFO - VM ([<Server: ctest-TestGWLessFWD-86069364-61553487>]) created on node: (nodei8), Zone: (nova:nodei8)
2018-10-11 10:08:37,729 - INFO - VMI 04f9a4c9-3bf2-4d62-9b53-732c41f0a5e2 verification in API Server passed
2018-10-11 10:08:41,565 - INFO - VM ([<Server: ctest-TestGWLessFWD-86069364-33711132>]) created on node: (nodel9), Zone: (nova:nodel9)
2018-10-11 10:08:41,567 - INFO - Waiting for VM ctest-TestGWLessFWD-86069364-33711132 to be up..
2018-10-11 10:08:55,864 - INFO - Ping to Metadata IP 169.254.0.3 of VM ctest-TestGWLessFWD-86069364-33711132 passed
2018-10-11 10:09:01,067 - INFO - Waiting for VM ctest-TestGWLessFWD-86069364-61553487 to be up..
2018-10-11 10:09:03,992 - INFO - Ping to Metadata IP 169.254.0.7 of VM ctest-TestGWLessFWD-86069364-61553487 passed
--Return--
> /contrail-test/common/gw_less_fwd/base.py(351)setup_gw_less_fwd()->{'vm_fixtures': {'vm1': <VMFixtu...61553487>, 'vm2': <VMFixtu...33711132>}, 'vmi_fixtures': {'vmi1': <port_fi...f438ae10>, 'vmi2': <port_fi...f43a46d0>}, 'vn_fixtures': {'vn1': <vn_test...f438a6d0>}}
2018-10-11 10:14:23,074 - INFO - Route: 30.238.247.6 is found in default routing instance as expected in agent on compute node: 192.168.11.169
2018-10-11 10:14:23,165 - INFO - Route: 30.238.247.5 is found in default routing instance as expected in agent on compute node: 192.168.11.164
2018-10-11 10:14:23,474 - INFO - Route: 30.238.247.6 is found in defaultrouting instance as expected incontrol node: 192.168.31.4
2018-10-11 10:14:23,724 - INFO - Route: 30.238.247.5 is found in defaultrouting instance as expected incontrol node: 192.168.31.4
[cirros@169.254.0.3] sudo: ping -s 56 -c 2 -W 1 30.238.247.5
[cirros@169.254.0.3] out: PING 30.238.247.5 (30.238.247.5): 56 data bytes
[cirros@169.254.0.3] out:
[cirros@169.254.0.3] out: --- 30.238.247.5 ping statistics ---
[cirros@169.254.0.3] out: 2 packets transmitted, 0 packets received, 100% packet loss
[cirros@169.254.0.3] out:

Warning: sudo() received nonzero return code 1 while executing 'ping -s 56 -c 2 -W 1 30.238.247.5'!

2018-10-11 10:14:28,020 - WARNING - Ping to IP 30.238.247.5 from VM ctest-TestGWLessFWD-86069364-33711132 failed
[cirros@169.254.0.3] sudo: ping -s 56 -c 2 -W 1 30.238.247.5
[cirros@169.254.0.3] out: PING 30.238.247.5 (30.238.247.5): 56 data bytes
[cirros@169.254.0.3] out:
[cirros@169.254.0.3] out: --- 30.238.247.5 ping statistics ---
[cirros@169.254.0.3] out: 2 packets transmitted, 0 packets received, 100% packet loss
[cirros@169.254.0.3] out:

[Pieter Malan (pmalan)]

We saw that VRCNTR_WIRE_DROP_INVALID_VIF gets incremented for packets arriving over the wire. The source side seems to be operating as expected. It is on the destination side where the packets are dropped.

INVALID_VIF gets incremented when the input_vif_index == -1, further investigation is required to determine why the input_vif_index is -1.

[Pieter Malan (pmalan)]

I am seeing this behaviour when using nfp_p1 port and not nfp_p0.

[Ankit Jain (ankitja)]

With nfp_p0 interface, the issue was not reproduced.