2.21-1-B22: Unicast traffic to VM got dropped due to Invalid NH

[Sandeep Sridhar]

Setup Details :

Cfgm/Control
10.204.74.76
10.204.74.78
10.204.74.79

Compute
10.204.74.64
10.204.74.65
10.204.74.77

TSN HA
10.204.74.61
10.204.74.66

I have created one VN 50.50.50.0/24 that has the following :

50.50.50.3 and 50.50.50.7 are the VMs on 10.204.74.64
50.50.50.4 is the VM on 10.204.74.65

50.50.50.5 and 50.50.50.6 are the BMS.

I see unicast traffic from 50.50.50.5 (BMS) to 50.50.50.4 (VM ) is not going through. No issues seen with broadcast traffic as ARP is getting resolved. Also, the unicast traffic to VMs on other compute nodes are fine too.

The VM's tap is receiving ICMP request and it is sending ICMP reply too. However, the physical interface (em1) doesn't see the ICMP reply. The "Invalid NH" is incrementing on the vRouter.

L2 table looks like this :

root@contrail65:~# rt --dump 1 --family bridge
Kernel L2 Bridge table 0/1

Flags: L=Label Valid, Df=DHCP flood

Index DestMac Flags Label/VNID Nexthop
97192 ff:ff:ff:ff:ff:ff LDf 4 40
151116 14:2:ec:31:d3:bc Df - 3
156232 14:2:ec:31:e3:94 LDf 4 38
202436 2:82:ff:57:a3:20 - 36 <<<<<<NH is 36 for VM MAC>>>>>>>>
203236 2:bf:9:67:b2:e LDf 17 28
214580 2:1e:31:7c:d:fb LDf 20 28
252916 0:0:5e:0:1:0 Df - 3

flow looks like this :

root@contrail65:~# flow -l
Flow table(size 34078720, entries 532480)

Entries: Created 90 Added 90 Processed 90
(Created Flows/CPU: 0 0 0 0 0 0 23 6 0 0 0 0 0 0 0 12 0 21 0 8 10 9 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop

 Index Source:Port Destination:Port Proto(V)
-------------------------------------------------------------------------
231056 50.50.50.5:3629 50.50.50.4:0 1 (1)
 (K(nh):21, Action:F, S(nh):38, Statistics:3485/341530 UdpSrcPort 63278)

446020 50.50.50.4:3629 50.50.50.5:0 1 (1)
 (K(nh):21, Action:F, S(nh):2, Statistics:3485/341530 UdpSrcPort 57547)

In the flow, NH is 21. There is a difference between NH in flow and the L2 table.

Also, If you notice for Index 446020 (ICMP reply to BMS), 2 seems to be an invalid next-hop. Not sure why this flow is not removed.

root@contrail65:~# nh --get 2
Error No such file or directory in kernel operation

gcores are collected and shared with Manish and team.

Greetings,
Sandeep.