Contrail 2.21-14: Flows not getting timedout in flow table

[Deepak Jeyaraman]

Using contrail 2.21-14 Icehouse, we are seeing issues with flows not getting timeout in flow-table.

These are mostly PING and SSH flows from our monitoring tool (10.102.86.8 ) and VNF IP - 10.102.80.117

The above is just an example

Initially things were fine, however left the system overnight, we see issues like high ICMP RTT, packet drops.

Our monitoring tool sends PING every 30 secs for each VNF (5-8 ICMP REQUESTS every 30 secs)

root@ccra-20-sim-vm:/var/log/icinga2# cat 4 | grep 80.117
14:44:45.533507 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 13863, seq 1, length 64
14:44:46.533601 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 13863, seq 2, length 64
14:44:47.540243 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 13863, seq 3, length 64
14:44:48.548223 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 13863, seq 4, length 64
14:44:49.556180 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 13863, seq 5, length 64
14:44:50.564161 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 13863, seq 6, length 64
14:44:51.572202 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 13863, seq 7, length 64
14:44:51.724506 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 13863, seq 1, length 64
14:44:51.724740 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 13863, seq 2, length 64
14:44:51.724925 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 13863, seq 3, length 64
14:44:52.577318 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 13863, seq 8, length 64
14:44:52.579841 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 13863, seq 8, length 64
14:44:53.583016 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 13863, seq 9, length 64
14:44:53.585461 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 13863, seq 9, length 64
14:45:15.533959 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14130, seq 1, length 64
14:45:16.534129 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14130, seq 2, length 64
14:45:17.534099 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14130, seq 3, length 64
14:45:18.534101 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14130, seq 4, length 64
14:45:19.534097 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14130, seq 5, length 64
14:45:20.534070 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14130, seq 6, length 64
14:45:21.534003 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14130, seq 7, length 64
14:45:22.091818 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 14130, seq 1, length 64
14:45:22.092350 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 14130, seq 2, length 64
14:45:22.092523 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 14130, seq 3, length 64
14:45:22.534920 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14130, seq 8, length 64
14:45:22.538668 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 14130, seq 8, length 64
14:45:23.535976 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14130, seq 9, length 64
14:45:23.538931 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 14130, seq 9, length 64
14:45:45.533009 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14450, seq 1, length 64
14:45:46.533714 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14450, seq 2, length 64
14:45:47.540221 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14450, seq 3, length 64
14:45:48.548247 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14450, seq 4, length 64
14:45:49.556182 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14450, seq 5, length 64
14:45:50.564242 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14450, seq 6, length 64
14:45:51.031441 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 14450, seq 1, length 64
14:45:51.031726 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 14450, seq 2, length 64
14:45:51.031840 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 14450, seq 3, length 64
14:45:51.565347 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14450, seq 7, length 64
14:45:51.567060 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 14450, seq 7, length 64
14:45:52.566448 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14450, seq 8, length 64
14:45:52.569723 IP 10.102.80.117 > 10.102.86.8: ICMP echo reply, id 14450, seq 8, length 64
14:46:15.534146 IP 10.102.86.8 > 10.102.80.117: ICMP echo request, id 14709, seq 1, length 64

root@ccra-22:~# flow -l | grep 80.117 | grep 86.8 | wc
   8035 40175 582589

Later, the monitoring tool is forced to stop monitoring the VNF - 10.102.80.117

Still the number of flows not decreasing even after 10 mins

root@ccra-22:~# flow -l | grep 80.117 | grep 86.8 | wc
   8035 40175 582589
root@ccra-22:~# flow -l | grep 80.117 | grep 86.8 | wc
   8035 40175 582589

from flow table:

507476 10.102.86.8:21289 10.102.80.117:0 1 (2)
507572 10.102.80.117:14130 10.102.86.8:0 1 (2)
507692 10.102.86.8:2356 10.102.80.117:0 1 (2)
507785 10.102.80.117:15286 10.102.86.8:0 1 (2)
507812 10.102.80.117:22 10.102.86.8:43514 6 (2)
507852 10.102.80.117:22 10.102.86.8:48498 6 (2)
507868 10.102.86.8:11758 10.102.80.117:0 1 (2)
507888 10.102.80.117:14041 10.102.86.8:0 1 (2)
507956 10.102.86.8:44173 10.102.80.117:22 6 (2)
507976 10.102.86.8:36356 10.102.80.117:22 6 (2)
508088 10.102.86.8:10523 10.102.80.117:0 1 (2)
508176 10.102.80.117:25071 10.102.86.8:0 1 (2)
508236 10.102.80.117:31886 10.102.86.8:0 1 (2)
508244 10.102.86.8:41339 10.102.80.117:22 6 (2)
508260 10.102.86.8:5452 10.102.80.117:0 1 (2)
508444 10.102.80.117:22 10.102.86.8:58134 6 (2)
508476 10.102.80.117:749 10.102.86.8:0 1 (2)
508548 10.102.80.117:11308 10.102.86.8:0 1 (2)
508625 10.102.86.8:19139 10.102.80.117:0 1 (2)
508700 10.102.80.117:26927 10.102.86.8:0 1 (2)
508800 10.102.80.117:5968 10.102.86.8:0 1 (2)
508812 10.102.80.117:5293 10.102.86.8:0 1 (2)
508853 10.102.86.8:43859 10.102.80.117:22 6 (2)
509040 10.102.86.8:12891 10.102.80.117:0 1 (2)
509084 10.102.86.8:52194 10.102.80.117:22 6 (2)
509204 10.102.80.117:30190 10.102.86.8:0 1 (2)
509212 10.102.86.8:41034 10.102.80.117:22 6 (2)
509256 10.102.80.117:22 10.102.86.8:52562 6 (2)
509305 10.102.80.117:22 10.102.86.8:43226 6 (2)
509340 10.102.80.117:22 10.102.86.8:36503 6 (2)
509434 10.102.86.8:6662 10.102.80.117:0 1 (2)
509509 10.102.80.117:20575 10.102.86.8:0 1 (2)
509512 10.102.80.117:16507 10.102.86.8:0 1 (2)

root@ccra-22:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-fabric-utils 2.21.1-14 14
contrail-install-packages 2.21.1-14~icehouse 14
contrail-lib 2.21.1-14 14
contrail-nodemgr 2.21.1-14 14

Tasks: 549 total, 2 running, 547 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.4 us, 1.1 sy, 0.0 ni, 93.5 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 23088614+total, 82431488 used, 14845465+free, 364124 buffers
KiB Swap: 23477043+total, 0 used, 23477043+free. 4549628 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 3254 root 20 0 56.591g 0.054t 46620 S 426.3 24.9 24027:19 contrail-vroute
 5531 libvirt+ 20 0 9034396 3.630g 12708 S 195.4 1.6 1885:40 qemu-system-x86
 6761 libvirt+ 20 0 8711064 3.617g 12708 S 195.4 1.6 1699:52 qemu-system-x86
29825 libvirt+ 20 0 8871384 3.591g 12704 S 142.1 1.6 1783:42 qemu-system-x86
42085 libvirt+ 20 0 9058392 3.634g 12708 S 142.1 1.7 1859:44 qemu-system-x86
42760 libvirt+ 20 0 8420544 3.590g 12704 S 142.1 1.6 1736:39 qemu-system-x86
43139 libvirt+ 20 0 9065304 3.636g 12708 S 142.1 1.7 1891:21 qemu-system-x86
17883 libvirt+ 20 0 8279056 3.585g 12708 S 136.2 1.6 1806:44 qemu-system-x86
25608 libvirt+ 20 0 8344568 3.594g 12708 S 136.2 1.6 1731:05 qemu-system-x86
48470 libvirt+ 20 0 9058012 3.637g 12792 S 136.2 1.7 1695:02 qemu-system-x86
 4744 libvirt+ 20 0 8824160 3.606g 12708 S 130.3 1.6 1797:12 qemu-system-x86
46340 libvirt+ 20 0 9061712 3.626g 12708 S 130.3 1.6 1901:41 qemu-system-x86
16759 root 20 0 145212 48888 3408 S 29.6 0.0 0:01.26 puppet
16892 root 20 0 0 0 0 S 5.9 0.0 0:52.04 kworker/16:2
17375 root 20 0 23940 1896 1076 R 5.9 0.0 0:00.02 top
    1 root 20 0 34164 3544 1468 S 0.0 0.0 10:48.75 init
    2 root 20 0 0 0 0 S 0.0 0.0 0:00.79 kthreadd
    3 root 20 0 0 0 0 S 0.0 0.0 2:06.03 ksoftirqd/0
    5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
    6 root 20 0 0 0 0 S 0.0 0.0 0:41.78 kworker/u96:0
    8 root 20 0 0 0 0 S 0.0 0.0 81:39.40 rcu_sched
    9 root 20 0 0 0 0 S 0.0 0.0 4:07.16 rcuos/0