vRouter drops all icmp echo-reply

[Konstantin]

Hello.

I have following Contrail version on Compute node

contrail-analytics-venv 1.03-386.el6 386
contrail-api-lib 1.03-386.el6 386
contrail-api-venv 1.03-386.el6 386
contrail-control-venv 1.03-386.el6 386
contrail-database-venv 1.03-386.el6 386
contrail-fabric-utils 1.03-386 386
contrail-interface-name 1.03-386.centos 386
contrail-libs 1.03-386.el6 386
contrail-openstack-vrouter 1.03-386.el6 386
contrail-setup 1.03-386.el6 386
contrail-vrouter 1.03-386.el6 386
contrail-vrouter-venv 1.03-386.el6 386

Physically I have 2 control nodes, 1 MX5 , 1 Compute node.
Control and compute nodes directly connected to MX5.

PC1 --- MX5 -----PC2
                 |
          |
       Compute

PC1 - 10.0.242.10
PC2 - 10.0.243.10

On Compute node I have 2 service instances: Firewall and NAT.

Logically I have following chain

PC1 -------- MX5 -------- FW ---------NAT --------MX5 ----------PC2
                  (vrf FWNATL) (vrf FWNATR)

I have run test ping from PC1 to PC2.
Result:

1. PC2 receive echo request and send echo reply

2. FW transmit echo-reply on tap interface tapadf5d1c7-ea

[root@Comp1Contrail ~]# tcpdump -i tapadf5d1c7-ea
tcpdump: WARNING: tapadf5d1c7-ea: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tapadf5d1c7-ea, link-type EN10MB (Ethernet), capture size 65535 bytes
16:16:48.467671 IP 10.0.242.10 > 10.0.243.10: ICMP echo request, id 512, seq 52098, length 1408
16:16:48.469717 IP 10.0.243.10 > 10.0.242.10: ICMP echo reply, id 512, seq 52098, length 1408

3. On the physical interface I cannot see echo-reply in direction from FW to PC1.

[root@Comp1Contrail ~]# tcpdump -i p5p0p0 -v
tcpdump: listening on p5p0p0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:22:00.482284 IP (tos 0x0, ttl 64, id 32854, offset 0, flags [DF], proto GRE (47), length 1456)
    10.0.197.20 > 10.0.197.10: GREv0, Flags [none], length 1436
        MPLS (label 22, exp 0, [S], ttl 127)
        IP (tos 0x0, ttl 127, id 51033, offset 0, flags [none], proto ICMP (1), length 1428)
    10.0.242.10 > 10.0.243.10: ICMP echo request, id 512, seq 39811, length 1408
16:22:00.482953 IP (tos 0x0, ttl 64, id 34255, offset 0, flags [none], proto GRE (47), length 1456)
    10.0.197.10 > 10.0.197.20: GREv0, Flags [none], length 1436
        MPLS (label 17, exp 0, [S], ttl 64)
        IP (tos 0x0, ttl 125, id 51033, offset 0, flags [none], proto ICMP (1), length 1428)
    172.16.2.5 > 10.0.243.10: ICMP echo request, id 1, seq 39811, length 1408
16:22:00.483811 IP (tos 0x0, ttl 64, id 18790, offset 0, flags [DF], proto GRE (47), length 1456)
    10.0.197.20 > 10.0.197.10: GREv0, Flags [none], length 1436
        MPLS (label 24, exp 0, [S], ttl 127)
        IP (tos 0x0, ttl 127, id 25877, offset 0, flags [none], proto ICMP (1), length 1428)
    10.0.243.10 > 172.16.2.5: ICMP echo reply, id 1, seq 39811, length 1408
16:22:01.982340 IP (tos 0x0, ttl 64, id 21350, offset 0, flags [DF], proto GRE (47), length 1456)
    10.0.197.20 > 10.0.197.10: GREv0, Flags [none], length 1436
        MPLS (label 22, exp 0, [S], ttl 127)
        IP (tos 0x0, ttl 127, id 51034, offset 0, flags [none], proto ICMP (1), length 1428)
    10.0.242.10 > 10.0.243.10: ICMP echo request, id 512, seq 40067, length 1408
16:22:01.983071 IP (tos 0x0, ttl 64, id 34256, offset 0, flags [none], proto GRE (47), length 1456)
    10.0.197.10 > 10.0.197.20: GREv0, Flags [none], length 1436
        MPLS (label 17, exp 0, [S], ttl 64)
        IP (tos 0x0, ttl 125, id 51034, offset 0, flags [none], proto ICMP (1), length 1428)
    172.16.2.5 > 10.0.243.10: ICMP echo request, id 1, seq 40067, length 1408
16:22:03.482477 IP (tos 0x0, ttl 64, id 18473, offset 0, flags [DF], proto GRE (47), length 1456)
    10.0.197.20 > 10.0.197.10: GREv0, Flags [none], length 1436
        MPLS (label 22, exp 0, [S], ttl 127)
        IP (tos 0x0, ttl 127, id 51035, offset 0, flags [none], proto ICMP (1), length 1428)
    10.0.242.10 > 10.0.243.10: ICMP echo request, id 512, seq 40323, length 1408
16:22:03.483358 IP (tos 0x0, ttl 64, id 34257, offset 0, flags [none], proto GRE (47), length 1456)
    10.0.197.10 > 10.0.197.20: GREv0, Flags [none], length 1436
        MPLS (label 17, exp 0, [S], ttl 64)
        IP (tos 0x0, ttl 125, id 51035, offset 0, flags [none], proto ICMP (1), length 1428)
    172.16.2.5 > 10.0.243.10: ICMP echo request, id 1, seq 40323, length 1408
16:22:03.484206 IP (tos 0x0, ttl 64, id 64026, offset 0, flags [DF], proto GRE (47), length 1456)
    10.0.197.20 > 10.0.197.10: GREv0, Flags [none], length 1436
        MPLS (label 24, exp 0, [S], ttl 127)
        IP (tos 0x0, ttl 127, id 25879, offset 0, flags [none], proto ICMP (1), length 1428)
    10.0.243.10 > 172.16.2.5: ICMP echo reply, id 1, seq 40323, length 1408

But at the same time ping from FW to PC1 – successful

[Konstantin (kfedor)]

I have attached lab scheme.
(more detailed)

Ping from PC1 to PC2

For FWNATL network

[root@Comp1Contrail /]# tcpdump -i tapadf5d1c7-ea -v
tcpdump: WARNING: tapadf5d1c7-ea: no IPv4 address assigned
tcpdump: listening on tapadf5d1c7-ea, link-type EN10MB (Ethernet), capture size 65535 bytes
14:33:02.305401 IP (tos 0x0, ttl 127, id 46914, offset 0, flags [none], proto ICMP (1), length 60)
    10.0.242.10 > 10.0.243.10: ICMP echo request, id 512, seq 13424, length 40
14:33:02.308336 IP (tos 0x0, ttl 125, id 19290, offset 0, flags [none], proto ICMP (1), length 60)
    10.0.243.10 > 10.0.242.10: ICMP echo reply, id 512, seq 13424, length 40

For FWNATM network

[root@Comp1Contrail /]# tcpdump -i tap261044a5-4c -v
tcpdump: WARNING: tap261044a5-4c: no IPv4 address assigned
tcpdump: listening on tap261044a5-4c, link-type EN10MB (Ethernet), capture size 65535 bytes
14:26:44.287796 IP (tos 0x0, ttl 126, id 46661, offset 0, flags [none], proto ICMP (1), length 60)
    10.0.242.10 > 10.0.243.10: ICMP echo request, id 512, seq 14447, length 40
14:26:44.293513 IP (tos 0x0, ttl 126, id 19038, offset 0, flags [none], proto ICMP (1), length 60)
    10.0.243.10 > 10.0.242.10: ICMP echo reply, id 512, seq 14447, length 40

[root@Comp1Contrail /]# tcpdump -i tap2cddec3a-01 -v
tcpdump: WARNING: tap2cddec3a-01: no IPv4 address assigned
tcpdump: listening on tap2cddec3a-01, link-type EN10MB (Ethernet), capture size 65535 bytes
14:27:12.789107 IP (tos 0x0, ttl 126, id 46680, offset 0, flags [none], proto ICMP (1), length 60)
    10.0.242.10 > 10.0.243.10: ICMP echo request, id 512, seq 19311, length 40
14:27:12.791192 IP (tos 0x0, ttl 126, id 19057, offset 0, flags [none], proto ICMP (1), length 60)
    10.0.243.10 > 10.0.242.10: ICMP echo reply, id 512, seq 19311, length 40

For network FWNATR

[root@Comp1Contrail /]# tcpdump -i tapa0928a51-eb -v
tcpdump: WARNING: tapa0928a51-eb: no IPv4 address assigned
tcpdump: listening on tapa0928a51-eb, link-type EN10MB (Ethernet), capture size 65535 bytes
14:34:42.811213 IP (tos 0x0, ttl 125, id 46981, offset 0, flags [none], proto ICMP (1), length 60)
    172.16.2.5 > 10.0.243.10: ICMP echo request, id 1, seq 30576, length 40
14:34:42.811828 IP (tos 0x0, ttl 127, id 19358, offset 0, flags [none], proto ICMP (1), length 60)
    10.0.243.10 > 172.16.2.5: ICMP echo reply, id 1, seq 30576, length 40

For global network

[root@Comp1Contrail /]# tcpdump -i p5p0p0 -v
tcpdump: WARNING: p5p0p0: no IPv4 address assigned
tcpdump: listening on p5p0p0, link-type EN10MB (Ethernet), capture size 65535 bytes
14:35:36.812859 IP (tos 0x0, ttl 64, id 46299, offset 0, flags [DF], proto GRE (47), length 88)
    10.0.197.20 > 10.0.197.10: GREv0, Flags [none], length 68
        MPLS (label 19, exp 0, [S], ttl 127)
        IP (tos 0x0, ttl 127, id 47017, offset 0, flags [none], proto ICMP (1), length 60)
    10.0.242.10 > 10.0.243.10: ICMP echo request, id 512, seq 39792, length 40
14:35:36.815780 IP (tos 0x0, ttl 64, id 1697, offset 0, flags [none], proto GRE (47), length 88)
    10.0.197.10 > 10.0.197.20: GREv0, Flags [none], length 68
        MPLS (label 17, exp 0, [S], ttl 64)
        IP (tos 0x0, ttl 125, id 47017, offset 0, flag...