EVPN VXLAN: Ping to DNS server IP is failing after first packet

[chhandak]

Trying with EVPN VXLAN changes for TSN HA (Agent and vrouter updated)

When pinging from BMS to DNS server IP, ping is failing after the first packet. In dropstats Flow Queue Limit Exceeded counter is increasing. FLow is stuck in Hold state.

root@5b11s16:~#
root@5b11s16:~# ping -I p2p1.6 1.1.1.2
PING 1.1.1.2 (1.1.1.2) from 1.1.1.3 p2p1.6: 56(84) bytes of data.
64 bytes from 1.1.1.2: icmp_seq=1 ttl=64 time=0.488 ms
^C
--- 1.1.1.2 ping statistics ---
4 packets transmitted, 1 received, 75% packet loss, time 3015ms
rtt min/avg/max/mdev = 0.488/0.488/0.488/0.000 ms

root@5b11s14:~# dropstats | grep -v " 0$"

Flow Queue Limit Exceeded 454

Discards 687
Cloned Original 1374

Invalid NH 9

root@5b11s14:~# dropstats | grep -v " 0$"

Flow Queue Limit Exceeded 464

Discards 695
Cloned Original 1390

Invalid NH 9

root@5b11s14:~# flow -l
Flow table(size 80609280, entries 629760)

Entries: Created 4 Added 0 Deleted 3 Changed 3 Processed 3 Used Overflow entries 0
(Created Flows/CPU: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0)(oflows 0)

Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
 Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
 Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified Dm=Delete Marked
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
   226064 1.1.1.3:5610 1 (3)
                         1.1.1.2:0
(Gen: 4, K(nh):0, Action:H, Flags:, QOS:-1, S(nh):0, Stats:81/6804, SPort 0,
 TTL 0, Sinfo 172.16.2.1)

[chhandak (chhandak)]

Hi Hari,

I have reproduced the issue. Please find the setup details below

Password (root/c0ntrail123)
Controller + TSN 10.87.69.1
TSN2 10.87.69.4
Agent 10.87.69.2
BMS 10.87.69.3 (Interface p2p1.6)

QFX:10.87.69.124 (root/Embe1mpls)

root@5b11s16:~# ping -I p2p1.6 17.17.17.2
PING 17.17.17.2 (17.17.17.2) from 17.17.17.3 p2p1.6: 56(84) bytes of data.
64 bytes from 17.17.17.2: icmp_seq=1 ttl=64 time=0.662 ms
^C
--- 17.17.17.2 ping statistics ---
4 packets transmitted, 1 received, 75% packet loss, time 3015ms
rtt min/avg/max/mdev = 0.662/0.662/0.662/0.000 ms

Thanks and Regards,
Chhandak

On 9/6/17, 6:44 PM, "<email address hidden> on behalf of Jeba Paulaiyan" <<email address hidden> on behalf of <email address hidden>> wrote:

>** Also affects: juniperopenstack/r4.0
> Importance: Undecided
> Status: New
>
>** Also affects: juniperopenstack/trunk
> Importance: High
> Assignee: Hari Prasad Killi (haripk)
> Status: New
>
>** Changed in: juniperopenstack/r4.0
> Importance: Undecided => High
>
>** Changed in: juniperopenstack/r4.0
> Assignee: (unassigned) => Hari Prasad Killi (haripk)
>
>** Changed in: juniperopenstack/r4.0
> Milestone: None => r4.0.1.0
>
>** Changed in: juniperopenstack/trunk
> Milestone: r4.0.1.0 => r4.1.0.0-fcs
>
>--
>You received this bug notification because you are a member of Contrail
>Systems engineering, which is subscribed to Juniper Openstack.
>https://bugs.launchpad.net/bugs/1715510
>
>Title:
> EVPN VXLAN: Ping to DNS server IP is failing after first packet
>
>Status in Juniper Openstack:
> New
>Status in Juniper Openstack r4.0 series:
> New
>Status in Juniper Openstack trunk series:
> New
>
>Bug description:
> Trying with EVPN VXLAN changes for TSN HA (Agent and vrouter updated)
>
> When pinging from BMS to DNS server IP, ping is failing after the
> first packet. In dropstats Flow Queue Limit Exceeded counter is
> increasing. FLow is stuck in Hold state.
>
> root@5b11s16:~#
> root@5b11s16:~# ping -I p2p1.6 1.1.1.2
> PING 1.1.1.2 (1.1.1.2) from 1.1.1.3 p2p1.6: 56(84) bytes of data.
> 64 bytes from 1.1.1.2: icmp_seq=1 ttl=64 time=0.488 ms
> ^C
> --- 1.1.1.2 ping statistics ---
> 4 packets transmitted, 1 received, 75% packet loss, time 3015ms
> rtt min/avg/max/mdev = 0.488/0.488/0.488/0.000 ms
>
>
> root@5b11s14:~# dropstats | grep -v " 0$"
>
> Flow Queue Limit Exceeded 454
>
>
> Discards 687
> Cloned Original 1374
>
> Invalid NH 9
>
>
> root@5b11s14:~# dropstats | grep -v " 0$"
>
> Flow Queue Limit Exceeded 464
>
>
> Discards 695
> Cloned Original 1390
>
> Invalid NH 9
>
> root@5b11s14:~# flow -l
> Flow table(size 80609280, entries 629760)
>
> Entries: Created 4 Added 0 Deleted 3 Changed 3 Processed 3 Used Overflow entries 0
> (Created Flows/CPU: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 4 0 0 0 0 0 0 0 0 0 0)(oflows 0)
>
> Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
> Other:K(nh)=Key_Next...

[Hari Prasad Killi (haripk)]

Arp for DNS address is resolved and DNS resolution works fine.

root@5b11s16:~# arp -an
? (10.87.69.126) at 30:7c:5e:0f:8f:c0 [ether] on p4p1
? (17.17.17.2) at 90:e2:ba:a7:30:cd [ether] on p2p1.6

root@5b11s16:~# nslookup
> server 17.17.17.2
Default server: 17.17.17.2
Address: 17.17.17.2#53
> test.com
Server: 17.17.17.2
Address: 17.17.17.2#53

Non-authoritative answer:
Name: test.com
Address: 69.172.200.235
> cnn.com
Server: 17.17.17.2
Address: 17.17.17.2#53

Non-authoritative answer:
Name: cnn.com
Address: 151.101.129.67
Name: cnn.com
Address: 151.101.193.67
Name: cnn.com
Address: 151.101.65.67
Name: cnn.com
Address: 151.101.1.67
>

However, only the first ping passes.

root@5b11s16:~# ping 17.17.17.2
PING 17.17.17.2 (17.17.17.2) 56(84) bytes of data.
64 bytes from 17.17.17.2: icmp_seq=1 ttl=64 time=0.486 ms
^C
--- 17.17.17.2 ping statistics ---
6 packets transmitted, 1 received, 83% packet loss, time 4999ms
rtt min/avg/max/mdev = 0.486/0.486/0.486/0.000 ms

Flow for the ping remains in Hold state.

    Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
   260168 17.17.17.3:12626 1 (2)
                         17.17.17.2:0
(Gen: 1, K(nh):0, Action:H, Flags:, QOS:0, S(nh):0, Stats:28/2352, SPort 0,
 TTL 0, Sinfo 172.16.2.1)

There is no functional impact seen, only ping to the DNS address fails.

[Manish Singh (manishs)]

QFX is not present in IM list.
http://10.87.69.1:8083/Snh_ShowRouteReq?x=bgp.evpn.0

Please check why QFX has removed itself.
http://10.87.69.1:8083/Snh_SandeshTraceRequest?x=BgpTraceBuf

2017-09-09 14:06:59.600 BgpPeerRouteTrace: BGP Peer default-domain:default-project:ip-fabric:__default__:5b11s14:default-domain:default-project:ip-fabric:__default__:5b11-qfx2 Delete BGP path 3-172.16.2.1:1-12345-172.16.2.1 in table bgp.evpn.0 controller/src/bgp/bgp_table.cc 453

Thanks,
Manish

On 9/9/17, 11:55 AM, "Chhandak Mukherjee" <email address hidden> wrote:

    10.87.69.4

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35468
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35470
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35472
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35470
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35472
Committed: http://github.com/Juniper/contrail-controller/commit/2d4a1970a14e4954c1c39f1c03f597aae9c55df7
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 2d4a1970a14e4954c1c39f1c03f597aae9c55df7
Author: Hari Prasad Killi <email address hidden>
Date: Mon Sep 11 15:01:59 2017 +0530

Disable policy flag on the host interface in TSN

There is no need to setup flows in TSN, hence use the NE with policy disabled
in case of TSN. Also remove unused relaxed_policy flag.

Change-Id: Ic695c3a443eddf71c0a954478280ccd4bb4b1ff3
closes-bug: #1715510

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35468
Committed: http://github.com/Juniper/contrail-controller/commit/fc6de955f0da926bc2208fbc090062fd9b6114d6
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit fc6de955f0da926bc2208fbc090062fd9b6114d6
Author: Hari Prasad Killi <email address hidden>
Date: Mon Sep 11 14:15:56 2017 +0530

Disable policy flag on the host interface in TSN

There is no need to setup flows in TSN, hence use the NE with policy disabled
in case of TSN. Also remove unused relaxed_policy flag.

Change-Id: Ic695c3a443eddf71c0a954478280ccd4bb4b1ff3
closes-bug: #1715510

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35470
Committed: http://github.com/Juniper/contrail-controller/commit/c64bafe9bd0d7bb460c747ae8c81925c4cbc479a
Submitter: Zuul (<email address hidden>)
Branch: master

commit c64bafe9bd0d7bb460c747ae8c81925c4cbc479a
Author: Hari Prasad Killi <email address hidden>
Date: Mon Sep 11 14:37:23 2017 +0530

Disable policy flag on the host interface in TSN

There is no need to setup flows in TSN, hence use the NE with policy disabled
in case of TSN. Also remove unused relaxed_policy flag.

Change-Id: Ic695c3a443eddf71c0a954478280ccd4bb4b1ff3
closes-bug: #1715510

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/35521
Submitter: Hari Prasad Killi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/35521
Committed: http://github.com/Juniper/contrail-controller/commit/4024831cecdab43a5924120d2560a880b451cf8a
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 4024831cecdab43a5924120d2560a880b451cf8a
Author: Hari Prasad Killi <email address hidden>
Date: Mon Sep 11 15:01:59 2017 +0530

Disable policy flag on the host interface in TSN

There is no need to setup flows in TSN, hence use the NE with policy disabled
in case of TSN. Also remove unused relaxed_policy flag.

Change-Id: Ic695c3a443eddf71c0a954478280ccd4bb4b1ff3
closes-bug: #1715510
(cherry picked from commit 2d4a1970a14e4954c1c39f1c03f597aae9c55df7)