BMS ping BMS failed due to arp respond from vhost0

[Sharon Zhang]

The test setup has BMSes connected to Opus TOR, Opus are connected to MX GW, and Contrail Vrouters are also connected to MX gateway.

Ping from Contrail VM to BMS works fine, but ping between two BMS in the same VN failed.

The detail topology will be attached to this PR.

Captured the packets, it shows that the arp request from BMS1 got 3 respond for the same host (BMS2).

BMS1: IP 101.1.1.20 MAC: 00:10:94:00:00:19
BMS2: IP 101.1.1.21 MAC: 00:10:94:00:00:17

Ping from BMS1 to BMS2: got arp respond from three hosts:

BMS2: 101.1.1.20 at 00:10:94:00:00:19
vhost0 on vrouter1 (sdn-st-dellpc-c): 101.1.1.20 at 14:fe:b5:cc:73:f2
vhost0 on vrouter2(sdn-st-dellpc-e): 101.1.1.20 at 14:fe:b5:cc:74:dc

The capture file on both BMS1 and BMS2 will be attached to the PR.

root@sdn-st-dellpc-e:~# ifconfig |grep 14:fe
em1 Link encap:Ethernet HWaddr 14:fe:b5:cc:74:da
em2 Link encap:Ethernet HWaddr 14:fe:b5:cc:74:dc
vhost0 Link encap:Ethernet HWaddr 14:fe:b5:cc:74:dc
root@sdn-st-dellpc-e:~# ifconfig vhost0
vhost0 Link encap:Ethernet HWaddr 14:fe:b5:cc:74:dc
          inet addr:192.14.1.10 Bcast:192.14.1.255 Mask:255.255.255.0
          inet6 addr: fe80::16fe:b5ff:fecc:74dc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:6690600 errors:0 dropped:196 overruns:0 frame:0
          TX packets:8191847 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1381280714 (1.3 GB) TX bytes:9748573454 (9.7 GB)

root@sdn-st-dellpc-e:/home/regress# ifconfig em2
em2 Link encap:Ethernet HWaddr 14:fe:b5:cc:74:dc
          inet6 addr: fe80::16fe:b5ff:fecc:74dc/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2989936882 errors:0 dropped:29942 overruns:0 frame:0
          TX packets:105124710 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:432178728813 (432.1 GB) TX bytes:20840328168 (20.8 GB)

root@sdn-st-dellpc-c:/var/tmp# ifconfig |grep 14:fe
em1 Link encap:Ethernet HWaddr 14:fe:b5:cc:73:f0
em2 Link encap:Ethernet HWaddr 14:fe:b5:cc:73:f2
vhost0 Link encap:Ethernet HWaddr 14:fe:b5:cc:73:f2
root@sdn-st-dellpc-c:/var/tmp# ifconfig vhost0
vhost0 Link encap:Ethernet HWaddr 14:fe:b5:cc:73:f2
          inet addr:192.11.1.10 Bcast:192.11.1.255 Mask:255.255.255.0
          inet6 addr: fe80::16fe:b5ff:fecc:73f2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:15223755 errors:0 dropped:585 overruns:0 frame:0
          TX packets:19568275 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2160613933 (2.1 GB) TX bytes:21182898261 (21.1 GB)

root@sdn-st-dellpc-c:/var/tmp# ifconfig em2
em2 Link encap:Ethernet HWaddr 14:fe:b5:cc:73:f2
          inet6 addr: fe80::16fe:b5ff:fecc:73f2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:3933896038 errors:0 dropped:69008 overruns:0 frame:0
          TX packets:660320590 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:551879637126 (551.8 GB) TX bytes:93958780865 (93.9 GB)

Since the last respond is from vrouter vhost0, hence ping will failed. Ping will go through sometime if the real bms host is the last one to respond the arp.

The contrail has 10VMs on the same VN, and the IP address for the VM is from 101.1.1.1-101.1.1.11

This is very critical issue. I tried to capture the packets on vhost0 interface on vrouter, but could not get the arp packets.

Which interface should I use to capture the packets from BMS?

›

[Sharon Zhang (sharonz)]

Attach the topology

[Sharon Zhang (sharonz)]

add captures from Sprient BMS1 (sender)

[Sharon Zhang (sharonz)]

Add capture from Sprient BMS2 (receiver)

[Hari Prasad Killi (haripk)]

Looks like https://bugs.launchpad.net/juniperopenstack/+bug/1485804, which is now fixed in latest R2.21 build. Is the setup having two MXs ?

[Sharon Zhang (sharonz)]

Yes, the setup has two MXs. But the setup does not have TSN.

Which build for R2.21 has the this fix? It is a blocker for me since I can not do traffic between BMS.

Thanks,

Sharon

[Sharon Zhang (sharonz)]

I could not find R2.21 image under /volume/contrail as listed below. Where can I find the image which has the fix?

ls /volume/contrail/
.apdisk file-list R1.05/ R2.0/ rsycn.sh ._.TemporaryItems
._.apdisk git-manifests/ R1.06/ R2.1/ Signed-Images/ VM-Templates/
build/ install_repo/ R1.06c1/ R2.20/ signing/
CentOS-7.0-1406-x86_64-DVD.iso mainline/ R1.10/ R2.22-dev/ .snapshot/
distro-packages/ R1.04/ R1.30/ Release-Archives/ .TemporaryItems/

[Hari Prasad Killi (haripk)]

R2.21 is not officially yet released officially. Fix for this issue is in dev build 94.

[Sharon Zhang (sharonz)]

Hi Hari,

Does build 93 have the fix? I just upgraded to 93 build last week.

Thanks,

Sharon

________________________________________
From: <email address hidden> <email address hidden> on behalf of Hari Prasad Killi <email address hidden>
Sent: Friday, September 04, 2015 10:11 PM
To: Sharon Zhang
Subject: [Bug 1491644] Re: BMS ping BMS failed due to arp respond from vhost0

R2.21 is not officially yet released officially. Fix for this issue is
in dev build 94.

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1491644

Title:
  BMS ping BMS failed due to arp respond from vhost0

Status in Juniper Openstack:
  New

Bug description:

  The test setup has BMSes connected to Opus TOR, Opus are connected to MX GW, and Contrail Vrouters are also connected to MX gateway.

  Ping from Contrail VM to BMS works fine, but ping between two BMS in
  the same VN failed.

  The detail topology will be attached to this PR.

  Captured the packets, it shows that the arp request from BMS1 got 3 respond for the same host (BMS2).

  BMS1: IP 101.1.1.20 MAC: 00:10:94:00:00:19
  BMS2: IP 101.1.1.21 MAC: 00:10:94:00:00:17

  Ping from BMS1 to BMS2: got arp respond from three hosts:

  BMS2: 101.1.1.20 at 00:10:94:00:00:19
  vhost0 on vrouter1 (sdn-st-dellpc-c): 101.1.1.20 at 14:fe:b5:cc:73:f2
  vhost0 on vrouter2(sdn-st-dellpc-e): 101.1.1.20 at 14:fe:b5:cc:74:dc

  The capture file on both BMS1 and BMS2 will be attached to the PR.

  root@sdn-st-dellpc-e:~# ifconfig |grep 14:fe
  em1 Link encap:Ethernet HWaddr 14:fe:b5:cc:74:da
  em2 Link encap:Ethernet HWaddr 14:fe:b5:cc:74:dc
  vhost0 Link encap:Ethernet HWaddr 14:fe:b5:cc:74:dc
  root@sdn-st-dellpc-e:~# ifconfig vhost0
  vhost0 Link encap:Ethernet HWaddr 14:fe:b5:cc:74:dc
            inet addr:192.14.1.10 Bcast:192.14.1.255 Mask:255.255.255.0
            inet6 addr: fe80::16fe:b5ff:fecc:74dc/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            RX packets:6690600 errors:0 dropped:196 overruns:0 frame:0
            TX packets:8191847 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:1381280714 (1.3 GB) TX bytes:9748573454 (9.7 GB)

  root@sdn-st-dellpc-e:/home/regress# ifconfig em2
  em2 Link encap:Ethernet HWaddr 14:fe:b5:cc:74:dc
            inet6 addr: fe80::16fe:b5ff:fecc:74dc/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
            RX packets:2989936882 errors:0 dropped:29942 overruns:0 frame:0
            TX packets:105124710 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000
            RX bytes:432178728813 (432.1 GB) TX bytes:20840328168 (20.8 GB)

  root@sdn-st-dellpc-c:/var/tmp# ifconfig |grep 14:fe
  em1 Link encap:Ethernet HWaddr 14:fe:b5:cc:73:f0
  em2 Link encap:Ethernet HWaddr 14:fe:b5:cc:73:f2
  vhost0 Link encap:Ethernet HWaddr 14:fe:b5:cc:73:f2
  root@sdn-st-dellpc-c:/var/tmp# ifconfig vhost0
  vhost0 Link encap:Ethernet HWaddr 14:fe:b5:cc:73:f2
            inet addr:192.11.1.10 Bcast:192.11.1.255 Mask:255.255.255....

[Sharon Zhang (sharonz)]

Can someone please let me know which build has the fix for this issue? I am blocked by this issue

I have upgraded to 93 build, and am seeing issues with the traffic between BMS and BMS, and Inter-VNI traffic between VM and MH BMS.

Thanks,

Sharon

[Sharon Zhang (sharonz)]

Hi,

I upgraded the contrail with build 95, still see this issue. This will cause traffic between two Intra-BMS not working

root@sdn-st-dellpc-b:/tmp# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.21-95 95
contrail-config 2.21-95 95
contrail-config-openstack 2.21-95 95
contrail-control 2.21-95 95
contrail-dns 2.21-95 95
contrail-f5 2.21-95 95
contrail-fabric-utils 2.21-95 95
contrail-heat 2.21-95 95
contrail-install-packages 2.21-95~icehouse 95
contrail-lib 2.21-95 95
contrail-nodemgr 2.21-95 95
contrail-nova-networkapi 2.21-95 95
contrail-openstack 2.21-95 95
contrail-openstack-analytics 2.21-95 95
contrail-openstack-config 2.21-95 95
contrail-openstack-control 2.21-95 95
contrail-openstack-dashboard 2.21-95 95
contrail-openstack-database 2.21-95 95
contrail-openstack-webui 2.21-95 95
contrail-setup 2.21-95 95
contrail-utils 2.21-95 95
contrail-web-controller 2.21-95 95
contrail-web-core 2.21-95 95
ifmap-python-client 0.1-2 95
ifmap-server 0.3.2-1contrail1 95
neutron-plugin-contrail 2.21-95 95
nova-api 1:2014.1.3-0ubuntu1~cloud0.3contrail95
nova-common 1:2014.1.3-0ubuntu1~cloud0.3contrail95
nova-conductor 1:2014.1.3-0ubuntu1~cloud0.3contrail95
nova-console 1:2014.1.3-0ubuntu1~cloud0.3contrail95
nova-consoleauth 1:2014.1.3-0ubuntu1~cloud0.3cont...

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13981
Submitter: Divakar Dharanalakota (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13982
Submitter: Divakar Dharanalakota (<email address hidden>)

[Divakar Dharanalakota (ddivakar)]

When BMSs are behind Mx and when there is MX redundancy in the network, if a BMS pings another BMS, the ARP cache of first BMS for second BMS is poisoned with Vrouter compute node's MAC, leading to connectivity failure between two BMS's.

Root cause:
When ARP request of BMS1 is flooded to a compute node by MX, Vrouter does the source IP lookup for BMS IP in Inet table. This lookup results in subnet route pointing to ECMP nexthop of two Mxs. This makes Vrouter respond with Vhost's MAC to force the packets to L3 processing though the ARP request is not meant for any VM's in that compute node.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/13981
Submitter: Divakar Dharanalakota (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/13981
Committed: http://github.org/Juniper/contrail-vrouter/commit/5f22a1425f7d5d4ba5005d4dc9f429ed772b263a
Submitter: Zuul
Branch: master

commit 5f22a1425f7d5d4ba5005d4dc9f429ed772b263a
Author: Divakar <email address hidden>
Date: Tue Sep 22 21:59:40 2015 +0530

Replyin to ARP request of ECMP source only if VM is hosted

When an ARP request is received on compute node on fabric interface from
an ECMP source, ARP response is sent with Vhost mac even though the ARP
request is not meant for any VM on that compute node. Because of this,
even if BMS pings another BMS, every compute node receiving this ARP
request is responding with Vhost mac leading to ARP cache poisoning in
BMS.

As a fix, only if ARP request is meant for a VM on compute node, the
response is sent with Vhost mac.

Change-Id: Iae8541c8404d6e6ce530f994b64b03dc0cd73170
closes-bug: #1491644

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17480
Submitter: Divakar Dharanalakota (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17481
Submitter: Divakar Dharanalakota (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/17482
Submitter: Divakar Dharanalakota (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17480
Committed: http://github.org/Juniper/contrail-vrouter/commit/806037a7bc25ad1f4dfff9e4713b414be62c38b2
Submitter: Zuul
Branch: R2.20.x

commit 806037a7bc25ad1f4dfff9e4713b414be62c38b2
Author: Divakar <email address hidden>
Date: Tue Sep 22 21:59:40 2015 +0530

Replyin to ARP request of ECMP source only if VM is hosted

When an ARP request is received on compute node on fabric interface from
an ECMP source, ARP response is sent with Vhost mac even though the ARP
request is not meant for any VM on that compute node. Because of this,
even if BMS pings another BMS, every compute node receiving this ARP
request is responding with Vhost mac leading to ARP cache poisoning in
BMS.

As a fix, only if ARP request is meant for a VM on compute node, the
response is sent with Vhost mac.

No source IP lookup for ARP requests from BMS

For the packets from VM to an ECMP destination we are forcing the
packets to be L3 routed. When ARP request comes for that VM from one of
the ECMP sources, though we have the stiching for VM's IP we give
Vhost's MAC to route the packets as packets need to be routed in this
direction as well. This functioanlity is added with the fix for the bug
1472796.
But the fix for the above bug should not handle the ARP request coming
from BMS (in TSN) as TSN is never a gateway for BMS. Such ARP request
should be flooded. So the fix is to not force the L3 if ARP request is
from BMS.

Change-Id: I4036dcd6eaf757b579de8ae391855aa7269a9ac1
closes-bug: #1485804
closes-bug: #1491644

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17482
Committed: http://github.org/Juniper/contrail-vrouter/commit/032fd49dac0308fb43f47b593321417d237222b9
Submitter: Zuul
Branch: R2.22.x

commit 032fd49dac0308fb43f47b593321417d237222b9
Author: Divakar <email address hidden>
Date: Tue Sep 22 21:59:40 2015 +0530

Replyin to ARP request of ECMP source only if VM is hosted

When an ARP request is received on compute node on fabric interface from
an ECMP source, ARP response is sent with Vhost mac even though the ARP
request is not meant for any VM on that compute node. Because of this,
even if BMS pings another BMS, every compute node receiving this ARP
request is responding with Vhost mac leading to ARP cache poisoning in
BMS.

As a fix, only if ARP request is meant for a VM on compute node, the
response is sent with Vhost mac.

Change-Id: Iae8541c8404d6e6ce530f994b64b03dc0cd73170
closes-bug: #1491644

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/17481
Committed: http://github.org/Juniper/contrail-vrouter/commit/737a135f6cd3fc2638ed4b5fe2039534a5d4f553
Submitter: Zuul
Branch: R2.20

commit 737a135f6cd3fc2638ed4b5fe2039534a5d4f553
Author: Divakar <email address hidden>
Date: Tue Sep 22 21:59:40 2015 +0530

Replyin to ARP request of ECMP source only if VM is hosted

When an ARP request is received on compute node on fabric interface from
an ECMP source, ARP response is sent with Vhost mac even though the ARP
request is not meant for any VM on that compute node. Because of this,
even if BMS pings another BMS, every compute node receiving this ARP
request is responding with Vhost mac leading to ARP cache poisoning in
BMS.

As a fix, only if ARP request is meant for a VM on compute node, the
response is sent with Vhost mac.

Change-Id: Iae8541c8404d6e6ce530f994b64b03dc0cd73170
closes-bug: #1491644