Juniper Openstack

vhost0 replies for BMS ARP requests ( STALE MAC case)

  1. Series r2.22.x
  2. Bug #1594165
Bug #1594165 reported by Robert Rosiak
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
Critical
Manish Singh
Juniper Openstack r2.23 "r2.23"
R2.22.x
Fix Committed
Critical
Manish Singh
Juniper Openstack r2.22.3
R3.0
Fix Committed
Critical
Manish Singh
Juniper Openstack r3.0.3.0
R3.0.2.x
Fix Committed
Critical
Manish Singh
Juniper Openstack r3.0.2.1
R3.1
Fix Committed
Critical
Manish Singh
Juniper Openstack r3.1.1.0
Trunk
Fix Committed
Critical
Manish Singh
Juniper Openstack r3.2.0.0-fcs "r3.2.0.0"

Bug Description

Case:
2 MX routers with virtual-gateway-address statement.
When there's no ARP entry in cache then BMS is sending a broacact ARP request, and it gets a proper MAC response from TSN.
However in case of STALE MAC addres in arp cache, BMS is sending a unicast ARP request with MAC value that was learned earlier.
For this ARP request the BMS is getting ARP reply from vhost0 MAC, and BMS is learing a wrong MAC address.

Here's the tcpdump on BMS
[root@BMS ~]# tcpdump -e -n -i bond0.1500 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0.1500, link-type EN10MB (Ethernet), capture size 65535 bytes
01:38:13.257002 1c:c1:de:e7:b4:70 > 02:2e:ee:16:58:fd, ethertype ARP (0x0806), length 42: Request who-has 11.0.1.111 tell 11.0.1.200, length 28
01:38:13.257215 e4:1f:13:7a:6a:7a > 1c:c1:de:e7:b4:70, ethertype ARP (0x0806), length 56: Reply 11.0.1.111 is-at e4:1f:13:7a:6a:7a, length 42
01:38:14.258962 1c:c1:de:e7:b4:70 > 02:2e:ee:16:58:fd, ethertype ARP (0x0806), length 42: Request who-has 11.0.1.111 tell 11.0.1.200, length 28
01:38:14.259253 e4:1f:13:7a:6a:7a > 1c:c1:de:e7:b4:70, ethertype ARP (0x0806), length 56: Reply 11.0.1.111 is-at e4:1f:13:7a:6a:7a, length 42

02:2e:ee:16:58:fd (11.0.1.111) is VM MAC on compute1
1c:c1:de:e7:b4:70 (11.0.1.200)is BMS MAC address on VLAN1500
e4:1f:13:7a:6a:7a is the vhost0 on compute1

root@compute-kvm-1:~# ifconfig vhost0
vhost0 Link encap:Ethernet HWaddr e4:1f:13:7a:6a:7a
          inet addr:10.10.1.13 Bcast:10.10.1.255 Mask:255.255.255.0

A workaround is to set on BMS a
echo 0 > /proc/sys/net/ipv4/neigh/bond0.1600/ucast_solicit
so that BMS will never send a unicast MAC request.

This issue is simmilar to this one: https://bugs.launchpad.net/juniperopenstack/r2.20/+bug/1485804
however fix was only for broadcast BMS MAC request.

Robert Rosiak (robert-rosiak)
information type: Proprietary → Public
Revision history for this message
chhandak (chhandak) wrote : Re: [Bug 1594165] [NEW] vhost0 replies for BMS ARP requests ( STALE MAC case)
Download full text (7.4 KiB)

Hi,

I have reproduced the issue in my setup.

Problem Description
--------------------
When BMS is sending unicast ARP request for VM, ARP is coming directly to the respective compute instead of TSN, compute is responding with vhost0 mac in case of dual MX. So BMS is having incorrect ARP for VM in arp table.

Details:
---------
ARP, Request who-has 1.1.1.4 (02:58:de:2e:8e:8c) tell 1.1.1.5, length 42 >>> VM MAC is 02:58:de:2e:8e:8c
14:43:11.877203 IP 172.17.90.8.58568 > 34.34.34.34.4789: VXLAN, flags [I] (0x08), vni 1111
ARP, Reply 1.1.1.4 is-at 90:e2:ba:a7:23:80, length 42 >>>>> ARP is responding with 90:e2:ba:a7:23:80

root@5b7s8:~# ifconfig vhost0
vhost0 Link encap:Ethernet HWaddr 90:e2:ba:a7:23:80
          inet addr:172.17.90.8 Bcast:172.17.90.255 Mask:255.255.255.0
          inet6 addr: fe80::92e2:baff:fea7:2380/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:9809980 errors:0 dropped:63 overruns:0 frame:0
          TX packets:8235068 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1994559085 (1.9 GB) TX bytes:10342201367 (10.3 GB)

root@5b7s8:~# vxlan --dump
VXLAN Table

 VNID NextHop
----------------
   1111 12
   2222 24

root@5b7s8:~# nh --get 12
Id:12 Type:Vrf_Translate Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1
              Flags:Valid, Vxlan,
              Vrf:1

root@5b7s8:~# rt --dump 1 --family inet| grep 1.1.1.4
1.1.1.4/32 32 P - 22 2:58:de:2e:8e:8c(266196)

root@5b7s8:~# nh --get 22
Id:22 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:1
              Flags:Valid, Policy,
              EncapFmly:0806 Oif:3 Len:14
              Encap Data: 02 58 de 2e 8e 8c 00 00 5e 00 01 00 08 00

Setup
------
Respective Compute : 10.87.121.75 (root/c0ntrail123)

#Management ip addresses of hosts in the cluster
host1 ='root@10.87.121.68'
host2 ='root@10.87.121.69’
host3 ='root@10.87.121.70'
host4 ='root@10.87.121.71'
host5 ='root@10.87.121.72’
host6 ='root@10.87.121.73'
host7 ='root@10.87.121.74'
host8 ='root@10.87.121.75'
host9 ='root@10.87.121.76'
host10 ='root@10.87.121.77’

#Role definition of the hosts.
env.roledefs = {
    'all': [host1,host2,host3,host4,host5,host6,host7,host8,host9,host10],
    'cfgm': [host1,host2,host3],
    'openstack': [host1],
    'webui': [host1],
    'control': [host2,host3],
    'compute': [host4,host5,host6,host7,host8,host9,host10],
    'tsn': [host4,host5,host6,host7],
    'toragent': [host4,host5,host6,host7],
    'collector': [host2,host3],
    'database': [host1,host2,host3],
    'build': [host_build],
}

env.hostnames = {
    'all': ['5b7s1','5b7s2','5b7s3','5b7s4','5b7s5','5b7s6','5b7s7','5b7s8','5b7s9','5b7s10']
}

Thanks and Regards,

Chhandak

On 6/19/16, 1:11 PM, "<email address hidden> on behalf of Robert Rosiak" <<email address hidden> on behalf of <email address hidden>> wrote:

>Public bug reported:
>
>Case:
>2 MX routers with virtual-gateway-address statement.
>When ...

Read more...

Revision history for this message
Hari Prasad Killi (haripk) wrote :
Download full text (7.7 KiB)

+ Manish

On 6/22/16, 3:25 AM, "Chhandak Mukherjee" <email address hidden> wrote:

>Hi,
>
>I have reproduced the issue in my setup.
>
>Problem Description
>--------------------
>When BMS is sending unicast ARP request for VM, ARP is coming directly to the respective compute instead of TSN, compute is responding with vhost0 mac in case of dual MX. So BMS is having incorrect ARP for VM in arp table.
>
>Details:
>---------
>ARP, Request who-has 1.1.1.4 (02:58:de:2e:8e:8c) tell 1.1.1.5, length 42 >>> VM MAC is 02:58:de:2e:8e:8c
>14:43:11.877203 IP 172.17.90.8.58568 > 34.34.34.34.4789: VXLAN, flags [I] (0x08), vni 1111
>ARP, Reply 1.1.1.4 is-at 90:e2:ba:a7:23:80, length 42 >>>>> ARP is responding with 90:e2:ba:a7:23:80
>
>root@5b7s8:~# ifconfig vhost0
>vhost0 Link encap:Ethernet HWaddr 90:e2:ba:a7:23:80
> inet addr:172.17.90.8 Bcast:172.17.90.255 Mask:255.255.255.0
> inet6 addr: fe80::92e2:baff:fea7:2380/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:9809980 errors:0 dropped:63 overruns:0 frame:0
> TX packets:8235068 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:1994559085 (1.9 GB) TX bytes:10342201367 (10.3 GB)
>
>
>
>
>root@5b7s8:~# vxlan --dump
>VXLAN Table
>
> VNID NextHop
>----------------
> 1111 12
> 2222 24
>
>root@5b7s8:~# nh --get 12
>Id:12 Type:Vrf_Translate Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1
> Flags:Valid, Vxlan,
> Vrf:1
>
>
>root@5b7s8:~# rt --dump 1 --family inet| grep 1.1.1.4
>1.1.1.4/32 32 P - 22 2:58:de:2e:8e:8c(266196)
>
>root@5b7s8:~# nh --get 22
>Id:22 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:1
> Flags:Valid, Policy,
> EncapFmly:0806 Oif:3 Len:14
> Encap Data: 02 58 de 2e 8e 8c 00 00 5e 00 01 00 08 00
>
>Setup
>------
>Respective Compute : 10.87.121.75 (root/c0ntrail123)
>
>
>#Management ip addresses of hosts in the cluster
>host1 ='root@10.87.121.68'
>host2 ='root@10.87.121.69’
>host3 ='root@10.87.121.70'
>host4 ='root@10.87.121.71'
>host5 ='root@10.87.121.72’
>host6 ='root@10.87.121.73'
>host7 ='root@10.87.121.74'
>host8 ='root@10.87.121.75'
>host9 ='root@10.87.121.76'
>host10 ='root@10.87.121.77’
>
>#Role definition of the hosts.
>env.roledefs = {
> 'all': [host1,host2,host3,host4,host5,host6,host7,host8,host9,host10],
> 'cfgm': [host1,host2,host3],
> 'openstack': [host1],
> 'webui': [host1],
> 'control': [host2,host3],
> 'compute': [host4,host5,host6,host7,host8,host9,host10],
> 'tsn': [host4,host5,host6,host7],
> 'toragent': [host4,host5,host6,host7],
> 'collector': [host2,host3],
> 'database': [host1,host2,host3],
> 'build': [host_build],
>}
>
>env.hostnames = {
> 'all': ['5b7s1','5b7s2','5b7s3','5b7s4','5b7s5','5b7s6','5b7s7','5b7s8','5b7s9','5b7s10']
>}
>
>
>Thanks and Regards,
>
>Chhandak
>
>
>
>
>
>
>On 6/19/16, 1:11 PM, "<email address hidden> on be...

Read more...

Revision history for this message
Manish Singh (manishs) wrote :

Before analyzing the issue brief explanation on why flood mac based arp works.
Arp request with flood mac is destined to TSN because OVS switch has TSN as multicast service node. TSN on receiving the same checks its broadcast tree and finds the source to be OVS switch. Since it knows that packet is from baremetal, output of source IP lookup done as part of route processing is not used and vrouter replies with stitched mac.

Now when unicast ARP is sent, OVS switch directly sends it to compute which is not TSN. On this compute source IP lookup is done. Since baremetal source IP is not seen by contrail(as it is never published via OVS), best prefix match is subnet published by MX. The nexthop of same is ECMP as it is dual MX setup. On seeing NH as ECMP, vrouter assumes that packet has to be routed from this source and replies with vhost mac.

Currently there is no fix for this issue.

Revision history for this message
Umamaheshwar (urao) wrote :

Customer is hitting this issue.

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/23702
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/23770
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/23770
Committed: http://github.org/Juniper/contrail-vrouter/commit/ed686d02d34b1f7793be674b99cf2a127d09704d
Submitter: Zuul
Branch: master

commit ed686d02d34b1f7793be674b99cf2a127d09704d
Author: Divakar <email address hidden>
Date: Mon Aug 29 22:32:00 2016 +0530

Respond with stitched MAC for unicast ARP request

When an unicast ARP request is received on fabric interface of a compute
node, from BMS behind QFX, if MX is in ecmp, the source IP lookup might
point to subnet route pointing to Ecmp nexthop. This is because there
would not be any host route for BMS in inet table. This results in Vrouter
responding with Vhost mac address though the destination IP address is
stitched. As such this behaviour is to ensure that Routing is forced if
the packet is from Ecmp source, though destination is in same subnet.
But this behaviour creates issues to BMS behind QFX, if BMS refreshes
ARP with unicast ARP request.

As a fix, the multicast ARP requests from Ecmp source on fabric
interface are dropped. If unicast ARP requests and if the destination
mac address of the ethernet packet is stitched mac, the ARP reply is
sent with stitched mac. If destination mac address does not match with
stitched mac, that ARP request is not processed by Vrouter.

closes-bug: #1594165

Change-Id: I72f4c44329b14c589343b30b84181cc8e0e05e81

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/23859
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/23860
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0.2.x

Review in progress for https://review.opencontrail.org/23861
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/23862
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/23863
Submitter: Divakar Dharanalakota (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/23859
Committed: http://github.org/Juniper/contrail-vrouter/commit/fa7c681675f49ab6c42951c00b347d0b510fcac4
Submitter: Zuul
Branch: R3.1

commit fa7c681675f49ab6c42951c00b347d0b510fcac4
Author: Divakar <email address hidden>
Date: Mon Aug 29 22:32:00 2016 +0530

Respond with stitched MAC for unicast ARP request

When an unicast ARP request is received on fabric interface of a compute
node, from BMS behind QFX, if MX is in ecmp, the source IP lookup might
point to subnet route pointing to Ecmp nexthop. This is because there
would not be any host route for BMS in inet table. This results in Vrouter
responding with Vhost mac address though the destination IP address is
stitched. As such this behaviour is to ensure that Routing is forced if
the packet is from Ecmp source, though destination is in same subnet.
But this behaviour creates issues to BMS behind QFX, if BMS refreshes
ARP with unicast ARP request.

As a fix, the multicast ARP requests from Ecmp source on fabric
interface are dropped. If unicast ARP requests and if the destination
mac address of the ethernet packet is stitched mac, the ARP reply is
sent with stitched mac. If destination mac address does not match with
stitched mac, that ARP request is not processed by Vrouter.

Change-Id: I58b91e410ecee809264d8914f75816cfb584ce17
closes-bug: #1594165

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23860
Committed: http://github.org/Juniper/contrail-vrouter/commit/7cc01225bb6650b4955b4d27aee63d3e62a49442
Submitter: Zuul
Branch: R3.0

commit 7cc01225bb6650b4955b4d27aee63d3e62a49442
Author: Divakar <email address hidden>
Date: Mon Aug 29 22:32:00 2016 +0530

Respond with stitched MAC for unicast ARP request

When an unicast ARP request is received on fabric interface of a compute
node, from BMS behind QFX, if MX is in ecmp, the source IP lookup might
point to subnet route pointing to Ecmp nexthop. This is because there
would not be any host route for BMS in inet table. This results in Vrouter
responding with Vhost mac address though the destination IP address is
stitched. As such this behaviour is to ensure that Routing is forced if
the packet is from Ecmp source, though destination is in same subnet.
But this behaviour creates issues to BMS behind QFX, if BMS refreshes
ARP with unicast ARP request.

As a fix, the multicast ARP requests from Ecmp source on fabric
interface are dropped. If unicast ARP requests and if the destination
mac address of the ethernet packet is stitched mac, the ARP reply is
sent with stitched mac. If destination mac address does not match with
stitched mac, that ARP request is not processed by Vrouter.

closes-bug: #1594165

Change-Id: I72f4c44329b14c589343b30b84181cc8e0e05e81

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23863
Committed: http://github.org/Juniper/contrail-vrouter/commit/fe90f0ae9adca50f6626eb0cee20270a0903cadf
Submitter: Zuul
Branch: R2.20

commit fe90f0ae9adca50f6626eb0cee20270a0903cadf
Author: Divakar <email address hidden>
Date: Mon Aug 29 22:32:00 2016 +0530

Respond with stitched MAC for unicast ARP request

When an unicast ARP request is received on fabric interface of a compute
node, from BMS behind QFX, if MX is in ecmp, the source IP lookup might
point to subnet route pointing to Ecmp nexthop. This is because there
would not be any host route for BMS in inet table. This results in Vrouter
responding with Vhost mac address though the destination IP address is
stitched. As such this behaviour is to ensure that Routing is forced if
the packet is from Ecmp source, though destination is in same subnet.
But this behaviour creates issues to BMS behind QFX, if BMS refreshes
ARP with unicast ARP request.

As a fix, the multicast ARP requests from Ecmp source on fabric
interface are dropped. If unicast ARP requests and if the destination
mac address of the ethernet packet is stitched mac, the ARP reply is
sent with stitched mac. If destination mac address does not match with
stitched mac, that ARP request is not processed by Vrouter.

closes-bug: #1594165

Change-Id: I72f4c44329b14c589343b30b84181cc8e0e05e81

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23861
Committed: http://github.org/Juniper/contrail-vrouter/commit/3e3cf5bdf4255fd6aec03d830fc7b3b9653eef7c
Submitter: Zuul
Branch: R3.0.2.x

commit 3e3cf5bdf4255fd6aec03d830fc7b3b9653eef7c
Author: Divakar <email address hidden>
Date: Mon Aug 29 22:32:00 2016 +0530

Respond with stitched MAC for unicast ARP request

When an unicast ARP request is received on fabric interface of a compute
node, from BMS behind QFX, if MX is in ecmp, the source IP lookup might
point to subnet route pointing to Ecmp nexthop. This is because there
would not be any host route for BMS in inet table. This results in Vrouter
responding with Vhost mac address though the destination IP address is
stitched. As such this behaviour is to ensure that Routing is forced if
the packet is from Ecmp source, though destination is in same subnet.
But this behaviour creates issues to BMS behind QFX, if BMS refreshes
ARP with unicast ARP request.

As a fix, the multicast ARP requests from Ecmp source on fabric
interface are dropped. If unicast ARP requests and if the destination
mac address of the ethernet packet is stitched mac, the ARP reply is
sent with stitched mac. If destination mac address does not match with
stitched mac, that ARP request is not processed by Vrouter.

closes-bug: #1594165

Change-Id: I72f4c44329b14c589343b30b84181cc8e0e05e81

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/23862
Committed: http://github.org/Juniper/contrail-vrouter/commit/5b9798ecf9664407ebb51e2ae6fbb8a77c37d530
Submitter: Zuul
Branch: R2.22.x

commit 5b9798ecf9664407ebb51e2ae6fbb8a77c37d530
Author: Divakar <email address hidden>
Date: Mon Aug 29 22:32:00 2016 +0530

Respond with stitched MAC for unicast ARP request

When an unicast ARP request is received on fabric interface of a compute
node, from BMS behind QFX, if MX is in ecmp, the source IP lookup might
point to subnet route pointing to Ecmp nexthop. This is because there
would not be any host route for BMS in inet table. This results in Vrouter
responding with Vhost mac address though the destination IP address is
stitched. As such this behaviour is to ensure that Routing is forced if
the packet is from Ecmp source, though destination is in same subnet.
But this behaviour creates issues to BMS behind QFX, if BMS refreshes
ARP with unicast ARP request.

As a fix, the multicast ARP requests from Ecmp source on fabric
interface are dropped. If unicast ARP requests and if the destination
mac address of the ethernet packet is stitched mac, the ARP reply is
sent with stitched mac. If destination mac address does not match with
stitched mac, that ARP request is not processed by Vrouter.

closes-bug: #1594165

Change-Id: I72f4c44329b14c589343b30b84181cc8e0e05e81

Umamaheshwar (urao)
tags: added: sscc-contrail
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.