Juniper Openstack

Bug #1594165
Comment #2

Comment 2 for bug 1594165

Revision history for this message

Hari Prasad Killi (haripk) wrote on 2016-06-22: Re: [Bug 1594165] [NEW] vhost0 replies for BMS ARP requests ( STALE MAC case)

+ Manish

On 6/22/16, 3:25 AM, "Chhandak Mukherjee" <email address hidden> wrote:

>Hi,
>
>I have reproduced the issue in my setup.
>
>Problem Description
>--------------------
>When BMS is sending unicast ARP request for VM, ARP is coming directly to the respective compute instead of TSN, compute is responding with vhost0 mac in case of dual MX. So BMS is having incorrect ARP for VM in arp table.
>
>Details:
>---------
>ARP, Request who-has 1.1.1.4 (02:58:de:2e:8e:8c) tell 1.1.1.5, length 42 >>> VM MAC is 02:58:de:2e:8e:8c
>14:43:11.877203 IP 172.17.90.8.58568 > 34.34.34.34.4789: VXLAN, flags [I] (0x08), vni 1111
>ARP, Reply 1.1.1.4 is-at 90:e2:ba:a7:23:80, length 42 >>>>> ARP is responding with 90:e2:ba:a7:23:80
>
>root@5b7s8:~# ifconfig vhost0
>vhost0 Link encap:Ethernet HWaddr 90:e2:ba:a7:23:80
> inet addr:172.17.90.8 Bcast:172.17.90.255 Mask:255.255.255.0
> inet6 addr: fe80::92e2:baff:fea7:2380/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:9809980 errors:0 dropped:63 overruns:0 frame:0
> TX packets:8235068 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:1994559085 (1.9 GB) TX bytes:10342201367 (10.3 GB)
>
>
>
>
>root@5b7s8:~# vxlan --dump
>VXLAN Table
>
> VNID NextHop
>----------------
> 1111 12
> 2222 24
>
>root@5b7s8:~# nh --get 12
>Id:12 Type:Vrf_Translate Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:1
> Flags:Valid, Vxlan,
> Vrf:1
>
>
>root@5b7s8:~# rt --dump 1 --family inet| grep 1.1.1.4
>1.1.1.4/32 32 P - 22 2:58:de:2e:8e:8c(266196)
>
>root@5b7s8:~# nh --get 22
>Id:22 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:1
> Flags:Valid, Policy,
> EncapFmly:0806 Oif:3 Len:14
> Encap Data: 02 58 de 2e 8e 8c 00 00 5e 00 01 00 08 00
>
>Setup
>------
>Respective Compute : 10.87.121.75 (root/c0ntrail123)
>
>
>#Management ip addresses of hosts in the cluster
>host1 ='root@10.87.121.68'
>host2 ='root@10.87.121.69’
>host3 ='root@10.87.121.70'
>host4 ='root@10.87.121.71'
>host5 ='root@10.87.121.72’
>host6 ='root@10.87.121.73'
>host7 ='root@10.87.121.74'
>host8 ='root@10.87.121.75'
>host9 ='root@10.87.121.76'
>host10 ='root@10.87.121.77’
>
>#Role definition of the hosts.
>env.roledefs = {
> 'all': [host1,host2,host3,host4,host5,host6,host7,host8,host9,host10],
> 'cfgm': [host1,host2,host3],
> 'openstack': [host1],
> 'webui': [host1],
> 'control': [host2,host3],
> 'compute': [host4,host5,host6,host7,host8,host9,host10],
> 'tsn': [host4,host5,host6,host7],
> 'toragent': [host4,host5,host6,host7],
> 'collector': [host2,host3],
> 'database': [host1,host2,host3],
> 'build': [host_build],
>}
>
>env.hostnames = {
> 'all': ['5b7s1','5b7s2','5b7s3','5b7s4','5b7s5','5b7s6','5b7s7','5b7s8','5b7s9','5b7s10']
>}
>
>
>Thanks and Regards,
>
>Chhandak
>
>
>
>
>
>
>On 6/19/16, 1:11 PM, "<email address hidden> on behalf of Robert Rosiak" <<email address hidden> on behalf of <email address hidden>> wrote:
>
>>Public bug reported:
>>
>>Case:
>>2 MX routers with virtual-gateway-address statement.
>>When there's no ARP entry in cache then BMS is sending a broacact ARP request, and it gets a proper MAC response from TSN.
>>However in case of STALE MAC addres in arp cache, BMS is sending a unicast ARP request with MAC value that was learned earlier.
>>For this ARP request the BMS is getting ARP reply from vhost0 MAC, and BMS is learing a wrong MAC address.
>>
>>
>>Here's the tcpdump on BMS
>>[root@BMS ~]# tcpdump -e -n -i bond0.1500 arp
>>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>listening on bond0.1500, link-type EN10MB (Ethernet), capture size 65535 bytes
>>01:38:13.257002 1c:c1:de:e7:b4:70 > 02:2e:ee:16:58:fd, ethertype ARP (0x0806), length 42: Request who-has 11.0.1.111 tell 11.0.1.200, length 28
>>01:38:13.257215 e4:1f:13:7a:6a:7a > 1c:c1:de:e7:b4:70, ethertype ARP (0x0806), length 56: Reply 11.0.1.111 is-at e4:1f:13:7a:6a:7a, length 42
>>01:38:14.258962 1c:c1:de:e7:b4:70 > 02:2e:ee:16:58:fd, ethertype ARP (0x0806), length 42: Request who-has 11.0.1.111 tell 11.0.1.200, length 28
>>01:38:14.259253 e4:1f:13:7a:6a:7a > 1c:c1:de:e7:b4:70, ethertype ARP (0x0806), length 56: Reply 11.0.1.111 is-at e4:1f:13:7a:6a:7a, length 42
>>
>>02:2e:ee:16:58:fd (11.0.1.111) is VM MAC on compute1
>>1c:c1:de:e7:b4:70 (11.0.1.200)is BMS MAC address on VLAN1500
>>e4:1f:13:7a:6a:7a is the vhost0 on compute1
>>
>>
>>root@compute-kvm-1:~# ifconfig vhost0
>>vhost0 Link encap:Ethernet HWaddr e4:1f:13:7a:6a:7a
>> inet addr:10.10.1.13 Bcast:10.10.1.255 Mask:255.255.255.0
>>
>>A workaround is to set on BMS a
>>echo 0 > /proc/sys/net/ipv4/neigh/bond0.1600/ucast_solicit
>>so that BMS will never send a unicast MAC request.
>>
>>This issue is simmilar to this one: https://bugs.launchpad.net/juniperopenstack/r2.20/+bug/1485804
>>however fix was only for broadcast BMS MAC request.
>>
>>** Affects: juniperopenstack
>> Importance: Undecided
>> Status: New
>>
>>
>>** Tags: bms vrouter
>>
>>** Information type changed from Proprietary to Public
>>
>>--
>>You received this bug notification because you are a member of Contrail
>>Systems engineering, which is subscribed to Juniper Openstack.
>>https://bugs.launchpad.net/bugs/1594165
>>
>>Title:
>> vhost0 replies for BMS ARP requests ( STALE MAC case)
>>
>>Status in Juniper Openstack:
>> New
>>
>>Bug description:
>> Case:
>> 2 MX routers with virtual-gateway-address statement.
>> When there's no ARP entry in cache then BMS is sending a broacact ARP request, and it gets a proper MAC response from TSN.
>> However in case of STALE MAC addres in arp cache, BMS is sending a unicast ARP request with MAC value that was learned earlier.
>> For this ARP request the BMS is getting ARP reply from vhost0 MAC, and BMS is learing a wrong MAC address.
>>
>>
>> Here's the tcpdump on BMS
>> [root@BMS ~]# tcpdump -e -n -i bond0.1500 arp
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> listening on bond0.1500, link-type EN10MB (Ethernet), capture size 65535 bytes
>> 01:38:13.257002 1c:c1:de:e7:b4:70 > 02:2e:ee:16:58:fd, ethertype ARP (0x0806), length 42: Request who-has 11.0.1.111 tell 11.0.1.200, length 28
>> 01:38:13.257215 e4:1f:13:7a:6a:7a > 1c:c1:de:e7:b4:70, ethertype ARP (0x0806), length 56: Reply 11.0.1.111 is-at e4:1f:13:7a:6a:7a, length 42
>> 01:38:14.258962 1c:c1:de:e7:b4:70 > 02:2e:ee:16:58:fd, ethertype ARP (0x0806), length 42: Request who-has 11.0.1.111 tell 11.0.1.200, length 28
>> 01:38:14.259253 e4:1f:13:7a:6a:7a > 1c:c1:de:e7:b4:70, ethertype ARP (0x0806), length 56: Reply 11.0.1.111 is-at e4:1f:13:7a:6a:7a, length 42
>>
>> 02:2e:ee:16:58:fd (11.0.1.111) is VM MAC on compute1
>> 1c:c1:de:e7:b4:70 (11.0.1.200)is BMS MAC address on VLAN1500
>> e4:1f:13:7a:6a:7a is the vhost0 on compute1
>>
>>
>> root@compute-kvm-1:~# ifconfig vhost0
>> vhost0 Link encap:Ethernet HWaddr e4:1f:13:7a:6a:7a
>> inet addr:10.10.1.13 Bcast:10.10.1.255 Mask:255.255.255.0
>>
>> A workaround is to set on BMS a
>> echo 0 > /proc/sys/net/ipv4/neigh/bond0.1600/ucast_solicit
>> so that BMS will never send a unicast MAC request.
>>
>> This issue is simmilar to this one: https://bugs.launchpad.net/juniperopenstack/r2.20/+bug/1485804
>> however fix was only for broadcast BMS MAC request.
>>
>>To manage notifications about this bug go to:
>>https://bugs.launchpad.net/juniperopenstack/+bug/1594165/+subscriptions

+ Manish

On 6/22/16, 3:25 AM, "Chhandak Mukherjee" <chhandak@juniper.net> wrote:

>Hi,
>
>I have reproduced the issue in my setup.
>
>Problem Description 
>--------------------
>When BMS is sending unicast ARP request for VM, ARP is coming directly to the respective compute instead of TSN, compute is responding with vhost0 mac in case of dual MX. So BMS is having incorrect ARP for VM in arp table. 
>
>Details: 
>--------- 
>ARP, Request who-has 1.1.1.4 (02:58:de:2e:8e:8c) tell 1.1.1.5, length 42 >>> VM MAC is 02:58:de:2e:8e:8c 
>14:43:11.877203 IP 172.17.90.8.58568 > 34.34.34.34.4789: VXLAN, flags [I] (0x08), vni 1111
>ARP, Reply 1.1.1.4 is-at 90:e2:ba:a7:23:80, length 42 >>>>> ARP is responding with 90:e2:ba:a7:23:80
>
>root@5b7s8:~# ifconfig  vhost0
>vhost0    Link encap:Ethernet  HWaddr 90:e2:ba:a7:23:80  
>          inet addr:172.17.90.8  Bcast:172.17.90.255  Mask:255.255.255.0
>          inet6 addr: fe80::92e2:baff:fea7:2380/64 Scope:Link
>          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>          RX packets:9809980 errors:0 dropped:63 overruns:0 frame:0
>          TX packets:8235068 errors:0 dropped:0 overruns:0 carrier:0
>          collisions:0 txqueuelen:1000 
>          RX bytes:1994559085 (1.9 GB)  TX bytes:10342201367 (10.3 GB)
>
>
>
>
>root@5b7s8:~# vxlan --dump 
>VXLAN Table
>
> VNID    NextHop
>----------------
>   1111    12
>   2222    24
>
>root@5b7s8:~# nh --get 12
>Id:12         Type:Vrf_Translate  Fmly: AF_INET  Rid:0  Ref_cnt:2          Vrf:1
>              Flags:Valid, Vxlan, 
>              Vrf:1
>
>
>root@5b7s8:~# rt --dump 1 --family inet| grep 1.1.1.4
>1.1.1.4/32             32            P          -             22        2:58:de:2e:8e:8c(266196)
>
>root@5b7s8:~# nh --get 22
>Id:22         Type:Encap          Fmly: AF_INET  Rid:0  Ref_cnt:5          Vrf:1
>              Flags:Valid, Policy, 
>              EncapFmly:0806 Oif:3 Len:14
>              Encap Data: 02 58 de 2e 8e 8c 00 00 5e 00 01 00 08 00 
>
>Setup
>------
>Respective Compute : 10.87.121.75 (root/c0ntrail123)
>
>
>#Management ip addresses of hosts in the cluster
>host1 ='root@10.87.121.68'
>host2 ='root@10.87.121.69’
>host3 ='root@10.87.121.70'
>host4 ='root@10.87.121.71'
>host5 ='root@10.87.121.72’
>host6 ='root@10.87.121.73'
>host7 ='root@10.87.121.74'
>host8 ='root@10.87.121.75'
>host9 ='root@10.87.121.76'
>host10 ='root@10.87.121.77’
>
>#Role definition of the hosts.
>env.roledefs = {
>    'all': [host1,host2,host3,host4,host5,host6,host7,host8,host9,host10],
>    'cfgm': [host1,host2,host3],
>    'openstack': [host1],
>    'webui': [host1],
>    'control': [host2,host3],
>    'compute': [host4,host5,host6,host7,host8,host9,host10],
>    'tsn': [host4,host5,host6,host7],
>    'toragent': [host4,host5,host6,host7],
>    'collector': [host2,host3],
>    'database': [host1,host2,host3],
>    'build': [host_build],
>}
>
>env.hostnames = {
>    'all': ['5b7s1','5b7s2','5b7s3','5b7s4','5b7s5','5b7s6','5b7s7','5b7s8','5b7s9','5b7s10']
>}
>
>
>Thanks and Regards,
>
>Chhandak 
>
>
>
>
>
>
>On 6/19/16, 1:11 PM, "bounces@canonical.com on behalf of Robert Rosiak" <bounces@canonical.com on behalf of 1594165@bugs.launchpad.net> wrote:
>
>>Public bug reported:
>>
>>Case:
>>2 MX routers with virtual-gateway-address statement.
>>When there's no ARP entry in cache then BMS is sending a broacact ARP request, and it gets a proper MAC response from TSN.
>>However in case of STALE MAC addres in arp cache, BMS is sending a unicast ARP request with MAC value that was learned earlier.
>>For this ARP request the BMS is getting ARP reply from vhost0 MAC, and BMS is learing a wrong MAC address.
>>
>>
>>Here's the tcpdump on BMS
>>[root@BMS ~]# tcpdump -e -n -i bond0.1500 arp
>>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>listening on bond0.1500, link-type EN10MB (Ethernet), capture size 65535 bytes
>>01:38:13.257002 1c:c1:de:e7:b4:70 > 02:2e:ee:16:58:fd, ethertype ARP (0x0806), length 42: Request who-has 11.0.1.111 tell 11.0.1.200, length 28
>>01:38:13.257215 e4:1f:13:7a:6a:7a > 1c:c1:de:e7:b4:70, ethertype ARP (0x0806), length 56: Reply 11.0.1.111 is-at e4:1f:13:7a:6a:7a, length 42
>>01:38:14.258962 1c:c1:de:e7:b4:70 > 02:2e:ee:16:58:fd, ethertype ARP (0x0806), length 42: Request who-has 11.0.1.111 tell 11.0.1.200, length 28
>>01:38:14.259253 e4:1f:13:7a:6a:7a > 1c:c1:de:e7:b4:70, ethertype ARP (0x0806), length 56: Reply 11.0.1.111 is-at e4:1f:13:7a:6a:7a, length 42
>>
>>02:2e:ee:16:58:fd (11.0.1.111) is VM MAC on compute1
>>1c:c1:de:e7:b4:70 (11.0.1.200)is BMS MAC address on VLAN1500
>>e4:1f:13:7a:6a:7a is the vhost0 on compute1
>>
>>
>>root@compute-kvm-1:~# ifconfig vhost0
>>vhost0 Link encap:Ethernet HWaddr e4:1f:13:7a:6a:7a
>>          inet addr:10.10.1.13 Bcast:10.10.1.255 Mask:255.255.255.0
>>
>>A workaround is to set on BMS a 
>>echo 0 > /proc/sys/net/ipv4/neigh/bond0.1600/ucast_solicit
>>so that BMS will never send a unicast MAC request.
>>
>>This issue is simmilar to this one: https://bugs.launchpad.net/juniperopenstack/r2.20/+bug/1485804
>>however fix was only for broadcast BMS MAC request.
>>
>>** Affects: juniperopenstack
>>     Importance: Undecided
>>         Status: New
>>
>>
>>** Tags: bms vrouter
>>
>>** Information type changed from Proprietary to Public
>>
>>-- 
>>You received this bug notification because you are a member of Contrail
>>Systems engineering, which is subscribed to Juniper Openstack.
>>https://bugs.launchpad.net/bugs/1594165
>>
>>Title:
>>  vhost0 replies for BMS ARP requests ( STALE MAC case)
>>
>>Status in Juniper Openstack:
>>  New
>>
>>Bug description:
>>  Case:
>>  2 MX routers with virtual-gateway-address statement.
>>  When there's no ARP entry in cache then BMS is sending a broacact ARP request, and it gets a proper MAC response from TSN.
>>  However in case of STALE MAC addres in arp cache, BMS is sending a unicast ARP request with MAC value that was learned earlier.
>>  For this ARP request the BMS is getting ARP reply from vhost0 MAC, and BMS is learing a wrong MAC address.
>>
>>  
>>  Here's the tcpdump on BMS
>>  [root@BMS ~]# tcpdump -e -n -i bond0.1500 arp
>>  tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>  listening on bond0.1500, link-type EN10MB (Ethernet), capture size 65535 bytes
>>  01:38:13.257002 1c:c1:de:e7:b4:70 > 02:2e:ee:16:58:fd, ethertype ARP (0x0806), length 42: Request who-has 11.0.1.111 tell 11.0.1.200, length 28
>>  01:38:13.257215 e4:1f:13:7a:6a:7a > 1c:c1:de:e7:b4:70, ethertype ARP (0x0806), length 56: Reply 11.0.1.111 is-at e4:1f:13:7a:6a:7a, length 42
>>  01:38:14.258962 1c:c1:de:e7:b4:70 > 02:2e:ee:16:58:fd, ethertype ARP (0x0806), length 42: Request who-has 11.0.1.111 tell 11.0.1.200, length 28
>>  01:38:14.259253 e4:1f:13:7a:6a:7a > 1c:c1:de:e7:b4:70, ethertype ARP (0x0806), length 56: Reply 11.0.1.111 is-at e4:1f:13:7a:6a:7a, length 42
>>
>>  02:2e:ee:16:58:fd (11.0.1.111) is VM MAC on compute1
>>  1c:c1:de:e7:b4:70 (11.0.1.200)is BMS MAC address on VLAN1500
>>  e4:1f:13:7a:6a:7a is the vhost0 on compute1
>>
>>  
>>  root@compute-kvm-1:~# ifconfig vhost0
>>  vhost0 Link encap:Ethernet HWaddr e4:1f:13:7a:6a:7a
>>            inet addr:10.10.1.13 Bcast:10.10.1.255 Mask:255.255.255.0
>>
>>  A workaround is to set on BMS a 
>>  echo 0 > /proc/sys/net/ipv4/neigh/bond0.1600/ucast_solicit
>>  so that BMS will never send a unicast MAC request.
>>
>>  This issue is simmilar to this one: https://bugs.launchpad.net/juniperopenstack/r2.20/+bug/1485804
>>  however fix was only for broadcast BMS MAC request.
>>
>>To manage notifications about this bug go to:
>>https://bugs.launchpad.net/juniperopenstack/+bug/1594165/+subscriptions