vRouter vhost0 responds on ARP with MAC address of VM

[Jakub Pavlik]

DHCP – instances did not received IP. CentOS and Windows machine only. It is caused by bad respond of vRouter MAC address of VM. VM dhcpclient reports that IP address is already used, so it declines offer.

It is OpenContrail build 7 days ago from branch R2.20 (github.com/tcpcloud/contrail-controller, vrouter, etc.)

Bug Reproduction
I created network named "test", then launch instance with name "test-instance" with CentOS7. Instance get IP trough DHCP immediately. Then I add Route Target (64512:30) to network "test", restart network service in instance and do not get IP. Logs from instance:

DHCPDECLINE on eth0 to 255.255.255.255 port 67 (xid=0x5fd443e3)
oct 23 10:57:06 test-instance dhclient[1273]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 3 (xid=0x62acb0d7)
oct 23 10:57:06 test-instance dhclient[1273]: DHCPREQUEST on eth0 to 255.255.255.255 port 67 (xid=0x62acb0d7)
oct 23 10:57:06 test-instance dhclient[1273]: DHCPOFFER from 10.0.152.34
oct 23 10:57:06 test-instance dhclient[1273]: DHCPACK from 10.0.152.34 (xid=0x1f39dfee)
oct 23 10:57:06 test-instance dhclient[1273]: DHCPDECLINE on eth0 to 255.255.255.255 port 67 (xid=0x62acb0d7)

tcpdump from tap interface on hypervisor:

11:54:43.587996 IP (tos 0x0, ttl 64, id 0, offset 0, flags [none], proto UDP (17), length 318)
    10.0.152.34.67 > 10.0.152.36.68: [no cksum] BOOTP/DHCP, Reply, length 290, xid 0x5e3de243, Flags [none] (0x0000)
          Your-IP 10.0.152.36
          Server-IP 10.0.152.34
          Client-Ethernet-Address 02:ef:d4:b4:c4:61
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: ACK
            Server-ID Option 54, length 4: 10.0.152.34
            Lease-Time Option 51, length 4: 4294967295
            Subnet-Mask Option 1, length 4: 255.255.255.224
            BR Option 28, length 4: 10.0.152.63
            Default-Gateway Option 3, length 4: 10.0.152.33
            Hostname Option 12, length 8: "net-test"
            Domain-Name-Server Option 6, length 4: 10.0.152.34
11:54:43.614120 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.152.36 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 28
11:54:43.614134 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.0.152.36 is-at 00:00:5e:00:01:00, length 28
11:54:43.615516 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.0.152.36 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 28
11:54:43.615524 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.0.152.36 is-at 00:00:5e:00:01:00, length 28
11:54:43.621239 IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from 02:ef:d4:b4:c4:61, length 300, xid 0x5e3de243, Flags [none] (0x0000)
          Client-Ethernet-Address 02:ef:d4:b4:c4:61
          Vendor-rfc1048 Extensions
            Magic Cookie 0x63825363
            DHCP-Message Option 53, length 1: Decline
            Server-ID Option 54, length 4: 10.0.152.34
            Requested-IP Option 50, length 4: 10.0.152.36
            Hostname Option 12, length 8: "net-test"

I see that vRouter (00:00:5e:00:01:00) reply that it has required IP address, which is apparentaly wrong.
This behavior is seen just when route target is added in network. Immediately after I removed route target, everything starts work smoothly.

[Jakub Pavlik (pavlk-jakub)]

Workaround on VM side e.g. CentOS is to modify /usr/sbin/dhclient-script to disable arping

        if arping -D -q -c2 -I "${interface}" "${new_ip_address}"; then
            dhconfig
            exit_with_hooks 0
        else # DAD failed, i.e. address is already in use
            dhconfig
            exit_with_hooks 0
# ARP_REPLY=$(arping -D -c2 -I "${interface}" "${new_ip_address}" | grep reply | awk '{print toupper($5)}' | #cut -d "[" -f2 | cut -d "]" -f1)
# OUR_MACS=$(ip link show | grep link | awk '{print toupper($2)}' | uniq)
# if [[ "${OUR_MACS}" = *"${ARP_REPLY}"* ]]; then
# # the reply can come from our system, that's OK (#1116004#c33)
# dhconfig
# exit_with_hooks 0
            else
                exit_with_hooks 1
            fi
        fi

[Edgar Magana (emagana)]

Thanks @Jakub. This is worth to give it a try.

[Hari Prasad Killi (haripk)]

A fix for this is submitted as part of https://bugs.launchpad.net/juniperopenstack/+bug/1513793

[Jakub Pavlik (pavlk-jakub)]

@Hari I cannot open this bug.

[amit surana (asurana-t)]

This bug is seen In topologies where the Contrail control node is peering with 2 gateway routers (such as MX), and each gateway router advertises a default route into the tenant VRF/VN. The tenant VN, if it has a RT configured, would import routes from both the gateway routers and as a result will have an ECMP default route. The dhclient script on the CentOS VM, after obtaining an IP via DHCP, runs a duplicate address detection procedure wherein it does an arping with the sender IP zero’d out. The vRouter upon receiving this arping, does a reverse route lookup (on sender IP of 0.0.0.0), matches the default route, and because its an ECMP route, returns the VRRP MAC. This in-turn caused the dhclient in the VM to think that a dup IP exists and so it declined the offered IP address.

The fix changes the vRouter behavior by having it not respond to such ARP probes.