vRouter does not send ARP request which destinatin IP address is set as Unicast MAC address

[Daisuke Nakajima]

10.0.0.1 and 10.0.0.2 are located different Compute node. Then 10.0.0.1 was pinging to 10.0.0.2. I captured some packet on both VMs. 10.0.0.1 was vm1_1.pcap, 10.0.0.2 was vm2_1.pcap.

10.0.0.1 sent ARP request which destination IP was set as Unicast MAC address, but 10.0.0.2 didn't receave those packets.
### vm1_1.pcap
"32","2016-03-09 14:59:27.648876","02:da:ec:d1:b6:86","02:9e:74:56:79:1d","ARP","42","Who has 10.0.0.2? Tell 10.0.0.1"
"35","2016-03-09 14:59:28.648867","02:da:ec:d1:b6:86","02:9e:74:56:79:1d","ARP","42","Who has 10.0.0.2? Tell 10.0.0.1"
"38","2016-03-09 14:59:29.648840","02:da:ec:d1:b6:86","02:9e:74:56:79:1d","ARP","42","Who has 10.0.0.2? Tell 10.0.0.1"

Instead of ARP request packets above, 10.0.0.2 shows ARP reply below. I don't know why 10.0.0.2 sent them without any trigger.
### vm2_1.pcap
"25","2016-03-09 14:59:27.997158","02:9e:74:56:79:1d","02:da:ec:d1:b6:86","ARP","42","10.0.0.2 is at 02:9e:74:56:79:1d"
"28","2016-03-09 14:59:28.997091","02:9e:74:56:79:1d","02:da:ec:d1:b6:86","ARP","42","10.0.0.2 is at 02:9e:74:56:79:1d"
"31","2016-03-09 14:59:29.997098","02:9e:74:56:79:1d","02:da:ec:d1:b6:86","ARP","42","10.0.0.2 is at 02:9e:74:56:79:1d"

However, No25,28,31 packets didn't see in vm1_1.pcap.

[Divakar Dharanalakota (ddivakar)]

We are hitting the bug:#1550632 while processing arp request in the Destination compute node. 1550632 is fixed so we should not have any more issues further. In the builds where 1550632 is not fixed, GRO has to be disabled for correct ARP functionality to appear. GRO can be disabled like this

echo 0 > /proc/sys/net/vrouter/perfr

Root cause:
In the destination compute node, after the bridging, when the ARP request is being handled in L2EncapNh of destination VM, packet is handed over for GRO processing. Post GRO completion, due to a bug in GRO handling, the ingress interface is no more treated at Fabric interface but dest VM's Tap interface. Because of this the ARP response packet is sent to destination VM rather sent as tunneled packet to source VM. Disabling the GRO sends the ARP response packet to source VM. Fix for 1550632 ensures that the ingress interface is retained and hence ARP response travels to right source VM.

-Divakar

[Daisuke Nakajima (dnakajima)]

1550632 was merged at R2.21 build 30, but this issue has been seen on my setup.
I did fping to other VM located on other compute node and confirmed ARP packet that set Destination MAC address was unicast MAC was not correctly processed on vRouter.
See vm2_ping.pcap. VM sent three unicast ARP packets (#97,100,103), but VM didn't get their reply.
After that, VM sent Broadcast ARP packets, then VM got its reply.

root@contrail-07:~# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-fabric-utils 2.21.2-30 30
contrail-install-packages 2.21.2-30~juno 30
contrail-lib 2.21.2-30 30
contrail-nodemgr 2.21.2-30 30
contrail-nova-vif 2.21.2-30 30
contrail-openstack-vrouter 2.21.2-30 30
contrail-setup 2.21.2-30 30
contrail-utils 2.21.2-30 30
contrail-vrouter-3.13.0-40-generic 2.21.2-30 30
contrail-vrouter-agent 2.21.2-30 30
contrail-vrouter-common 2.21.2-30 30
contrail-vrouter-init 2.21.2-30 30
contrail-vrouter-utils 2.21.2-30 30
nova-common 1:2014.2.3-0ubuntu1~cloud0.3contrail30
nova-compute 1:2014.2.3-0ubuntu1~cloud0.3contrail30
nova-compute-kvm 1:2014.2.3-0ubuntu1~cloud0.3contrail30
nova-compute-libvirt 1:2014.2.3-0ubuntu1~cloud0.3contrail30
python-contrail 2.21.2-30 30
python-contrail-vrouter-api 2.21.2-30 30
python-neutronclient 1:2.3.8-0ubuntu1~cloud0.2contrail30
python-nova 1:2014.2.3-0ubuntu1~cloud0.3contrail30
python-opencontrail-vrouter-netns 2.21.2-30 30