Comment 2 for bug 1465000

I spoke with Jay Vosburgh about this issue, as we've started touching on aspects of networking deep inside the kernel. When testing this, there are a few additional points to consider:

(1) "sudo ip neigh flush all" (or the equivalent) must be executed whenever a change to the sysctl values or /etc/network/interfaces occurs. This must be done on any nodes under test, the MAAS server, and the gateway. (otherwise, the systems under test may behave inconsistently until ARP times out.)[1]

(2) The reason this works in a similar situation for NetworkManager (wired and wireless on the same subnet) is because a metric (preference) value is added to the per-interface route to the connected subnet. This eliminates any ambiguity in the route table, and also causes ARP to only respond on the interface with the lowest metric value, thereby forcing all traffic through a single interface.[2]

(3) Another potentially useful sysctl in this situation is "arp_ignore".[3] If the "arp_ignore" sysctl is set to 1, this will cause ARP to only respond on a particular interface if the address is actually configured there. (it seems to me that this sysctl and arp_filter would achieve the same result; one does so by implicit filtering, the other is explicit.)

[1] Normally when a host has multiple network interfaces on a single subnet, the reason is for redundancy (failover). In a failover situation, a userland process (or, in the case of a bond interface, the kernel) is often responsible for managing the failover by moving an IP address/MAC to the new desired interface. (if the MAC is moved, and/or the interface is brought down and back up again, the kernel could also send the gratuitous ARP if the "arp_notify" sysctl is set.) After the gratuitous ARPs are sent, the neighbor tables of the other hosts on the subnet will be updated. (Obviously, ifupdown doesn't do this for us.)

[2] note that one thing that NetworkManager does NOT do is send gratuitous ARPs when route changes happen. So you may have to wait until remote hosts expire their neighbor cache and re-ARP for in-flight TCP sessions, etc, to recover. (since rp_filter=1 by default)

[3] arp_ignore - INTEGER
 Define different modes for sending replies in response to
 received ARP requests that resolve local target IP addresses:
 0 - (default): reply for any local target IP address, configured
 on any interface
 1 - reply only if the target IP address is local address
 configured on the incoming interface
 2 - reply only if the target IP address is local address
 configured on the incoming interface and both with the
 sender's IP address are part from same subnet on this interface
 3 - do not reply for local addresses configured with scope host,
 only resolutions for global and link addresses are replied
 4-7 - reserved
 8 - do not reply for all local addresses

 The max value from conf/{all,interface}/arp_ignore is used
 when ARP request is received on the {interface}