Comment 1 for bug 1329029

This is a bug... it seems to happen after bringing interfaces up and
down multiple times... it's an ubuntu bug I think.

I would be all for binding, actually. There is an issue with this
though... Can you guarantee that traffic is going out and coming back
in the same port this way?

The kernel routes using the path of least resistance... so you may
send a packet out eth4, but when it comes back, it could well be eth0
that accepts the return packet in.

Here's an example using 2 nics on a default 14.04 install:

first, this is about where we are:
ubuntu@supermicro:~$ netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 2864 0 0 0 583 0
0 0 BMRU
eth1 1500 0 58 0 0 0 2035 0
0 0 BMRU
lo 65536 0 0 0 0 0 0 0
0 0 LRU

Note eth1 shows 2035 packets out and eth0 shows 2864 comig in.

Now a ping flood of 10000 packets:
sudo ping -I eth1 -f -c 10000 10.0.0.1
PING 10.0.0.1 (10.0.0.1) from 10.0.0.128 eth1: 56(84) bytes of data.

--- 10.0.0.1 ping statistics ---
10000 packets transmitted, 10000 received, 0% packet loss, time 298ms
rtt min/avg/max/mdev = 0.210/0.258/0.497/0.024 ms, ipg/ewma 0.295/0.260 ms

and recheck netstat
ubuntu@supermicro:~$ netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 12897 0 0 0 600 0
0 0 BMRU
eth1 1500 0 60 0 0 0 12035 0
0 0 BMRU
lo 65536 0 0 0 0 0 0 0
0 0 LRU

Notice that the outgoing 10K packets were on eth1, but the incoming
were on eth0.

One way to fix this would be to restart networking between each NIC...
 that seems to restore the default route properly.

There is SUPPOSED to be a kernel parameter to fix this routing
problem, but I can't remember what it is and it's been forever and a
day since I last had to set it, so I've long since forgotten.

but there is something you can set in /proc/sys that changes the
kernel's routing behaviour.