Flannel provides the networking at the container level for k8s and optionally for swarm. This is done by overlay: messages are encapsulated with a new header and sent over the underlying network. This introduces an overhead and potentially significant performance degradation.
In a k8s cluster as provisioned by Magnum, a performance study was done using a 10GB network and the following bandwidth was observed:
-Direct server to server (base scenario for comparison): 9.39 GBits/sec
-Direct VM to VM (base scenario for comparison): 7.74 GBits/sec
-Flannel with host-gw backend (between pods in different hosts): 6.0 GBits/sec
-Flannel with VxLAN backend (between pods in different hosts): 1.71 GBits/sec
-Flannel with UDP backend (between pods in different hosts): 0.385 GBits/sec
A large part of the degradation is due to the processing of the header.
In a shared public cloud, the bandwidth degradation may not be a big issue since there may not be a network performance guarantee to the user, and the actual network bandwidth is shared among the users.
However, there are scenarios where the bandwidth degradation is not acceptable, such as a private cloud, or a public cloud where a high performance network is offered.
This data was presented at a talk at the Tokyo Summit, 2015.
Angus Lees proposed the host-gw backend as an improvement. This works for our case because all the minions of the k8s cluster are on one Neutron network.
From the discussion on the patch, we will change the label flannel_use_vxlan to something like flannel_backend where the three options can be specified: udp, vxlan, host-gw. The default will be host-gw.