neutron DVR poor performance

[Matthias Backhausen]

Scenario:
2 VMs of same tenant but in different subnets talk to each other. The traffic flow is ...

Traffic VM1 to VM2:
========= CN1 ==================== ======== CN2 ===========
VM1--->br-int--->Router--->br-int--->br-tun---------->br-tun--->br-int--->VM2

Traffic VM2 to VM1:
========= CN2 ==================== ======== CN1 ===========
VM2--->br-int--->Router--->br-int---br-tun----------->br-tun--->br-int--->VM1

This works as designed; however obviously br-int of CN1 never gets traffic from Router of CN1 (except the very first ARP response), same for br-int of CN2. This might lead to flow (or mac?) timeout on br-int after 300 secs and degrades performance massively because traffic is flooded.

Changing the mac-addr aging timer influences the issue; change to 30 (default 300) and the issue occurs after 30 seconds (instead 300)
#ovs-vsctl set bridge br-int other_config:mac-aging-time=30

[Armando Migliaccio (armando-migliaccio)]

Can you share your neutron configuration files? Traffic should not flood.

[Matthias Backhausen (matthias-backhausen)]

Added dvr related config in file dvr.txt

[Swaminathan Vasudevan (swaminathan-vasudevan)]

How did you measure the performance? any input would help us.

[Lynn (lynn-li)]

Tried to reproduce the performance issue.. I ran netperf and tcpdump. I don't see any packets dropped from network node. Attached result file.

If any performance measurement details/steps can be attached, it would be a big help.

[Matthias Backhausen (matthias-backhausen)]

Hi Lynn,
will provide my test results by end of the week.
Thanks,
Matthias

Mit TouchDown von meinem Android-Telefon gesendet (www.nitrodesk.com)

-----Original Message-----
From: Lynn [<email address hidden>]
Received: Dienstag, 14 Juli 2015, 0:10
To: Backhausen, Matthias [<email address hidden>]
Subject: [Bug 1463831] Re: neutron DVR poor performance

Tried to reproduce the performance issue.. I ran netperf and tcpdump. I
don't see any packets dropped from network node. Attached result file.

If any performance measurement details/steps can be attached, it would
be a big help.

** Attachment added: "netPerf_tcpdump_results.txt"
   https://bugs.launchpad.net/neutron/+bug/1463831/+attachment/4428436/+files/netPerf_tcpdump_results.txt

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1463831

Title:
  neutron DVR poor performance

Status in neutron:
  Incomplete

Bug description:
  Scenario:
  2 VMs of same tenant but in different subnets talk to each other. The traffic flow is ...

  Traffic VM1 to VM2:
  ========= CN1 ==================== ======== CN2 ===========
  VM1--->br-int--->Router--->br-int--->br-tun---------->br-tun--->br-int--->VM2

  Traffic VM2 to VM1:
  ========= CN2 ==================== ======== CN1 ===========
  VM2--->br-int--->Router--->br-int---br-tun----------->br-tun--->br-int--->VM1

  This works as designed; however obviously br-int of CN1 never gets
  traffic from Router of CN1 (except the very first ARP response), same
  for br-int of CN2. This might lead to flow (or mac?) timeout on br-int
  after 300 secs and degrades performance massively because traffic is
  flooded.

  Changing the mac-addr aging timer influences the issue; change to 30 (default 300) and the issue occurs after 30 seconds (instead 300)
  #ovs-vsctl set bridge br-int other_config:mac-aging-time=30

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1463831/+subscriptions

[Matthias Backhausen (matthias-backhausen)]

Hi,

we could reproduce the issue that br-int floods packets out all ports in some cases. There are no drops, but the flodding has negative impact to performance. Therefore to ease the issue, as a first step it might be useful to focus on the flooding instead of measuring performance.

Routing between two VMs (of same tenant) is asymmetric, as described previously.
The command #ovs-appctl fdb/show br-int clearly shows age of mac-address decreases until it ages-out,
after that ovs starts to flood packets. SSH connection seems to counteract the problem, so dont do ssh to the VM.
Also if the VM starts new connection with arp, the router responds and fills up mac-addr table of br-int.
So my advice is to just do a very basic test, with no ssh and just single connection between 2 VMs.

Hope that helps,
Matthias

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Lynn
Sent: Tuesday, July 14, 2015 12:01 AM
To: Backhausen, Matthias
Subject: [Bug 1463831] Re: neutron DVR poor performance

Tried to reproduce the performance issue.. I ran netperf and tcpdump. I don't see any packets dropped from network node. Attached result file.

If any performance measurement details/steps can be attached, it would be a big help.

** Attachment added: "netPerf_tcpdump_results.txt"
   https://bugs.launchpad.net/neutron/+bug/1463831/+attachment/4428436/+files/netPerf_tcpdump_results.txt

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1463831

Title:
  neutron DVR poor performance

Status in neutron:
  Incomplete

Bug description:
  Scenario:
  2 VMs of same tenant but in different subnets talk to each other. The traffic flow is ...

  Traffic VM1 to VM2:
  ========= CN1 ==================== ======== CN2 ===========
  VM1--->br-int--->Router--->br-int--->br-tun---------->br-tun--->br-int--->VM2

  Traffic VM2 to VM1:
  ========= CN2 ==================== ======== CN1 ===========
  VM2--->br-int--->Router--->br-int---br-tun----------->br-tun--->br-int--->VM1

  This works as designed; however obviously br-int of CN1 never gets
  traffic from Router of CN1 (except the very first ARP response), same
  for br-int of CN2. This might lead to flow (or mac?) timeout on br-int
  after 300 secs and degrades performance massively because traffic is
  flooded.

  Changing the mac-addr aging timer influences the issue; change to 30 (default 300) and the issue occurs after 30 seconds (instead 300)
  #ovs-vsctl set bridge br-int other_config:mac-aging-time=30

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1463831/+subscriptions

[Adolfo Duarte (adolfo-duarte)]

 I believe i understand what Mathias is saying; and he is correct to a degree.
If the mac address of the router in the br-int gets aged out and the br-int never sees traffic from the router (its mac address) it will flood as is expected by regular switching.

However, this should NOT affect a dvr router *because* an openflow rule should be entered in table 1 of br-int which points all traffic destined to the mac address of the router to the port the router is connected on:
something like:
dl_src=router-mac, output: port_router_attached_to.

that rule should prevent flooding of any packets destined to the router mac address entering the br-int switch from the vms.

So if you are seeing flooding of packets destined for the mac-address of the router that rule is being removed by something, or it has been made invalid.

I just looked in my setup which has current master branch and I don't see that rule. So there might be a problem here. I'll look more and update, but meantime, Mathias, if you see the problem gain do check the openflow rules of hte br-int bridge:
ovs-ofctl dump-flows br-int, and make sure you have a rule for the router mac address.

[Adolfo Duarte (adolfo-duarte)]

Update.
Testing shows the behavior described here is caused by the router staying "quite" for the mac age time. which is roughly around 3 minutes.
In other words, the reason the br-int bridge starts flooding packets destined for the router mac address is because the router does not send any ehternet frames out the port which is connected to the br-int for a period of around 3 minutes. (mac aging time).
Although this is a very unlikely scenario, it does present a very small posibility that the system will get in that state.
Even in control tests, this scenario is quite difficult to reproduce.

There exists two possible solutions:
One: an openflow rule can be configured on the br-int with the mac address of the router mac pointing to the correct port.

Two: the router can be configured to make sure it does not stay quite longer than the mac aging time. For example a gratuitous arp packet can be sent periodically.

The prefer method would be the latter as the first one would required lots of book keeping (adding and removing flows), while the second method uses the built in learning of openvswitch.

Either one of the solutions will require some work. And it is not clear if the needed changes to the system are warranted by the impact of this behavior. The likelyhood of a router staying "quite" for the mac-aging time is not very high.

[Adolfo Duarte (adolfo-duarte)]

I would suggest making the priority of this defect low

[Matthias Backhausen (matthias-backhausen)]

Hello Adolfo,

thanks for the update! I think this is *not* unlikely situation that router is quiet.

The local dvr router will never send traffic to the local vm, except arp-reply. This is due to the asymmetric routing concept of dvr.

Please correct me if I'm wrong.

We also haven't seen a openflow rule in br-int fixing this issue.

[Adolfo Duarte (adolfo-duarte)]

From the testing I've done and observations of working systems, what you describe above is correct only in a vacum.
the local dvr router does not only have to not send traffic to the local vm. It has to stay completely quite, as in no traffic of any sort. The moment any frame comes out of the router, for any reason the mac address of the router will be repopulated in the br-int.

[Adolfo Duarte (adolfo-duarte)]

(con'td). As it has been stated above in thise defect report, it is actually very dificult to reproduce because the sistuation has to be artificially created so the router does not create any traffic at all.
The question is not really weather or not this can happen. It is how easy is the trigger to reproduce. ip traffic is always simplex. It goes from ip a to ip b. There is no gurantee anywhere that the traffic will take the same path back in the case of a tcp flow. Which means I can actually reproduce this behavior in a regualr vanilla network. For example, with a pc connected to two routers, one is the outgoing gateway, and the other is the incoming gateway. In that case there could be a situation where you can make the physical switch behave in the exact manner that br-int is doing here.
But those situations usually don't arise because there is a plethora of other traffic type (not necessarily ip but other type of ehternet traffic) which causes the mac address to be populated in the mac-table of the switch.

The same applies here, the router might not be talking to the vm, but it might be talking to something else.

But your point is valid. if this situation is quite common, and the trigger occurs quite often, then perhaps it is a good idea to try to mitigate this behavior.
But even in this case, the vm itself usually will send an arp request because its arp table has a time limite as well.
So you see, I am not saying this is not possible, I am just questioning if how oftent it happens, and if there is already not a solution built into the protocols being used (tcp, icmp, ethernet, etc..) which might alivitate this problem already.

[Armando Migliaccio (armando-migliaccio)]

This bug is > 180 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]