[L2][scale] add a trunk size config option for bundle flow install

Bug #1906487 reported by LIU Yulong
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
In Progress
Medium
LIU Yulong

Bug Description

We recently tested the openvswitch-agent flow installation performance, we have following two results:

1. the AGENT_RES_PROCESSING_STEP is 100 (default)
We can see the 4296 OpenFlow rules are installed in 20s.

2020-12-02 16:01:32.843 792 INFO neutron.agent.common.ovs_lib [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] Processing 4296 OpenFlow rules.
2020-12-02 16:01:52.698 792 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-106cf38e-e20a-443e-b1ad-24ab1b7dc7cb - - - - -] process_network_ports - iteration:266 - agent port security group processed in 31.044

2020-12-02T08:01:32.303Z|00967|connmgr|INFO|br-int<->unix#440: 1 flow_mods in the last 0 s (1 adds)
2020-12-02T08:01:33.353Z|00968|connmgr|INFO|br-int<->unix#443: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:33.864Z|00969|connmgr|INFO|br-int<->unix#446: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:34.435Z|00970|connmgr|INFO|br-int<->unix#449: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:34.966Z|00971|connmgr|INFO|br-int<->unix#452: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:35.399Z|00972|connmgr|INFO|br-int<->unix#455: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:35.904Z|00973|connmgr|INFO|br-int<->unix#458: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:36.335Z|00974|connmgr|INFO|br-int<->unix#461: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:36.820Z|00975|connmgr|INFO|br-int<->unix#464: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:37.288Z|00976|connmgr|INFO|br-int<->unix#467: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:37.805Z|00977|connmgr|INFO|br-int<->unix#470: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:38.329Z|00978|connmgr|INFO|br-int<->unix#473: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:38.792Z|00979|connmgr|INFO|br-int<->unix#476: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:39.248Z|00980|connmgr|INFO|br-int<->unix#479: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:39.687Z|00981|connmgr|INFO|br-int<->unix#482: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:40.164Z|00982|connmgr|INFO|br-int<->unix#485: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:40.568Z|00983|connmgr|INFO|br-int<->unix#488: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:40.978Z|00984|connmgr|INFO|br-int<->unix#491: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:41.453Z|00985|connmgr|INFO|br-int<->unix#494: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:41.892Z|00986|connmgr|INFO|br-int<->unix#497: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:42.264Z|00987|connmgr|INFO|br-int<->unix#500: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:42.639Z|00988|connmgr|INFO|br-int<->unix#503: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:43.077Z|00989|connmgr|INFO|br-int<->unix#506: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:43.452Z|00990|connmgr|INFO|br-int<->unix#509: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:43.879Z|00991|connmgr|INFO|br-int<->unix#512: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:44.385Z|00992|connmgr|INFO|br-int<->unix#515: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:44.821Z|00993|connmgr|INFO|br-int<->unix#518: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:45.299Z|00994|connmgr|INFO|br-int<->unix#521: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:45.809Z|00995|connmgr|INFO|br-int<->unix#524: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:46.310Z|00996|connmgr|INFO|br-int<->unix#527: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:46.805Z|00997|connmgr|INFO|br-int<->unix#530: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:47.185Z|00998|connmgr|INFO|br-int<->unix#533: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:47.567Z|00999|connmgr|INFO|br-int<->unix#536: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:47.987Z|01000|connmgr|INFO|br-int<->unix#539: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:48.417Z|01001|connmgr|INFO|br-int<->unix#542: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:48.865Z|01002|connmgr|INFO|br-int<->unix#545: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:49.408Z|01003|connmgr|INFO|br-int<->unix#548: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:49.918Z|01004|connmgr|INFO|br-int<->unix#551: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:50.399Z|01005|connmgr|INFO|br-int<->unix#554: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:50.871Z|01006|connmgr|INFO|br-int<->unix#557: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:51.337Z|01007|connmgr|INFO|br-int<->unix#560: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:51.627Z|01008|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 16 flow_mods in the 7 s starting 27 s ago (12 adds, 4 deletes)
2020-12-02T08:01:51.771Z|01009|connmgr|INFO|br-int<->unix#563: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:52.285Z|01010|connmgr|INFO|br-int<->unix#566: 100 flow_mods in the last 0 s (100 adds)
2020-12-02T08:01:52.695Z|01011|connmgr|INFO|br-int<->unix#569: 96 flow_mods in the last 0 s (96 adds)

2. the AGENT_RES_PROCESSING_STEP is 1000 (changed)
12222 OpenFlow rules are installed in 6s.

2020-12-02 16:13:41.855 3090 INFO neutron.agent.common.ovs_lib [None req-0eef620d-613e-4485-ad2f-b9380c3fc381 - - - - -] Processing 12222 OpenFlow rules.
2020-12-02 16:13:47.846 3090 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-0eef620d-613e-4485-ad2f-b9380c3fc381 - - - - -] process_network_ports - iteration:0 - agent port security group processed in 202.523

2020-12-02T08:13:42.299Z|01310|connmgr|INFO|br-int<->unix#1298: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:42.718Z|01311|connmgr|INFO|br-int<->unix#1301: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:43.174Z|01312|connmgr|INFO|br-int<->unix#1304: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:43.698Z|01313|connmgr|INFO|br-int<->unix#1307: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:44.087Z|01314|connmgr|INFO|br-int<->unix#1310: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:44.554Z|01315|connmgr|INFO|br-int<->unix#1313: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:45.050Z|01316|connmgr|INFO|br-int<->unix#1316: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:45.534Z|01317|connmgr|INFO|br-int<->unix#1319: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:45.939Z|01318|connmgr|INFO|br-int<->unix#1322: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:46.388Z|01319|connmgr|INFO|br-int<->unix#1325: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:46.869Z|01320|connmgr|INFO|br-int<->unix#1328: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:47.323Z|01321|connmgr|INFO|br-int<->unix#1331: 1000 flow_mods in the last 0 s (1000 adds)
2020-12-02T08:13:47.834Z|01322|connmgr|INFO|br-int<->unix#1334: 222 flow_mods in the last 0 s (222 adds)

So we can say maybe we should increase the trunk (step) size for the ovs-ofctl bundle installaion. We do not want to add a fixed value because the vswitchd may be in high pressure according to your deployment, the flow install performace may not be as good as you expected. So we could add a config option for users, they can tune the size for their cloud based on the load of the agent, vswitchd and cpu usage and so on.

Tags: loadimpact ovs
LIU Yulong (dragon889)
description: updated
Revision history for this message
LIU Yulong (dragon889) wrote :
Changed in neutron:
status: New → In Progress
Changed in neutron:
importance: Undecided → Medium
tags: added: loadimpact ovs
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by "liuyulong <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/765072

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.