Changing vr_flow_entries above 512K causes flow timeout

[Jon Hickman]

Venu was running test where he was setting the vr_flow_entries to 4M and was running a PPS test with 1600 flows active but the performance was not as good as it was when we ran Contrail with vr_flow_entries at 512K.

After looking at it we found that the Agilio vRouter offload was not updating all the flow statistics when the number of flow entries set to anything > 512K. This caused the flows to timeout and the extra work of adding and removing flows during the PPS test was noticeable.

So we checked our Agilio vRouter requirement for number of flows supported and it says 1M flows. Now we are not doing 1M flows at the moment and that needs to be fixed but as we will explain further adding flow capability comes at a cost so we should be judicious in our decisions.

Lets explain a little about the number of flow entries effects the way that must maintain flow statistics and how that plays into overhead burden even when flows are not active.

The flow statistics table entries must be as large as the vr_flow_entries+flow table overflow limit.

We have a internal flow statistics table that is kept for packet and byte counters associated with each flow. To keep the agent from timing out flows while packets are offloaded we must send to agent all the flow counter for every entry at a rate that faster than the shortest timeout value of 1sec. So right now we attempt to send back the entire flow stats at 3x per second to assure a 1 sec timeout still will work.

So the default size of the table at 512K flow entries + 103 overflow entries is 615K worth of entries. (this is as large as we support with current 154 release)

Increasing that to a table size of 1024K flow entries + 200 overflow entries is 1224K worth of entries.

Making it increase to a table size of 4096K flow entries + 800 overflow entries is 4896K worth of entries.

As you can see the larger the number of flow entries supported in the offload, the more overhead is taken for updating the agent with flow counters. This happens whether or not the 1 or 4M flows are active. Our sizing is presently static so we would live with the largest number allowed for operation with Agilio vRouter then factor in the rate at which the updates safely would need sent to agent to maintain a 1 sec timeout.

Another option that can play with this is if we are not required to maintain the 1 sec timeout then the update rate could be reduced.

I have not done the math yet on how how much of an impact it would have on fallback traffic to maintain 4M flow entries yet.

This also has ramifications to mirroring because to offload mirroring we need to keep per flow entry the mirroring meta data.

So there should probably be a discussion with Raja on:
1. What is the requirement for Agilio vRouter max flow entries.
2. Can the update rate be reduced via supporting a longer minimum timeout value.
3. Discussion about how this impacts mirroring meta data.

[Jon Hickman (jhickman8x3)]

We have done some testing and we are able to handle a flow entry setting of 4M and still update the flow statistics fast enough to keep the flows from timing out.

We still have an issue with being able to handle the Mirror Meta data with 4M entries. There is not enough memory on the SmartNIC to support 4M 540B mirror entries. That comes out to 2.7G and we only have 2G on the card.

Raja suggested that we use a mirror id to index the Mirror metadata vs using flowid. This would allow us to cap Mirrors to say 512K which we are able to fit.

So we are still investigating that aspect and will update.

[Jon Hickman (jhickman8x3)]

We have modified our flow entries to safely run with 512K flow entries.

These changes were included in Contrail 3.1.0.0 build 177, we are still discussing a long term solution.

[Jon Hickman (jhickman8x3)]

As we increase vr_flow_entries support with a higher value the problem becomes that we are not able to fit the Mirroring metadata table in without re-writing the mechanism of accessing the entries and controlling to a maximum TBD value of entries. Presently Mirroring metadata is referenced via the flow id so tables are 1 for 1.

Option 1:
Settings:

    Set flow stats entries on NFP to 4096K + 800K (4896K) total.
    Leave Mirroring metatdata at 615K entries.
    Leave flow stats update rate same frequency.

Limitations:

    Mirroring would only be functional with vr_flow_entries set > 512K.
    Flow timeout for 4096K vr_flow_entries would increase to 8 seconds minimum vs 1 second.
    Possible collision issues with our internal flow hash when number of entries > 1024K

Option 2:
Settings:

    Set flow stats entries on NFP to 1024K + 200K (1224K) total.
    Set Mirroring metatdata at 1224K entries.
    Leave flow stats update rate same frequency.
    Lower NFP internal flow hash table size from 4M to 3M.

Limitations:

    If vr_flow_entries > 1024K would result in premature flow timeouts.
    Mirroring would only be functional with vr_flow_entries set > 1024K.
    Flow timeout for 1024K vr_flow_entries would increase to 4 seconds minimum vs 1 second.
    Possibility of more NFP flow hash collisions cause by decreasing from 4M to 3M. We had sized to 4M to match a 1024K vr_flow_entry setting on host.

Option 3:

    Set flow stats entries on NFP to 4096K + 800K (4896K) total.
    Redo the Mirroring metatdata access and put a TBD maximum number supported.

Limitations:

    Mirroring entries supported would be TBD value and not tied to flow entries.
    Possible collision issues with our internal flow hash when number of entries > 1024K

NOTE: the Mirroring portion of this is only for "flow" mirroring. Port mirroring does not have an issue because it does not use the Mirroring metadata.