[RFE] Create optional bulk resource_extend

[Nate Johnston]

When performing bulk actions, one area that is still very much forced into the singleton method of processing is the resource_extend framework, where extensions can register to extend object data. This RFE is to propose the changes needed to make resource_extend work in a bulk context.

The details of the proposal are as follows:

1.) Add an optional additional argument "bulk" to @resource_extend.extends that would default to False. If it is True this signals that the function being decorated can support bulk extensions. Normally the arguments to such a function are in the form (resource, db_model); if it is operating in bulk mode then the first argument would be treated as [resource, model] in an array, and the second argument would be ignored. This can be tested for inside the function by checking if the first argument is an array.

2.) Create a new resource_extend.apply_funcs_bulk function that would act like the current resource_extend.apply_funcs but would take an array of object data instead of a single object resource and db model object. This new function would consult the "bulk" attribute on registered extender functions and would pass in the array of object data to a bulk function, but if the function was bulk=False then it would loop through as it does now.

3.) This would require a revamp of the _resource_extend_functions data structure in neutron-lib/db/resource_extends.py. Currently it is a dict with the resource type as a string key, and the value is an array of functions to be applied. This would need to be changed to take advantage of the bulk attribute. I am not sure how best to do that at this point.

One scenario where this would help is with bulk port creation, because resource extender functions could implement more efficient commits of their various data. Another example is a bug from when I was last bug deputy [1], where a user is complaining of slow responses due to large quantities of SQL queries when using large numbers of trunk ports. I think that with a bulk resource extend the trunk port extender could be implemented as qa single SQL query that fetches information for all affected ports, thus significantly increasing performance.

Please let me know what you think of this idea. I raised it as an RFE because it requires significant enough modification to an element of Neutron that is both foundational and that not many people may have dealt with directly that an open discussion of the merits and implementation would be a positive contribution.

[1] https://bugs.launchpad.net/neutron/+bug/1842150

[Miguel Lavalle (minsel)]

Looks like a good idea, aligned with improving performance overall. If we look at the profiling data we have gathered so far, we spend a lot of time accessing the DB. Would a PoC of this benefit from profiling the before and after situation? I'll be glad to help :-)

[Nate Johnston (nate-johnston)]

@mlavalle yes, I think it would be perfect to conduct such an experiment. Especially because with the design I set forth, it would be easy to go back and forth between resource_extend implementations by changing the call made from create_port_bulk. I'm sure there is a similar situation with the trunk port scenario, I just don't yet know the precise line of code like I do with bulk port. But we should absolutely do before/after testing, and your help with that - as with all things - is greatly appreciated.

[Slawek Kaplonski (slaweq)]

RFE is approved now.
As was agreed on drivers meeting, please work on some PoC patch first and than, together with Miguel try to profile it with his profiling tool when it will be ready.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/689945

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/689945
Reason: If needed, we can restore this patch and continue working on this.