api slow due to missing LRP index

[Max]

With many routers/Logical_Router_Ports the neutron-api can get slow at scale.

When profiling the api we noticed that much time gets spend in functions like:

sync_routers_and_rports -> _is_router_port_changed
schedule_unhosted_gateways -> get_unhosted_gateways -> get_gateway_chassis_binding
schedule_unhosted_gateways -> get_gateway_chassis_az_hints

This happens due to LRP queries without an index:
  lrp = idlutils.row_by_value(self.idl, 'Logical_Router_Port', 'name', gateway_name)

After adding an Logical_Router_Port name index the API performance improves a lot.

[Max (maxlamprecht)]

https://review.opendev.org/c/openstack/ovsdbapp/+/938004

[Brian Haley (brian-haley)]

I assigned to Max since he proposed a patch to fix the issue.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/ovsdbapp/+/938004
Committed: https://opendev.org/openstack/ovsdbapp/commit/1cbe61683a0bf40a7894efaa4d069b96e8862344
Submitter: "Zuul (22348)"
Branch: master

commit 1cbe61683a0bf40a7894efaa4d069b96e8862344
Author: Max Lamprecht <email address hidden>
Date: Tue Dec 3 14:31:17 2024 +0100

    Add Northbound Logical_Router_Port name index

    This improves Logical_Router_Port lookup speeds at scale.

    Closes-Bug: #2092133
    Change-Id: I9cc8460244c030f83194736cbb71c3cf449cbf76

[Terry Wilson (otherwiseguy)]

This *shouldn't* be required/make any difference since Logical_Router_Port has:

    "indexes": [["name"]],

and ovsdbapp's autocreate_indices() should automatically create an index for it. I'd like to see some numbers of before/after the change just to make sure we don't have a bug where indices aren't being properly created.

[Max (maxlamprecht)]

Hi, actually we noticed the same (it was only better for a few days - probably due to the api restart)

We have seen that the index went missing again after X time (We saw it fur mutiple indices - not only LRPs)
We were able to observe this by our profiling and with patching multiple print statements in the lookup function.

You're right with the autocreate_indices function, but I think we have a bug here.

We discovered that right before we see the "missing indices" we called self.clear() in ovs [1]
It seems like it get called for a new full sync(comment: "clear table rows for new dump")

With "table.rows = custom_index.IndexedRows(table)" this clear() function reinitializes the complete table.rows object which also clears all indices by the ovsdbapp lib [2]

After this happend we observed again that the api worker was slower and we were not using the db indices.

I'm currently working on a patch
I think it's not the best way to patch ovs in order to keep the previous indices. This can break upgrades if there was an index on a column which got removed by upgrades or so.

Currently we are testing a after_clear hook in ovs that can be used by the ovsdbapp lib(execute the autocreate_indices function)

We are currently trying that out in our dev stage. If everything works well we will submit the patches later this week:)

[1] https://github.com/openvswitch/ovs/blob/8b7f1eb8db1aa99ccf7b542662129450caff65e0/python/ovs/db/idl.py#L495
[2]https://github.com/openstack/ovsdbapp/blob/14300779028fab5b1756dc35544d08ec0bb1f1b4/ovsdbapp/backend/ovs_idl/__init__.py#L81

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/ovsdbapp 2.10.1 Epoxy release.