Slow deletion of high number of static routes on router with ML2/OVN

[Ihtisham ul Haq]

With ML2/OVN if one create about 1000 of static routes on a router, the addition takes <5 seconds but deletion of those static routes takes about >90 seconds.

That is because neutron sends a separate deletion command to OVN for each of the static route, which is very inefficient, instead of doing it in a single transaction.

Neutron version: latest/main branch
Deployment tool: devstack

Steps to reproduce:

$ openstack network create static_route_test
$ subnet create static_route_test --subnet-range 10.0.0.0/20 --network static_route_test
$ openstack router create static_route_test
$ openstack route add subnet static_route_test static_route_test

# 1000 static routes
$ for byte2 in {0..3}; do for byte3 in {0..249}; do echo "{\"destination\": \"192.${byte2}.${byte3}.6/24\", \"nexthop\": \"10.0.0.2\"}"; done; done|jq -s '{"router": {"routes": .}}' > max_routes_body_1000.json

## Addition
$ time curl -g -i -X PUT http://192.168.0.77:9696/networking/v2.0/routers/5d28dfba-b779-4504-bbd0-3e382e9980ac/add_extraroutes -H "Content-Type: application/json" -H "X-Auth-Token: $(openstack token issue -f value -c id)" -d '@max_routes_body_1000.json'
Response time: 0m4.883s

## Deletion
$ time curl -g -i -X PUT http://192.168.0.77:9696/networking/v2.0/routers/5d28dfba-b779-4504-bbd0-3e382e9980ac -H "Content-Type: application/json" -H "X-Auth-Token: $(openstack token issue -f value -c id)" -d '{"router": {"routes": []}}'
Response time: 1m36.843s

# 2000 static routes
$ for byte2 in {0..7}; do for byte3 in {0..249}; do echo "{\"destination\": \"192.${byte2}.${byte3}.6/24\", \"nexthop\": \"10.0.0.2\"}"; done; done|jq -s '{"router": {"routes": .}}' > max_routes_body_2000.json

## Addition
$ time curl -g -i -X PUT http://192.168.0.77:9696/networking/v2.0/routers/5d28dfba-b779-4504-bbd0-3e382e9980ac/add_extraroutes -H "Content-Type: application/json" -H "X-Auth-Token: $(openstack token issue -f value -c id)" -d '@max_routes_body_2000.json'
Response time: 0m6.443s

## Deletion
$ time curl -g -i -X PUT http://192.168.0.77:9696/networking/v2.0/routers/5d28dfba-b779-4504-bbd0-3e382e9980ac -H "Content-Type: application/json" -H "X-Auth-Token: $(openstack token issue -f value -c id)" -d '{"router": {"routes": []}}'
Response time: 6m32.338s

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/914900

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/914900
Committed: https://opendev.org/openstack/neutron/commit/305153883b64c35c60649a4ba7e3f5a5b60a503c
Submitter: "Zuul (22348)"
Branch: master

commit 305153883b64c35c60649a4ba7e3f5a5b60a503c
Author: Ihtisham ul Haq <email address hidden>
Date: Tue Apr 2 17:43:43 2024 +0200

    Optimize deletion of static routes

    Instead of having a separate OVN command for deletion of each
    static route, send out the deletion as a single command.

    Which significantly improves the performance. Previously
    deletion of 1000 routes took >1m30s and >6m30s for 2000 routes,
    with this change it takes ~5s and ~8s, respectivily.

    Closes-Bug: #2060054
    Change-Id: Iaa5204e2e48795c31c502160041bd128189eef5a