Allocate IPs in bulk requests in separate transactions
In the ML2 plugin in create_port_bulk method, we are iterating over
list of the ports to be created and do everything for all ports in
single DB transaction (which makes totally sense as this is bulk
request).
But one of the things which was done during that huge transaction was
allocation of the IP addresses for all ports. That action is prone for
race conditions and can fail often, especially when there is no many IP
addresses available in the subnet(s) for the ports.
In case of the error while allocating IP address even for one port from
the whole bulk request, whole create_port_bulk method was retried so
allocations (and everything else) for all ports was reverted and started
from scratch. That takes a lot of time so some requests may be processed
very long time, like e.g. 2-3 minutes in my tests.
To reproduce that issue I did simple script which created network with
/24 subnet and then sent 24 requests to create 10 ports in bulk in each
request. That was in totall 240 ports created in that subnet.
I measured time of the creation of all those ports in the current master
branch (without this patch) and with the patch. Results are like below:
Reviewed: https:/ /review. opendev. org/c/openstack /neutron/ +/822794 /opendev. org/openstack/ neutron/ commit/ 56d2bda51d9750f 3aee8d0817b7993 b555010571
Committed: https:/
Submitter: "Zuul (22348)"
Branch: stable/xena
commit 56d2bda51d9750f 3aee8d0817b7993 b555010571
Author: Slawek Kaplonski <email address hidden>
Date: Tue Dec 14 12:44:31 2021 +0100
Allocate IPs in bulk requests in separate transactions
In the ML2 plugin in create_port_bulk method, we are iterating over
list of the ports to be created and do everything for all ports in
single DB transaction (which makes totally sense as this is bulk
request).
But one of the things which was done during that huge transaction was
allocation of the IP addresses for all ports. That action is prone for
race conditions and can fail often, especially when there is no many IP
addresses available in the subnet(s) for the ports.
In case of the error while allocating IP address even for one port from
the whole bulk request, whole create_port_bulk method was retried so
allocations (and everything else) for all ports was reverted and started
from scratch. That takes a lot of time so some requests may be processed
very long time, like e.g. 2-3 minutes in my tests.
To reproduce that issue I did simple script which created network with
/24 subnet and then sent 24 requests to create 10 ports in bulk in each
request. That was in totall 240 ports created in that subnet.
I measured time of the creation of all those ports in the current master
branch (without this patch) and with the patch. Results are like below:
+-- ---+--- ------- -----+- ------- ----+-- ------- ------- ------- ----+ ---+--- ------- -----+- ------- ----+-- ------- ------- ------- ----+ ---+--- ------- -----+- ------- ----+-- ------- ------- ------- ----+ ---+--- ------- -----+- ------- ----+-- ------- ------- ------- ----+
| Run | Master branch | This patch | Simulate bulk by creation |
| | [mm:ss] | [mm:ss] | of 10 ports one by one |
+--
| 1 | 01:37 | 01:02 | 00:57 |
| 2 | 02:06 | 00:40 | 01:03 |
| 3 | 02:08 | 00:41 | 00:59 |
| 4 | 02:14 | 00:45 | 00:55 |
| 5 | 01:58 | 00:45 | 00:57 |
| 6 | 02:37 | 00:53 | 01:05 |
| 7 | 01:59 | 00:42 | 00:58 |
| 8 | 02:01 | 00:41 | 00:57 |
| 9 | 02:39 | 00:42 | 00:55 |
| 10 | 01:59 | 00:41 | 00:56 |
+--
| AVG | 00:02:07 | 00:00:45 | 00:58 |
+--
Closes-Bug: #1954763 155130add6f1c69 f2772113c27 5c5ce5ad1067952 d8f3d9f992)
Change-Id: I8877c658446fed
(cherry picked from commit 82aabb0aa962a3c