Comment 4 for bug 2038529

I may have found the root cause,

The deletion of the listeners is performed in parallel in a cascade delete:

https://opendev.org/openstack/octavia/src/branch/master/octavia/controller/worker/v2/flows/load_balancer_flows.py#L267-L271

In a local run:

Oct 05 07:42:47 gthiemon-devstack octavia-worker[2318143]: DEBUG octavia.common.base_taskflow [-] Task 'delete_update_vip_22963119-46f2-4602-ad3e-383a901ef690' (05c388fb-869e-46c6-8262-253267b1d961) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=2318143) _task_receiver /opt/stack/taskflow/taskflow/listeners/logging.py:190}}
Oct 05 07:42:47 gthiemon-devstack octavia-worker[2318143]: DEBUG octavia.common.base_taskflow [-] Task 'delete_update_vip_206f37f9-9472-45b6-83ac-ca56728e29db' (31a421ad-f23e-4e27-a285-3521427e5776) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=2318143) _task_receiver /opt/stack/taskflow/taskflow/listeners/logging.py:190}}
Oct 05 07:42:47 gthiemon-devstack octavia-worker[2318143]: DEBUG octavia.common.base_taskflow [-] Task 'delete_update_vip_41f809ea-b22c-4640-899a-6283163a6644' (f19ef136-79f0-4002-8b6d-58df240f750e) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=2318143) _task_receiver /opt/stack/taskflow/taskflow/listeners/logging.py:190}}
Oct 05 07:42:47 gthiemon-devstack octavia-worker[2318143]: DEBUG octavia.common.base_taskflow [-] Task 'delete_update_vip_b60a0b2c-eb69-425d-bf9e-b3932c21d25e' (c95ec23e-2641-4d76-8436-a147c028bb25) transitioned into state 'RUNNING' from state 'PENDING' {{(pid=2318143) _task_receiver /opt/stack/taskflow/taskflow/listeners/logging.py:190}}
[..]
Oct 05 07:42:52 gthiemon-devstack octavia-worker[2318143]: DEBUG octavia.common.base_taskflow [-] Task 'delete_update_vip_d7f478ef-0d2c-4ae1-9c0e-43dae2ed81fb' (84d40bae-c0a8-4696-880b-6785fcebe9a9) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=2318143) _task_receiver /opt/stack/taskflow/taskflow/listeners/logging.py:178}}
Oct 05 07:42:53 gthiemon-devstack octavia-worker[2318143]: DEBUG octavia.common.base_taskflow [-] Task 'delete_update_vip_3385191a-22a5-482c-b446-a09db3fb7de9' (fe60f53f-ea74-4265-a53a-07152f99af29) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=2318143) _task_receiver /opt/stack/taskflow/taskflow/listeners/logging.py:178}}
Oct 05 07:42:53 gthiemon-devstack octavia-worker[2318143]: DEBUG octavia.common.base_taskflow [-] Task 'delete_update_vip_14c1e5ee-ab93-4c3a-a880-a71bf30f5320' (ea2844ff-2c16-4ea6-a956-5234df516171) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=2318143) _task_receiver /opt/stack/taskflow/taskflow/listeners/logging.py:178}}
Oct 05 07:42:54 gthiemon-devstack octavia-worker[2318143]: DEBUG octavia.common.base_taskflow [-] Task 'delete_update_vip_700e6723-8990-4494-933c-a8d94f6d3b36' (dd5f7980-88e1-4698-866f-4004c34fefce) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' {{(pid=2318143) _task_receiver /opt/stack/taskflow/taskflow/listeners/logging.py:178}}

And each task fetches the current state of the SGs, then computes a delta and applies new SGs (if needed) when updating the VIP. So yes, there's a race condition, it cannot work if the tasks are executed simultaneously.

I think that running the tasks in parallel is fine here, but we need to ensure that we don't do things that depends on or could be impacted by the results of the other tasks.