| 2022-01-20 16:51:51 |
Ching Kuo |
description |
When using creating a fully populated LB with ovn as provider, the listener/pools will stuck in PENDING_CREATE STATE.
I Haven't tested what the result for other Octavia provider as I only have OVN in my environment.
Steps to reproduce:
Create a fully populated LB with API (only includes listener in this example):
curl -i -X POST -H "Content-Type: application/json" -H "X-Auth-Token: $OS_TOKEN" -d '{"loadbalancer":{"admin_state_up":true,"listeners":[{"name":"http_listener","protocol":"TCP","protocol_port":80}],"vip_subnet_id":"SUBNET_ID","provider":"ovn","name":"best_load_balancer","tags":["test_tag"]}}' https://XXXXXXXXXXXXX:9876/v2.0/lbaas/loadbalancers
openstack loadbalancer listener show http_listener
+-----------------------------+--------------------------------------+
| Field | Value |
+-----------------------------+--------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2022-01-20T16:37:58 |
| default_pool_id | None |
| default_tls_container_ref | None |
| description | |
| id | de2bbd23-ca08-445f-b58b-e36c8ea709ad |
| insert_headers | None |
| l7policies | |
| loadbalancers | 467846cf-7b1a-49f8-ae2f-398da103e6e4 |
| name | http_listener |
| operating_status | OFFLINE |
| project_id | 76235f59db4f4b289513b08c3739d2d6 |
| protocol | TCP |
| protocol_port | 80 |
| provisioning_status | PENDING_CREATE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | None |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
| tls_ciphers | None |
| tls_versions | None |
| alpn_protocols | None |
| tags | |
+-----------------------------+--------------------------------------+
The listener will remain in PENDING_CREATE state forever.
OpenStack Version: stabe/xena
Found this issue while using cloud-provider-openstack with K8S to create LB, as it create LB with fully populated request, it will have listeners and pools in PENDING_CREATE status. |
Description
===========
When using creating a fully populated LB with ovn as provider, the listener/pools will stuck in PENDING_CREATE STATE.
I Haven't tested what the result for other Octavia provider as I only have OVN in my environment.
Found this issue while using cloud-provider-openstack with K8S to create LB, as it create LB with fully populated request, it will have listeners and pools in PENDING_CREATE status.
Steps to reproduce
==================
Create a fully populated LB with API (only includes listener in this example):
curl -i -X POST -H "Content-Type: application/json" -H "X-Auth-Token: $OS_TOKEN" -d '{"loadbalancer":{"admin_state_up":true,"listeners":[{"name":"http_listener","protocol":"TCP","protocol_port":80}],"vip_subnet_id":"SUBNET_ID","provider":"ovn","name":"best_load_balancer","tags":["test_tag"]}}' https://XXXXXXXXXXXXX:9876/v2.0/lbaas/loadbalancers
Expected Result
===============
LB and listener are both created and in ACTIVE state
Actual Result
=============
openstack loadbalancer listener show http_listener
+-----------------------------+--------------------------------------+
| Field | Value |
+-----------------------------+--------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2022-01-20T16:37:58 |
| default_pool_id | None |
| default_tls_container_ref | None |
| description | |
| id | de2bbd23-ca08-445f-b58b-e36c8ea709ad |
| insert_headers | None |
| l7policies | |
| loadbalancers | 467846cf-7b1a-49f8-ae2f-398da103e6e4 |
| name | http_listener |
| operating_status | OFFLINE |
| project_id | 76235f59db4f4b289513b08c3739d2d6 |
| protocol | TCP |
| protocol_port | 80 |
| provisioning_status | PENDING_CREATE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | None |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
| tls_ciphers | None |
| tls_versions | None |
| alpn_protocols | None |
| tags | |
+-----------------------------+--------------------------------------+
The listener will remain in PENDING_CREATE state forever.
Environment
===========
Deployment method: Kolla-Ansible
Octavia version:
octavia-api --version
9.0.1.dev8
OpenStack Version: stabe/xena
octavia.conf is generated with Kolla-Ansible with no modification.
Logs
====
2022-01-20 15:52:57.991 24 DEBUG ovn_octavia_provider.helper [-] Handling request lb_create with info {'id': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'vip_address': '10.112.7.174', 'vip_network_id': 'a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0', 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:52:58.037 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbCreateCommand(table=Load_Balancer, columns={'name': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'protocol': [], 'external_ids': {'neutron:vip': '10.112.7.174', 'neutron:vip_port_id': '3935726a-93bb-4930-8a61-6eb07cbe22af', 'enabled': 'True'}, 'selection_fields': ['ip_src', 'ip_dst', 'tp_src', 'tp_dst']}, row=False) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.167 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): LsLbAddCommand(switch=42ec735c-75c0-4930-a4c7-504e48877605, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=ac4b72e3-ef04-4c02-ae47-c398db4b409c, col_values=(('external_ids', {'ls_refs': '{"neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0": 1}'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): LrLbAddCommand(router=8a21ef14-97cb-428b-bfc1-a806c3a48243, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.169 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.169 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DbSetCommand(table=Load_Balancer, record=ac4b72e3-ef04-4c02-ae47-c398db4b409c, col_values=(('external_ids', {'lr_ref': 'neutron-9f704c9f-b469-40cc-b31e-b6575bcc0a51'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.210 24 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345
Logs when creating LB and listener seperately
=============================================
2022-01-20 15:54:55.583 20 INFO octavia.api.v2.controllers.load_balancer [req-8d50c4c2-37c7-4e66-ab5b-61dbcbaa78d9 - 76235f59db4f4b289513b08c3739d2d6 - default default] Sending create Load Balancer c120740a-c6b3-431f-8419-29b824973fde to provider ovn
2022-01-20 15:54:55.584 20 DEBUG ovn_octavia_provider.helper [-] Handling request lb_create with info {'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'vip_address': '10.112.7.102', 'vip_network_id': 'a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0', 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:54:55.625 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbCreateCommand(table=Load_Balancer, columns={'name': 'c120740a-c6b3-431f-8419-29b824973fde', 'protocol': [], 'external_ids': {'neutron:vip': '10.112.7.102', 'neutron:vip_port_id': '5d26f051-a891-4934-ba99-e1d76a83b8cd', 'enabled': 'True'}, 'selection_fields': ['ip_src', 'ip_dst', 'tp_src', 'tp_dst']}, row=False) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.789 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): LsLbAddCommand(switch=42ec735c-75c0-4930-a4c7-504e48877605, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.790 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'ls_refs': '{"neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0": 1}'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.791 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): LrLbAddCommand(router=8a21ef14-97cb-428b-bfc1-a806c3a48243, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.791 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.792 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.792 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'lr_ref': 'neutron-9f704c9f-b469-40cc-b31e-b6575bcc0a51'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.859 20 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345
2022-01-20 15:54:56.302 20 INFO octavia.api.v2.controllers.listener [req-bec2b26f-6c47-43d5-8952-c9640b5c129f - 76235f59db4f4b289513b08c3739d2d6 - default default] Sending create Listener 54f59773-7e9c-4030-9523-c241b025cdf4 to provider ovn
2022-01-20 15:54:56.302 20 DEBUG ovn_octavia_provider.helper [-] Handling request listener_create with info {'id': '54f59773-7e9c-4030-9523-c241b025cdf4', 'protocol': 'TCP', 'loadbalancer_id': 'c120740a-c6b3-431f-8419-29b824973fde', 'protocol_port': 443, 'default_pool_id': None, 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:54:56.303 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('protocol', 'tcp'),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.342 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'listener_54f59773-7e9c-4030-9523-c241b025cdf4': '443:'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('protocol', 'tcp'),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DbClearCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, column=vips) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('vips', {}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.373 20 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'listeners': [{'id': '54f59773-7e9c-4030-9523-c241b025cdf4', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}], 'loadbalancers': [{'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'provisioning_status': 'ACTIVE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345 |
|
| 2022-01-20 16:52:57 |
Ching Kuo |
description |
Description
===========
When using creating a fully populated LB with ovn as provider, the listener/pools will stuck in PENDING_CREATE STATE.
I Haven't tested what the result for other Octavia provider as I only have OVN in my environment.
Found this issue while using cloud-provider-openstack with K8S to create LB, as it create LB with fully populated request, it will have listeners and pools in PENDING_CREATE status.
Steps to reproduce
==================
Create a fully populated LB with API (only includes listener in this example):
curl -i -X POST -H "Content-Type: application/json" -H "X-Auth-Token: $OS_TOKEN" -d '{"loadbalancer":{"admin_state_up":true,"listeners":[{"name":"http_listener","protocol":"TCP","protocol_port":80}],"vip_subnet_id":"SUBNET_ID","provider":"ovn","name":"best_load_balancer","tags":["test_tag"]}}' https://XXXXXXXXXXXXX:9876/v2.0/lbaas/loadbalancers
Expected Result
===============
LB and listener are both created and in ACTIVE state
Actual Result
=============
openstack loadbalancer listener show http_listener
+-----------------------------+--------------------------------------+
| Field | Value |
+-----------------------------+--------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2022-01-20T16:37:58 |
| default_pool_id | None |
| default_tls_container_ref | None |
| description | |
| id | de2bbd23-ca08-445f-b58b-e36c8ea709ad |
| insert_headers | None |
| l7policies | |
| loadbalancers | 467846cf-7b1a-49f8-ae2f-398da103e6e4 |
| name | http_listener |
| operating_status | OFFLINE |
| project_id | 76235f59db4f4b289513b08c3739d2d6 |
| protocol | TCP |
| protocol_port | 80 |
| provisioning_status | PENDING_CREATE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | None |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
| tls_ciphers | None |
| tls_versions | None |
| alpn_protocols | None |
| tags | |
+-----------------------------+--------------------------------------+
The listener will remain in PENDING_CREATE state forever.
Environment
===========
Deployment method: Kolla-Ansible
Octavia version:
octavia-api --version
9.0.1.dev8
OpenStack Version: stabe/xena
octavia.conf is generated with Kolla-Ansible with no modification.
Logs
====
2022-01-20 15:52:57.991 24 DEBUG ovn_octavia_provider.helper [-] Handling request lb_create with info {'id': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'vip_address': '10.112.7.174', 'vip_network_id': 'a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0', 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:52:58.037 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbCreateCommand(table=Load_Balancer, columns={'name': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'protocol': [], 'external_ids': {'neutron:vip': '10.112.7.174', 'neutron:vip_port_id': '3935726a-93bb-4930-8a61-6eb07cbe22af', 'enabled': 'True'}, 'selection_fields': ['ip_src', 'ip_dst', 'tp_src', 'tp_dst']}, row=False) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.167 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): LsLbAddCommand(switch=42ec735c-75c0-4930-a4c7-504e48877605, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=ac4b72e3-ef04-4c02-ae47-c398db4b409c, col_values=(('external_ids', {'ls_refs': '{"neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0": 1}'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): LrLbAddCommand(router=8a21ef14-97cb-428b-bfc1-a806c3a48243, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.169 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.169 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DbSetCommand(table=Load_Balancer, record=ac4b72e3-ef04-4c02-ae47-c398db4b409c, col_values=(('external_ids', {'lr_ref': 'neutron-9f704c9f-b469-40cc-b31e-b6575bcc0a51'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.210 24 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345
Logs when creating LB and listener seperately
=============================================
2022-01-20 15:54:55.583 20 INFO octavia.api.v2.controllers.load_balancer [req-8d50c4c2-37c7-4e66-ab5b-61dbcbaa78d9 - 76235f59db4f4b289513b08c3739d2d6 - default default] Sending create Load Balancer c120740a-c6b3-431f-8419-29b824973fde to provider ovn
2022-01-20 15:54:55.584 20 DEBUG ovn_octavia_provider.helper [-] Handling request lb_create with info {'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'vip_address': '10.112.7.102', 'vip_network_id': 'a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0', 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:54:55.625 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbCreateCommand(table=Load_Balancer, columns={'name': 'c120740a-c6b3-431f-8419-29b824973fde', 'protocol': [], 'external_ids': {'neutron:vip': '10.112.7.102', 'neutron:vip_port_id': '5d26f051-a891-4934-ba99-e1d76a83b8cd', 'enabled': 'True'}, 'selection_fields': ['ip_src', 'ip_dst', 'tp_src', 'tp_dst']}, row=False) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.789 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): LsLbAddCommand(switch=42ec735c-75c0-4930-a4c7-504e48877605, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.790 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'ls_refs': '{"neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0": 1}'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.791 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): LrLbAddCommand(router=8a21ef14-97cb-428b-bfc1-a806c3a48243, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.791 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.792 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.792 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'lr_ref': 'neutron-9f704c9f-b469-40cc-b31e-b6575bcc0a51'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.859 20 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345
2022-01-20 15:54:56.302 20 INFO octavia.api.v2.controllers.listener [req-bec2b26f-6c47-43d5-8952-c9640b5c129f - 76235f59db4f4b289513b08c3739d2d6 - default default] Sending create Listener 54f59773-7e9c-4030-9523-c241b025cdf4 to provider ovn
2022-01-20 15:54:56.302 20 DEBUG ovn_octavia_provider.helper [-] Handling request listener_create with info {'id': '54f59773-7e9c-4030-9523-c241b025cdf4', 'protocol': 'TCP', 'loadbalancer_id': 'c120740a-c6b3-431f-8419-29b824973fde', 'protocol_port': 443, 'default_pool_id': None, 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:54:56.303 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('protocol', 'tcp'),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.342 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'listener_54f59773-7e9c-4030-9523-c241b025cdf4': '443:'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('protocol', 'tcp'),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DbClearCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, column=vips) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('vips', {}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.373 20 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'listeners': [{'id': '54f59773-7e9c-4030-9523-c241b025cdf4', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}], 'loadbalancers': [{'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'provisioning_status': 'ACTIVE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345 |
Description
===========
When using creating a fully populated LB with ovn as provider, the listener/pools will stuck in PENDING_CREATE STATE.
I Haven't tested what the result for other Octavia provider as I only have OVN in my environment.
Found this issue while using cloud-provider-openstack with K8S to create LB, as it creates LB with fully populated request, it will have listeners and pools in PENDING_CREATE status.
Steps to reproduce
==================
Create a fully populated LB with API (only includes listener in this example):
curl -i -X POST -H "Content-Type: application/json" -H "X-Auth-Token: $OS_TOKEN" -d '{"loadbalancer":{"admin_state_up":true,"listeners":[{"name":"http_listener","protocol":"TCP","protocol_port":80}],"vip_subnet_id":"SUBNET_ID","provider":"ovn","name":"best_load_balancer","tags":["test_tag"]}}' https://XXXXXXXXXXXXX:9876/v2.0/lbaas/loadbalancers
Expected Result
===============
LB and listener are both created and in ACTIVE state
Actual Result
=============
openstack loadbalancer listener show http_listener
+-----------------------------+--------------------------------------+
| Field | Value |
+-----------------------------+--------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2022-01-20T16:37:58 |
| default_pool_id | None |
| default_tls_container_ref | None |
| description | |
| id | de2bbd23-ca08-445f-b58b-e36c8ea709ad |
| insert_headers | None |
| l7policies | |
| loadbalancers | 467846cf-7b1a-49f8-ae2f-398da103e6e4 |
| name | http_listener |
| operating_status | OFFLINE |
| project_id | 76235f59db4f4b289513b08c3739d2d6 |
| protocol | TCP |
| protocol_port | 80 |
| provisioning_status | PENDING_CREATE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | None |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
| tls_ciphers | None |
| tls_versions | None |
| alpn_protocols | None |
| tags | |
+-----------------------------+--------------------------------------+
The listener will remain in PENDING_CREATE state forever.
Environment
===========
Deployment method: Kolla-Ansible
Octavia version:
octavia-api --version
9.0.1.dev8
OpenStack Version: stabe/xena
octavia.conf is generated with Kolla-Ansible with no modification.
Logs
====
2022-01-20 15:52:57.991 24 DEBUG ovn_octavia_provider.helper [-] Handling request lb_create with info {'id': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'vip_address': '10.112.7.174', 'vip_network_id': 'a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0', 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:52:58.037 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbCreateCommand(table=Load_Balancer, columns={'name': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'protocol': [], 'external_ids': {'neutron:vip': '10.112.7.174', 'neutron:vip_port_id': '3935726a-93bb-4930-8a61-6eb07cbe22af', 'enabled': 'True'}, 'selection_fields': ['ip_src', 'ip_dst', 'tp_src', 'tp_dst']}, row=False) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.167 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): LsLbAddCommand(switch=42ec735c-75c0-4930-a4c7-504e48877605, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=ac4b72e3-ef04-4c02-ae47-c398db4b409c, col_values=(('external_ids', {'ls_refs': '{"neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0": 1}'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): LrLbAddCommand(router=8a21ef14-97cb-428b-bfc1-a806c3a48243, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.169 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.169 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DbSetCommand(table=Load_Balancer, record=ac4b72e3-ef04-4c02-ae47-c398db4b409c, col_values=(('external_ids', {'lr_ref': 'neutron-9f704c9f-b469-40cc-b31e-b6575bcc0a51'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.210 24 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345
Logs when creating LB and listener seperately
=============================================
2022-01-20 15:54:55.583 20 INFO octavia.api.v2.controllers.load_balancer [req-8d50c4c2-37c7-4e66-ab5b-61dbcbaa78d9 - 76235f59db4f4b289513b08c3739d2d6 - default default] Sending create Load Balancer c120740a-c6b3-431f-8419-29b824973fde to provider ovn
2022-01-20 15:54:55.584 20 DEBUG ovn_octavia_provider.helper [-] Handling request lb_create with info {'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'vip_address': '10.112.7.102', 'vip_network_id': 'a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0', 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:54:55.625 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbCreateCommand(table=Load_Balancer, columns={'name': 'c120740a-c6b3-431f-8419-29b824973fde', 'protocol': [], 'external_ids': {'neutron:vip': '10.112.7.102', 'neutron:vip_port_id': '5d26f051-a891-4934-ba99-e1d76a83b8cd', 'enabled': 'True'}, 'selection_fields': ['ip_src', 'ip_dst', 'tp_src', 'tp_dst']}, row=False) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.789 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): LsLbAddCommand(switch=42ec735c-75c0-4930-a4c7-504e48877605, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.790 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'ls_refs': '{"neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0": 1}'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.791 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): LrLbAddCommand(router=8a21ef14-97cb-428b-bfc1-a806c3a48243, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.791 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.792 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.792 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'lr_ref': 'neutron-9f704c9f-b469-40cc-b31e-b6575bcc0a51'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.859 20 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345
2022-01-20 15:54:56.302 20 INFO octavia.api.v2.controllers.listener [req-bec2b26f-6c47-43d5-8952-c9640b5c129f - 76235f59db4f4b289513b08c3739d2d6 - default default] Sending create Listener 54f59773-7e9c-4030-9523-c241b025cdf4 to provider ovn
2022-01-20 15:54:56.302 20 DEBUG ovn_octavia_provider.helper [-] Handling request listener_create with info {'id': '54f59773-7e9c-4030-9523-c241b025cdf4', 'protocol': 'TCP', 'loadbalancer_id': 'c120740a-c6b3-431f-8419-29b824973fde', 'protocol_port': 443, 'default_pool_id': None, 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:54:56.303 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('protocol', 'tcp'),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.342 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'listener_54f59773-7e9c-4030-9523-c241b025cdf4': '443:'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('protocol', 'tcp'),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DbClearCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, column=vips) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('vips', {}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.373 20 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'listeners': [{'id': '54f59773-7e9c-4030-9523-c241b025cdf4', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}], 'loadbalancers': [{'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'provisioning_status': 'ACTIVE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345 |
|
