[ovn-octavia-provider] Listner/Pool Stuck in PENDING_CREATE Using Fully Populated LB API

Bug #1958561 reported by Ching Kuo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Fernando Royo

Bug Description

Description
===========
When using creating a fully populated LB with ovn as provider, the listener/pools will stuck in PENDING_CREATE STATE.

I Haven't tested what the result for other Octavia provider as I only have OVN in my environment.

Found this issue while using cloud-provider-openstack with K8S to create LB, as it creates LB with fully populated request, it will have listeners and pools in PENDING_CREATE status.

Steps to reproduce
==================

Create a fully populated LB with API (only includes listener in this example):

curl -i -X POST -H "Content-Type: application/json" -H "X-Auth-Token: $OS_TOKEN" -d '{"loadbalancer":{"admin_state_up":true,"listeners":[{"name":"http_listener","protocol":"TCP","protocol_port":80}],"vip_subnet_id":"SUBNET_ID","provider":"ovn","name":"best_load_balancer","tags":["test_tag"]}}' https://XXXXXXXXXXXXX:9876/v2.0/lbaas/loadbalancers

Expected Result
===============

LB and listener are both created and in ACTIVE state

Actual Result
=============

openstack loadbalancer listener show http_listener
+-----------------------------+--------------------------------------+
| Field | Value |
+-----------------------------+--------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2022-01-20T16:37:58 |
| default_pool_id | None |
| default_tls_container_ref | None |
| description | |
| id | de2bbd23-ca08-445f-b58b-e36c8ea709ad |
| insert_headers | None |
| l7policies | |
| loadbalancers | 467846cf-7b1a-49f8-ae2f-398da103e6e4 |
| name | http_listener |
| operating_status | OFFLINE |
| project_id | 76235f59db4f4b289513b08c3739d2d6 |
| protocol | TCP |
| protocol_port | 80 |
| provisioning_status | PENDING_CREATE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | None |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
| tls_ciphers | None |
| tls_versions | None |
| alpn_protocols | None |
| tags | |
+-----------------------------+--------------------------------------+

The listener will remain in PENDING_CREATE state forever.

Environment
===========

Deployment method: Kolla-Ansible

Octavia version:
octavia-api --version
9.0.1.dev8

OpenStack Version: stabe/xena

octavia.conf is generated with Kolla-Ansible with no modification.

Logs
====

2022-01-20 15:52:57.991 24 DEBUG ovn_octavia_provider.helper [-] Handling request lb_create with info {'id': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'vip_address': '10.112.7.174', 'vip_network_id': 'a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0', 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:52:58.037 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbCreateCommand(table=Load_Balancer, columns={'name': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'protocol': [], 'external_ids': {'neutron:vip': '10.112.7.174', 'neutron:vip_port_id': '3935726a-93bb-4930-8a61-6eb07cbe22af', 'enabled': 'True'}, 'selection_fields': ['ip_src', 'ip_dst', 'tp_src', 'tp_dst']}, row=False) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.167 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): LsLbAddCommand(switch=42ec735c-75c0-4930-a4c7-504e48877605, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=ac4b72e3-ef04-4c02-ae47-c398db4b409c, col_values=(('external_ids', {'ls_refs': '{"neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0": 1}'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): LrLbAddCommand(router=8a21ef14-97cb-428b-bfc1-a806c3a48243, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.168 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.169 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=ac4b72e3-ef04-4c02-ae47-c398db4b409c, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.169 24 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DbSetCommand(table=Load_Balancer, record=ac4b72e3-ef04-4c02-ae47-c398db4b409c, col_values=(('external_ids', {'lr_ref': 'neutron-9f704c9f-b469-40cc-b31e-b6575bcc0a51'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:52:58.210 24 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': '5b743d3f-3c66-4735-89d3-f02f26cd63cc', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345

Logs when creating LB and listener seperately
=============================================
2022-01-20 15:54:55.583 20 INFO octavia.api.v2.controllers.load_balancer [req-8d50c4c2-37c7-4e66-ab5b-61dbcbaa78d9 - 76235f59db4f4b289513b08c3739d2d6 - default default] Sending create Load Balancer c120740a-c6b3-431f-8419-29b824973fde to provider ovn
2022-01-20 15:54:55.584 20 DEBUG ovn_octavia_provider.helper [-] Handling request lb_create with info {'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'vip_address': '10.112.7.102', 'vip_network_id': 'a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0', 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:54:55.625 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbCreateCommand(table=Load_Balancer, columns={'name': 'c120740a-c6b3-431f-8419-29b824973fde', 'protocol': [], 'external_ids': {'neutron:vip': '10.112.7.102', 'neutron:vip_port_id': '5d26f051-a891-4934-ba99-e1d76a83b8cd', 'enabled': 'True'}, 'selection_fields': ['ip_src', 'ip_dst', 'tp_src', 'tp_dst']}, row=False) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.789 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): LsLbAddCommand(switch=42ec735c-75c0-4930-a4c7-504e48877605, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.790 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'ls_refs': '{"neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0": 1}'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.791 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): LrLbAddCommand(router=8a21ef14-97cb-428b-bfc1-a806c3a48243, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.791 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.792 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=4): LsLbAddCommand(switch=neutron-a5bb4f81-fbe7-4a3c-b6d6-b25d22e48ec0, lb=e5cc4da9-05be-4428-9600-e202c29adcb5, may_exist=True) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.792 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=5): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'lr_ref': 'neutron-9f704c9f-b469-40cc-b31e-b6575bcc0a51'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:55.859 20 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'loadbalancers': [{'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345
2022-01-20 15:54:56.302 20 INFO octavia.api.v2.controllers.listener [req-bec2b26f-6c47-43d5-8952-c9640b5c129f - 76235f59db4f4b289513b08c3739d2d6 - default default] Sending create Listener 54f59773-7e9c-4030-9523-c241b025cdf4 to provider ovn
2022-01-20 15:54:56.302 20 DEBUG ovn_octavia_provider.helper [-] Handling request listener_create with info {'id': '54f59773-7e9c-4030-9523-c241b025cdf4', 'protocol': 'TCP', 'loadbalancer_id': 'c120740a-c6b3-431f-8419-29b824973fde', 'protocol_port': 443, 'default_pool_id': None, 'admin_state_up': True} request_handler /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:318
2022-01-20 15:54:56.303 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('protocol', 'tcp'),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.342 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=0): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('external_ids', {'listener_54f59773-7e9c-4030-9523-c241b025cdf4': '443:'}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=1): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('protocol', 'tcp'),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=2): DbClearCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, column=vips) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.343 20 DEBUG ovsdbapp.backend.ovs_idl.transaction [-] Running txn n=1 command(idx=3): DbSetCommand(table=Load_Balancer, record=e5cc4da9-05be-4428-9600-e202c29adcb5, col_values=(('vips', {}),)) do_commit /var/lib/kolla/venv/lib/python3.8/site-packages/ovsdbapp/backend/ovs_idl/transaction.py:90
2022-01-20 15:54:56.373 20 DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'listeners': [{'id': '54f59773-7e9c-4030-9523-c241b025cdf4', 'provisioning_status': 'ACTIVE', 'operating_status': 'ONLINE'}], 'loadbalancers': [{'id': 'c120740a-c6b3-431f-8419-29b824973fde', 'provisioning_status': 'ACTIVE'}]} _update_status_to_octavia /var/lib/kolla/venv/lib/python3.8/site-packages/ovn_octavia_provider/helper.py:345

Ching Kuo (genekuo)
description: updated
description: updated
description: updated
Ching Kuo (genekuo)
description: updated
description: updated
tags: added: ovn-octavia-provider
Revision history for this message
Fernando Royo (froyoredhat) wrote :
Changed in neutron:
assignee: nobody → Fernando Royo (froyoredhat)
status: New → Fix Released
Revision history for this message
Ching Kuo (genekuo) wrote :

Yes, it is duplicated.

I'm not sure why this earlier bug report is neglected and a new bug report is created and use...

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.