[ovn-octavia-provider] Load Balancer remained with ACTIVE state even with PENDING_UPDATE listener

Bug #1956745 reported by Luis Tomas Bolivar
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Luis Tomas Bolivar

Bug Description

Description of problem:

The Load Balancer remained with ACTIVE and ONLINE state even when the listener had provisioning_status PENDING_UPDATE and the pool was with ERROR, what made the load-balancer to be considered functional by Kuryr and a load-balancer member removal was attempt without success as pool was immutable:

2021-11-03 17:04:53.893 1 ERROR kuryr_kubernetes.handlers.logging openstack.exceptions.ConflictException: ConflictException: 409: Client Error for url: https://10.x.x.x:13876/v2.0/lbaas/po
ols/302709f7-d122-49d7-9d05-8e57ccf2ad11/members/214e892a-4e25-46ce-8bba-9ae005304931, Pool 302709f7-d122-49d7-9d05-8e57ccf2ad11 is immutable and cannot be updated.
2021-11-03 17:04:53.893 1 ERROR kuryr_kubernetes.handlers.logging

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer show 8d03213c-ab26-4391-9d26-4878fa6b5d02
+---------------------+-------------------------------------------+
| Field | Value |
+---------------------+-------------------------------------------+
| admin_state_up | True |
| created_at | 2021-11-02T05:21:15 |
| description | |
| flavor_id | None |
| id | 8d03213c-ab26-4391-9d26-4878fa6b5d02 |
| listeners | 39f36082-3a07-44c0-9b00-41ba12db49fa |
| name | openshift-marketplace/community-operators |
| operating_status | ONLINE |
| pools | 302709f7-d122-49d7-9d05-8e57ccf2ad11 |
| project_id | fd0d6a21436d4ff987682c3a0419569d |
| provider | ovn |
| provisioning_status | ACTIVE |
| updated_at | 2021-11-02T13:45:52 |
| vip_address | 172.30.237.127 |
| vip_network_id | f738aa91-61e1-4571-8eac-f3979164393f |
| vip_port_id | 72502dfe-e6f8-4e9b-9f00-b846bcf693f8 |
| vip_qos_policy_id | None |
| vip_subnet_id | 14d2bae5-7de6-4f77-882b-1d5c919251fc |
+---------------------+-------------------------------------------+

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer listener show 39f36082-3a07-44c0-9b00-41ba12db49fa
+-----------------------------+-----------------------------------------------------+
| Field | Value |
+-----------------------------+-----------------------------------------------------+
| admin_state_up | True |
| connection_limit | -1 |
| created_at | 2021-11-02T05:21:44 |
| default_pool_id | 302709f7-d122-49d7-9d05-8e57ccf2ad11 |
| default_tls_container_ref | None |
| description | |
| id | 39f36082-3a07-44c0-9b00-41ba12db49fa |
| insert_headers | None |
| l7policies | |
| loadbalancers | 8d03213c-ab26-4391-9d26-4878fa6b5d02 |
| name | openshift-marketplace/community-operators:TCP:50051 |
| operating_status | ONLINE |
| project_id | fd0d6a21436d4ff987682c3a0419569d |
| protocol | TCP |
| protocol_port | 50051 |
| provisioning_status | PENDING_UPDATE |
| sni_container_refs | [] |
| timeout_client_data | 50000 |
| timeout_member_connect | 5000 |
| timeout_member_data | 50000 |
| timeout_tcp_inspect | 0 |
| updated_at | 2021-11-02T13:44:27 |
| client_ca_tls_container_ref | None |
| client_authentication | NONE |
| client_crl_container_ref | None |
| allowed_cidrs | None |
+-----------------------------+-----------------------------------------------------+

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer pool show 302709f7-d122-49d7-9d05-8e57ccf2ad11
+----------------------+-----------------------------------------------------+
| Field | Value |
+----------------------+-----------------------------------------------------+
| admin_state_up | True |
| created_at | 2021-11-02T05:21:49 |
| description | |
| healthmonitor_id | |
| id | 302709f7-d122-49d7-9d05-8e57ccf2ad11 |
| lb_algorithm | SOURCE_IP_PORT |
| listeners | 39f36082-3a07-44c0-9b00-41ba12db49fa |
| loadbalancers | 8d03213c-ab26-4391-9d26-4878fa6b5d02 |
| members | 214e892a-4e25-46ce-8bba-9ae005304931 |
| name | openshift-marketplace/community-operators:TCP:50051 |
| operating_status | OFFLINE |
| project_id | fd0d6a21436d4ff987682c3a0419569d |
| protocol | TCP |
| provisioning_status | ERROR |
| session_persistence | None |
| updated_at | 2021-11-02T13:45:52 |
| tls_container_ref | None |
| ca_tls_container_ref | None |
| crl_container_ref | None |
| tls_enabled | False |
+----------------------+-----------------------------------------------------+

(shiftstack) [stack@undercloud-0 ~]$ openstack loadbalancer member list 302709f7-d122-49d7-9d05-8e57ccf2ad11
+--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+
| id | name | project_id | provisioning_status | address | protocol_port | operating_status | weight |
+--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+
| 214e892a-4e25-46ce-8bba-9ae005304931 | openshift-marketplace/community-operators-zbmxw:50051 | fd0d6a21436d4ff987682c3a0419569d | ACTIVE | 10.128.46.128 | 50051 | NO_MONITOR | 1 |
+--------------------------------------+-------------------------------------------------------+----------------------------------+---------------------+---------------+---------------+------------------+--------+

Changed in neutron:
status: New → In Progress
Changed in neutron:
assignee: nobody → Luis Tomas Bolivar (ltomasbo)
Changed in neutron:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ovn-octavia-provider (master)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/823544
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/4d01470f5e397bb6de1b10db9f839fa72697ff6b
Submitter: "Zuul (22348)"
Branch: master

commit 4d01470f5e397bb6de1b10db9f839fa72697ff6b
Author: Luis Tomas Bolivar <email address hidden>
Date: Wed Jan 5 15:48:57 2022 +0100

    Set listeners back to ACTIVE upon pool/member action failures

    This patch ensure the listeners are set back from PENDING_UPDATE
    to ACTIVE in case of member or pool update failure.

    Closes-Bug: #1956745
    Change-Id: I95860ae305c3d0c10859afb972d000328e23d614

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ovn-octavia-provider (stable/xena)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ovn-octavia-provider (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/825215

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ovn-octavia-provider (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/825505

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ovn-octavia-provider (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/825515

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ovn-octavia-provider (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/825214
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/bdfe09f0b09c947b540eb30f79f7fc24dc8812ce
Submitter: "Zuul (22348)"
Branch: stable/xena

commit bdfe09f0b09c947b540eb30f79f7fc24dc8812ce
Author: Luis Tomas Bolivar <email address hidden>
Date: Wed Jan 5 15:48:57 2022 +0100

    Set listeners back to ACTIVE upon pool/member action failures

    This patch ensure the listeners are set back from PENDING_UPDATE
    to ACTIVE in case of member or pool update failure.

    Closes-Bug: #1956745
    Change-Id: I95860ae305c3d0c10859afb972d000328e23d614
    (cherry picked from commit 4d01470f5e397bb6de1b10db9f839fa72697ff6b)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ovn-octavia-provider (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/825215
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/f0b44088132dd84a2dea3d07d6e44fd01008a18d
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit f0b44088132dd84a2dea3d07d6e44fd01008a18d
Author: Luis Tomas Bolivar <email address hidden>
Date: Wed Jan 5 15:48:57 2022 +0100

    Set listeners back to ACTIVE upon pool/member action failures

    This patch ensure the listeners are set back from PENDING_UPDATE
    to ACTIVE in case of member or pool update failure.

    Closes-Bug: #1956745
    Change-Id: I95860ae305c3d0c10859afb972d000328e23d614
    (cherry picked from commit 4d01470f5e397bb6de1b10db9f839fa72697ff6b)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ovn-octavia-provider (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/825505
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/c1345ced65315ea12ba7d7efb8ee0652f5e2ce97
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit c1345ced65315ea12ba7d7efb8ee0652f5e2ce97
Author: Luis Tomas Bolivar <email address hidden>
Date: Wed Jan 5 15:48:57 2022 +0100

    Set listeners back to ACTIVE upon pool/member action failures

    This patch ensure the listeners are set back from PENDING_UPDATE
    to ACTIVE in case of member or pool update failure.

    Closes-Bug: #1956745
    Change-Id: I95860ae305c3d0c10859afb972d000328e23d614
    (cherry picked from commit 4d01470f5e397bb6de1b10db9f839fa72697ff6b)

tags: added: in-stable-victoria
tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ovn-octavia-provider (stable/ussuri)

Reviewed: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/825515
Committed: https://opendev.org/openstack/ovn-octavia-provider/commit/6885cfb046e4eae933f1347f34aefd6e28268d32
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 6885cfb046e4eae933f1347f34aefd6e28268d32
Author: Luis Tomas Bolivar <email address hidden>
Date: Thu Jan 20 15:52:46 2022 +0100

    Set listeners back to ACTIVE upon pool/member action failures

    This patch ensure the listeners are set back from PENDING_UPDATE
    to ACTIVE in case of member or pool update failure.

    From ovn-octavia-provider master:
        https://review.opendev.org/c/openstack/ovn-octavia-provider/+/823544

    Closes-Bug: #1956745
    (manually cherry picked from commit
    4d01470f5e397bb6de1b10db9f839fa72697ff6b)

    Change-Id: I74725c598d46578d2b8e9c39be71abb79257da6f

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider 2.0.0.0rc1

This issue was fixed in the openstack/ovn-octavia-provider 2.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider 1.0.1

This issue was fixed in the openstack/ovn-octavia-provider 1.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn train-eol

This issue was fixed in the openstack/networking-ovn train-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider 1.2.0

This issue was fixed in the openstack/ovn-octavia-provider 1.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider ussuri-eol

This issue was fixed in the openstack/ovn-octavia-provider ussuri-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.