[OVN Octavia Provider] OVN provider status update failures can leave orphaned resources

Bug #1900763 reported by Brian Haley
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Brian Haley

Bug Description

This is related to Octavia issue https://storyboard.openstack.org/#!/story/2008254

When the OVN Octavia Provider driver calls into octavia-lib to update the status of a loadbalancer, for example, the code in helper:_update_status_to_octavia() it might fail:

DEBUG ovn_octavia_provider.helper [-] Updating status to octavia: {'listeners': [{'id': '7033179d-2ddb-4714-9c06-b7d399498238', 'provisioning_status>
ERROR ovn_octavia_provider.helper [-] Error while updating the load balancer status: 'NoneType' object has no attribute 'update': octavia_lib.api.dr>
ERROR ovn_octavia_provider.helper [-] Unexpected exception in request_handler: octavia_lib.api.drivers.exceptions.UpdateStatusError: ('The status up>
ERROR ovn_octavia_provider.helper Traceback (most recent call last):
ERROR ovn_octavia_provider.helper File "/opt/stack/ovn-octavia-provider/ovn_octavia_provider/helper.py", line 318, in _update_status_to_octavia
ERROR ovn_octavia_provider.helper self._octavia_driver_lib.update_loadbalancer_status(status)
ERROR ovn_octavia_provider.helper File "/usr/local/lib/python3.8/dist-packages/octavia_lib/api/drivers/driver_lib.py", line 121, in update_loadbal>
ERROR ovn_octavia_provider.helper raise driver_exceptions.UpdateStatusError(
ERROR ovn_octavia_provider.helper octavia_lib.api.drivers.exceptions.UpdateStatusError: 'NoneType' object has no attribute 'update'

This is failing because the listener associated with the loadbalancer was not found, it's DB transaction was in-flight. That's the related Octavia issue from above, but a fix for that will not solve the problem.

A side-effect is this listener is now "stuck":

$ openstack loadbalancer listener delete 7033179d-2ddb-4714-9c06-b7d399498238
Load Balancer 2cc1d429-b176-48e5-adaa-946be2af0d51 is immutable and cannot be updated. (HTTP 409) (Request-ID: req-0e1e53ac-4db9-4779-b1f3-11210fe46f7f)

The provider driver needs to retry the operation, typically even the very next call succeeds.

summary: [OVN Octavia Provider] OVN provider status update failures can leave
- orphaned resourcess
+ orphaned resources
Miguel Lavalle (minsel)
Changed in neutron:
status: New → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ovn-octavia-provider (master)

Fix proposed to branch: master
Review: https://review.opendev.org/759139

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
Brian Haley (brian-haley) wrote :

Changed to Committed manually since I don't know if Zuul is doing this again yet.

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 7.4.0

This issue was fixed in the openstack/networking-ovn 7.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider 1.0.0

This issue was fixed in the openstack/ovn-octavia-provider 1.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider 0.4.0

This issue was fixed in the openstack/ovn-octavia-provider 0.4.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ovn-octavia-provider 0.1.3

This issue was fixed in the openstack/ovn-octavia-provider 0.1.3 release.

Changed in neutron:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.