neutron

[OVN] Stale Logical Router Port entry fail to be removed by maintenance task

Bug #1920968 reported by Lucas Alvares Gomes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Lucas Alvares Gomes

Bug Description

Reported at: https://bugzilla.redhat.com/show_bug.cgi?id=1940724

The maintenance task invokes the delete_router_port() method from OVNClient when it tries to delete a stale Logical_Router_Port entry but, if the router is also gone an exception will be raised because the code in delete_router_port() will try to fetch the router from the Neutron database.

We need to make the delete_router_port() more resilient to those situations, if the router is gone it means that the router port is also gone so we just need to get rid of it in the OVN database as well.

2021-03-19 08:01:00.271 39 DEBUG futurist.periodics [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Submitting periodic callback 'networking_ovn.common.maintenance.DBInconsistenciesPeriodics.check_for_inconsistencies' _process_scheduled /usr/lib/python3.6/site-packages/futurist/periodics.py:639
2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Synchronizing Neutron and OVN databases check_for_inconsistencies /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:341
2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Number of inconsistencies found at delete: router_ports=1 _log /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:322
2021-03-19 08:01:00.291 39 DEBUG networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Fixing resource 6c742662-a761-4e77-8b5a-8f9834916fac (type: router_ports) at delete check_for_inconsistencies /usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py:375
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Failed to fix deleted resource 6c742662-a761-4e77-8b5a-8f9834916fac (type: router_ports): neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 198, in _get_router
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance context, l3_models.Router, router_id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/neutron_lib/db/model_query.py", line 159, in get_by_id
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance return query.filter(model.id == object_id).one()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib64/python3.6/site-packages/sqlalchemy/orm/query.py", line 3282, in one
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance raise orm_exc.NoResultFound("No row was found for one()")
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance sqlalchemy.orm.exc.NoResultFound: No row was found for one()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance During handling of the above exception, another exception occurred:
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance Traceback (most recent call last):
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 380, in check_for_inconsistencies
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance self._fix_delete(row)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/networking_ovn/common/maintenance.py", line 261, in _fix_delete
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance res_map['ovn_delete'](row.resource_uuid)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/networking_ovn/common/ovn_client.py", line 1502, in delete_router_port
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance router = self._l3_plugin.get_router(context, router_id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 233, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance return method(*args, **kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance setattr(e, '_RETRY_EXCEEDED', True)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance self.force_reraise()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance six.reraise(self.type_, self.value, self.tb)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance raise value
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance return f(*args, **kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance ectxt.value = e.inner_exc
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance self.force_reraise()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance six.reraise(self.type_, self.value, self.tb)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance raise value
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance return f(*args, **kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance LOG.debug("Retry wrapper got retriable exception: %s", e)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance self.force_reraise()
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance six.reraise(self.type_, self.value, self.tb)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance raise value
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance return f(*dup_args, **dup_kwargs)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 533, in get_router
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance router = self._get_router(context, id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance File "/usr/lib/python3.6/site-packages/neutron/db/l3_db.py", line 200, in _get_router
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance raise l3_exc.RouterNotFound(router_id=router_id)
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance neutron_lib.exceptions.l3.RouterNotFound: Router 56335450-362d-4c49-9a28-44d5d2908847 could not be found
2021-03-19 08:01:00.348 39 ERROR networking_ovn.common.maintenance
2021-03-19 08:01:00.350 39 INFO networking_ovn.common.maintenance [req-d9433ac0-a5a9-4d7a-927f-c8dc32015600 - - - - -] Maintenance task: Synchronization finished (took 0.06 seconds)

Lucas Alvares Gomes (lucasagomes)
Changed in neutron:
assignee: nobody → Lucas Alvares Gomes (lucasagomes)
status: New → Confirmed
importance: Undecided → High
tags: added: ovn
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.0.0.0rc1

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

Revision history for this message
Lajos Katona (lajos-katona) wrote :

The patch for it: https://review.opendev.org/c/openstack/neutron/+/782551

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/networking-ovn 7.4.1

This issue was fixed in the openstack/networking-ovn 7.4.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 16.3.2

This issue was fixed in the openstack/neutron 16.3.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 17.1.2

This issue was fixed in the openstack/neutron 17.1.2 release.

Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Brian Haley (brian-haley)
Changed in neutron:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.