Concurrent routerroute update fails on deletion with AttributeError

[Sebastian Lohff]

When updating a router and providing a set of extra routes / routerroutes that result in some routes being deleted, it might happen that two workers fetch the routes at the same time and then both try to delete the route. As the route is fetched before deletion, in one of the two workers the get_object() will return None, on which delete() is called, resulting in an AttributeError:

AttributeError: 'NoneType' object has no attribute 'delete'

The result is not fulfilled properly and a 500 is returned to the user.

This was observed on neutron yoga, though the same code (+ a breaking test) seem to confirm this on current master.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/912629

[Brian Haley (brian-haley)]

Do you have an example traceback just to be complete? thanks

[Sebastian Lohff (sebageek)]

Sure thing!

AttributeError: 'NoneType' object has no attribute 'delete'
  File "neutron/api/v2/resource.py", line 98, in resource
    result = method(request=request, **args)
  File "neutron_lib/db/api.py", line 139, in wrapped
    setattr(e, '_RETRY_EXCEEDED', True)
  File "oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "neutron_lib/db/api.py", line 135, in wrapped
    return f(*args, **kwargs)
  File "oslo_db/api.py", line 154, in wrapper
    ectxt.value = e.inner_exc
  File "oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "oslo_db/api.py", line 142, in wrapper
    return f(*args, **kwargs)
  File "neutron_lib/db/api.py", line 183, in wrapped
    LOG.debug("Retry wrapper got retriable exception: %s", e)
  File "oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "neutron_lib/db/api.py", line 179, in wrapped
    return f(*dup_args, **dup_kwargs)
  File "neutron/api/v2/base.py", line 253, in _handle_action
    ret_value = getattr(self._plugin, name)(*arg_list, **kwargs)
  File "neutron_lib/db/api.py", line 218, in wrapped
    return method(*args, **kwargs)
  File "neutron_lib/db/api.py", line 139, in wrapped
    setattr(e, '_RETRY_EXCEEDED', True)
  File "oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "neutron_lib/db/api.py", line 135, in wrapped
    return f(*args, **kwargs)
  File "oslo_db/api.py", line 154, in wrapper
    ectxt.value = e.inner_exc
  File "oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "oslo_db/api.py", line 142, in wrapper
    return f(*args, **kwargs)
  File "neutron_lib/db/api.py", line 183, in wrapped
    LOG.debug("Retry wrapper got retriable exception: %s", e)
  File "oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "neutron_lib/db/api.py", line 179, in wrapped
    return f(*dup_args, **dup_kwargs)
  File "neutron/db/extraroute_db.py", line 223, in add_extraroutes
    router = self.update_router(
  File "neutron/db/extraroute_db.py", line 61, in update_router
    routes_added, routes_removed = self._update_extra_routes(
  File "neutron/db/extraroute_db.py", line 136, in _update_extra_routes
    l3_obj.RouterRoute.get_object(

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/912629
Committed: https://opendev.org/openstack/neutron/commit/27b2f22df10e6e41c6fc4e1ce7f839aeb3dc3e13
Submitter: "Zuul (22348)"
Branch: master

commit 27b2f22df10e6e41c6fc4e1ce7f839aeb3dc3e13
Author: Sebastian Lohff <email address hidden>
Date: Tue Mar 12 19:34:17 2024 +0100

    Don't delete already deleted extra router routes

    When handling the deletion of extra routes we need to handle the case
    that the route is already deleted by another call in the time we have
    fetched the extra routes and try to delete it. This is a classic race
    condition when two calls try to update the routes of a router at the
    same time. The default MariaDB/MySQL transaction isolation level does
    not suffice to prevent this scenario. Directly deleting the route
    without fetching it solves this problem.

    Change-Id: Ie8238310569eb7c1c53296195800bef5c9cb92a3
    Closes-Bug: #2057698