neutron

Routed Networks DHCP agents getting ports in unconnected segments

Bug #1849726 reported by Gustavo Randich
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
New
Undecided
Unassigned

Bug Description

Using Queens in Ubuntu 18.04, with Routed Provider Networks (network segments).

DHCP agent RPC Callback in Neutron Server/API is returning DHCP reserved ports binded to fixed_ips of subnets in other segments, raising the error copied below when DHCP agent starts.

In our understanding, this situation arises when DHCP agents are stopped and its network namespace is removed, but it corresponding port not deleted. In our case we installed too many DHCP agents in each segment, so we decided to stop and uninstall several of them, cleaning the Linux network namespaces but not deleting the orphaned dhcp ports. When we tried to reinstall an agent in other compute, we hit the error.

---------------
Currently the list of subnets is correctly filtered (https://review.opendev.org/#/c/339365/) and works ok for newly started DHCP agents:

  subnets = [subnet for subnet in subnets
              if subnet['network_id'] not in routed_net_ids or
              subnet['id'] in seg_subnet_ids]

...but the list of ports is not filtered accordingly. We worked around this issue filtering ports immediatly below subnet filter:

  ports = [ port for port in ports
              if port['fixed_ips'][0]['subnet_id'] in seg_subnet_ids ]

------------------
ERROR oslo_messaging.rpc.server [req-bd6e9240-7844-4e0b-9c13-ee05e24f01a9 - - - - -] Exception during message handling: HostNotCompatibleWithFixedIps: Host r11-hostxx is not connected to a segment where the existing fixed_ips on port 1aaa7da6-e11f-44d3-adc8-5577359edc96 will function given the routed network topology.
ERROR oslo_messaging.rpc.server Traceback (most recent call last):
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner
ERROR oslo_messaging.rpc.server return func(*args, **kwargs)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped
ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 87, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 135, in wrapper
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped
ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 122, in wrapped
ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 305, in update_dhcp_port
ERROR oslo_messaging.rpc.server return self._port_action(plugin, context, port, 'update_port')
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/api/rpc/handlers/dhcp_rpc.py", line 100, in _port_action
ERROR oslo_messaging.rpc.server return plugin.update_port(context, port['id'], port)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 627, in inner
ERROR oslo_messaging.rpc.server return f(self, context, *args, **kwargs)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped
ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 91, in wrapped
ERROR oslo_messaging.rpc.server setattr(e, '_RETRY_EXCEEDED', True)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 87, in wrapped
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
ERROR oslo_messaging.rpc.server ectxt.value = e.inner_exc
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_db/api.py", line 135, in wrapper
ERROR oslo_messaging.rpc.server return f(*args, **kwargs)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 126, in wrapped
ERROR oslo_messaging.rpc.server LOG.debug("Retry wrapper got retriable exception: %s", e)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR oslo_messaging.rpc.server self.force_reraise()
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 122, in wrapped
ERROR oslo_messaging.rpc.server return f(*dup_args, **dup_kwargs)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/plugin.py", line 1359, in update_port
ERROR oslo_messaging.rpc.server port)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/api.py", line 161, in wrapped
ERROR oslo_messaging.rpc.server return method(*args, **kwargs)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/db_base_plugin_v2.py", line 1345, in update_port
ERROR oslo_messaging.rpc.server new_port=new_port)
ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/neutron/db/ipam_backend_mixin.py", line 783, in update_port
ERROR oslo_messaging.rpc.server host=host, port_id=old_port['id'])
ERROR oslo_messaging.rpc.server HostNotCompatibleWithFixedIps: Host r11-hostxx is not connected to a segment where the existing fixed_ips on port 1aaa7da6-e11f-44d3-adc8-5577359edc96 will function given the routed network topology.

Revision history for this message
Brian Haley (brian-haley) wrote :

This does look like a possible bug, let me try to further triage it.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/691760

Changed in neutron:
assignee: nobody → Brian Haley (brian-haley)
status: New → In Progress
Revision history for this message
Slawek Kaplonski (slaweq) wrote : auto-abandon-script

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

Changed in neutron:
assignee: Brian Haley (brian-haley) → nobody
status: In Progress → New
tags: added: timeout-abandon
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/691760
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.