router set --disable <router>
And in neutron-l3-agent.log you’ll see:
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent [-] Error while deleting router ac2068d0-2dba-4411-b74e-e5fd5ecd1179: TypeError: unhashable type: 'dict'
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib64/python3.11/site-packages/neutron/agent/l3/agent.py", line 514, in _safe_router_removed
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent self.l3_ext_manager.delete_router(self.context, ri.router)
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib64/python3.11/site-packages/neutron/agent/l3/l3_agent_extensions_manager.py", line 63, in delete_router
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent extension.obj.delete_router(context, data)
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib64/python3.11/site-packages/neutron/services/logapi/agent/l3/base.py", line 105, in delete_router
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent self.log_driver.stop_logging(context, router_info=data)
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib64/python3.11/site-packages/neutron_fwaas/services/logapi/agents/drivers/iptables/log.py", line 250, in stop_logging
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent self._cleanup_nflog_process(router_info)
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib64/python3.11/site-packages/neutron_fwaas/services/logapi/agents/drivers/iptables/log.py", line 180, in _cleanup_nflog_process
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent if router_info in self.nflog_proc_map:
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent TypeError: unhashable type: 'dict'
2024-08-21 06:59:47.588 16 ERROR neutron.agent.l3.agent
2024-08-21 06:59:47.589 16 WARNING neutron.agent.l3.agent [-] Hit retry limit with router update for ac2068d0-2dba-4411-b74e-e5fd5ecd1179, action 3
The issue is caused by an incorrect key lookup which was preventing the killing of the process on router deletion which was leading to orphaned processes
When flow logging is enabled, try
router set --disable <router> l3-agent. log you’ll see:
And in neutron-
2024-08-21 06:59:47.588 16 ERROR neutron. agent.l3. agent [-] Error while deleting router ac2068d0- 2dba-4411- b74e-e5fd5ecd11 79: TypeError: unhashable type: 'dict' agent.l3. agent Traceback (most recent call last): agent.l3. agent File "/var/lib/ kolla/venv/ lib64/python3. 11/site- packages/ neutron/ agent/l3/ agent.py" , line 514, in _safe_router_ removed agent.l3. agent self.l3_ ext_manager. delete_ router( self.context, ri.router) agent.l3. agent File "/var/lib/ kolla/venv/ lib64/python3. 11/site- packages/ neutron/ agent/l3/ l3_agent_ extensions_ manager. py", line 63, in delete_router agent.l3. agent extension. obj.delete_ router( context, data) agent.l3. agent File "/var/lib/ kolla/venv/ lib64/python3. 11/site- packages/ neutron/ services/ logapi/ agent/l3/ base.py" , line 105, in delete_router agent.l3. agent self.log_ driver. stop_logging( context, router_info=data) agent.l3. agent File "/var/lib/ kolla/venv/ lib64/python3. 11/site- packages/ neutron_ fwaas/services/ logapi/ agents/ drivers/ iptables/ log.py" , line 250, in stop_logging agent.l3. agent self._cleanup_ nflog_process( router_ info) agent.l3. agent File "/var/lib/ kolla/venv/ lib64/python3. 11/site- packages/ neutron_ fwaas/services/ logapi/ agents/ drivers/ iptables/ log.py" , line 180, in _cleanup_ nflog_process agent.l3. agent if router_info in self.nflog_ proc_map: agent.l3. agent ^^^^^^^ ^^^^^^^ ^^^^^^^ ^^^^^^^ ^^^^^^ agent.l3. agent TypeError: unhashable type: 'dict' agent.l3. agent agent.l3. agent [-] Hit retry limit with router update for ac2068d0- 2dba-4411- b74e-e5fd5ecd11 79, action 3
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.588 16 ERROR neutron.
2024-08-21 06:59:47.589 16 WARNING neutron.
The issue is caused by an incorrect key lookup which was preventing the killing of the process on router deletion which was leading to orphaned processes