After some more investigation it seems that there is problem with configuration of iptables rules by l3 agent and that is causing this issue: Sep 24 05:11:31.013868 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-ab6a74e7-3d3f-4442-ba8d-910bf49347c1', 'ip6tables-save'] {{(pid=81761) execute_rootwrap_daemon /opt/stack/neutron/neutron/agent/linux/utils.py:103}} Sep 24 05:11:31.029396 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: DEBUG neutron.agent.linux.iptables_manager [-] IPTablesManager.apply completed with success. 6 iptables commands were issued {{(pid=81761) _apply_synchronized /opt/stack/neutron/neutron/agent/linux/iptables_manager.py:625}} Sep 24 05:11:31.029652 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.linux.iptables_manager [-] IPTables Rules did not converge. Diff: # Generated by iptables_manager Sep 24 05:11:31.029652 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: *nat Sep 24 05:11:31.029652 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: :neutron-l3-agent-fip-pf - [0:0] Sep 24 05:11:31.029652 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: -I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-fip-pf Sep 24 05:11:31.029652 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: COMMIT Sep 24 05:11:31.029652 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: # Completed by iptables_manager Sep 24 05:11:31.030209 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: DEBUG oslo_concurrency.lockutils [-] Releasing lock "iptables-qrouter-ab6a74e7-3d3f-4442-ba8d-910bf49347c1" {{(pid=81761) lock /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:282}} Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info [-] IPTables Rules did not converge. Diff: # Generated by iptables_manager Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: *nat Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: :neutron-l3-agent-fip-pf - [0:0] Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: -I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-fip-pf Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: COMMIT Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: # Completed by iptables_manager: neutron_lib.exceptions.l3.IpTablesApplyException: IPTables Rules did not converge. Diff: # Generated by iptables_manager Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: *nat Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: :neutron-l3-agent-fip-pf - [0:0] Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: -I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-fip-pf Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: COMMIT Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: # Completed by iptables_manager Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info Traceback (most recent call last): Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/common/utils.py", line 167, in call Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info return func(*args, **kwargs) Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1247, in process Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info self.process_address_scope() Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1217, in process_address_scope Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info self.process_floating_ip_address_scope_rules() Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info next(self.gen) Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 436, in defer_apply Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info self.defer_apply_off() Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 450, in defer_apply_off Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info self._apply() Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 476, in _apply Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info raise l3_exc.IpTablesApplyException(msg) Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info neutron_lib.exceptions.l3.IpTablesApplyException: IPTables Rules did not converge. Diff: # Generated by iptables_manager Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info *nat Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info :neutron-l3-agent-fip-pf - [0:0] Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info -I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-fip-pf Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info COMMIT Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info # Completed by iptables_manager Sep 24 05:11:31.037953 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.router_info Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent [-] Failed to process compatible router: ab6a74e7-3d3f-4442-ba8d-910bf49347c1: neutron_lib.exceptions.l3.IpTablesApplyException: IPTables Rules did not converge. Diff: # Generated by iptables_manager Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: *nat Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: :neutron-l3-agent-fip-pf - [0:0] Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: -I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-fip-pf Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: COMMIT Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: # Completed by iptables_manager Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent Traceback (most recent call last): Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 771, in _process_routers_if_compatible Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent self._process_router_if_compatible(router) Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 615, in _process_router_if_compatible Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent self._process_added_router(router) Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 623, in _process_added_router Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent ri.process() Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/common/utils.py", line 170, in call Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent self.logger(e) Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent self.force_reraise() Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb) Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent raise value Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/common/utils.py", line 167, in call Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent return func(*args, **kwargs) Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1247, in process Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent self.process_address_scope() Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/l3/router_info.py", line 1217, in process_address_scope Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent self.process_floating_ip_address_scope_rules() Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent next(self.gen) Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 436, in defer_apply Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent self.defer_apply_off() Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 450, in defer_apply_off Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent self._apply() Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent File "/opt/stack/neutron/neutron/agent/linux/iptables_manager.py", line 476, in _apply Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent raise l3_exc.IpTablesApplyException(msg) Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent neutron_lib.exceptions.l3.IpTablesApplyException: IPTables Rules did not converge. Diff: # Generated by iptables_manager Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent *nat Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent :neutron-l3-agent-fip-pf - [0:0] Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent -I neutron-l3-agent-PREROUTING 1 -j neutron-l3-agent-fip-pf Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent COMMIT Sep 24 05:11:31.045277 ubuntu-focal-rax-dfw-0020004694 neutron-l3-agent[81761]: ERROR neutron.agent.l3.agent # Completed by iptables_manager