Addding route in qrouter namesapce failing as Nexthop has invalid gateway.

[Mohankumar]

Sometimes we see a case that rfp in qrouter namespace and fpr in fip namespace is down . which results in adding route with those interface in l3 agent code at dvr_fip_ns.py failing as "Nexthop has invalid gateway"

The better way would be check the interface status before add the route entry at https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_fip_ns.py#L458

Stack trace:
============
2024-03-29 08:00:19.027 5996 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: Error: Nexthop has invalid gateway.

2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: Error: Nexthop has invalid gateway.
: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: Error: Nexthop has invalid gateway.
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info File "/opt/openstack/neutron/lib/python3.6/site-packages/neutron/common/utils.py", line 159, in call
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info File "/opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1207, in process
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info self.process_external()
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info File "/opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/l3/dvr_local_router.py", line 724, in process_external
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info self.connect_rtr_2_fip()
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info File "/opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/l3/dvr_local_router.py", line 739, in connect_rtr_2_fip
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info self.fip_ns.create_rtr_2_fip_link(self)
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info File "/opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 456, in create_rtr_2_fip_link
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info rtr_2_fip_dev.route.add_gateway(str(fip_2_rtr.ip), table=FIP_RT_TBL)
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info File "/opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 609, in add_gateway
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info self._as_root([ip_version], tuple(args))
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info File "/opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 410, in _as_root
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info use_root_namespace=use_root_namespace)
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info File "/opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 124, in _as_root
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info namespace=namespace)
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info File "/opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 132, in _execute
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info File "/opt/openstack/neutron/lib/python3.6/site-packages/neutron/agent/linux/utils.py", line 160, in execute
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info returncode=returncode)
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: Error: Nexthop has invalid gateway.
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info
2024-03-29 08:00:19.028 5996 ERROR neutron.agent.l3.router_info

[Oleg Bondarev (obondarev)]

Hi Mohankumar,

> The better way would be check the interface status before add the route entry

this will help to avoid error in logs but how will it help?

> rfp in qrouter namespace and fpr in fip namespace is down

are there steps to reproduce?

[Mohankumar (mohankumar-n)]

Hi Oleg Bondarev,

> If interface link id down It fails to add static route so the forwarding failing . So it breaking communication

https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_fip_ns.py#L458

> Reproduce stpes , is hard to write down . But we often seeing in our infras with fip attached to vm and DVR router setup

[Brian Haley (brian-haley)]

What version of neutron is this? Just for completeness. Thanks.