SNAT namespace prematurely created then deleted on hosts, resulting in removal of RFP/FPR link to FIP namespace

[Arjun Baindur]

Seems like collateral from https://bugs.launchpad.net/neutron/+bug/1850779

I think this fix causes problems. We have multiple nodes that are DVR_SNAT mode. Snat namespace is scheduled to 1 of them.

When l3-agent is restarted on the othre nodes, now, initialize() is invoked always for DvrEdgeRouter which creates the SNAT namespace prematurely. This in turn causes external_gateway_added() to later detect that this host is NOT hosting snat router, but the namespace exists, so it removes it by triggering external_gateway_removed(dvr_edge_router --> dvr_local_router)

Problem is that the dvr_local_router code for external_gateway_removed() ends up DELETING the rfp/fpr pair and severs the qrouter connection to fip namespace (and deletes all the FIP routes in fip namespace as a result).

Prior to this bug fix, _create_snat_namespace was only invoked in _create_dvr_gateway(), which was only invoked when the node was actually hosting SNAT for the router.

Even without the breaking issue of deleting the rtr_2_fip link, this fix unnecessarily creates SNAT namespace on every host, only for it to be deleted.

FYI this is for non-HA routers

1. Where the qrouter to FIP link is deleted: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_local_router.py#L599

This results in connectivity breakage

2. Above #1 is triggered by code here in edge router which sees snat namespace, but SNAT is scheduled to different host: https://github.com/openstack/neutron/blob/master/neutron/agent/l3/dvr_edge_router.py#L56

3. SNAT namespace is created on wrong host because of bug fix for 1850779 which moved it to DvrEdgeRouter intilization

[LIU Yulong (dragon889)]

hi Arjun Baindur,

Could you please add some steps of how to reproduce this bug? The router "HA" and "distributed" state with the config options of L3 agents are also needed.

And did you try the fix [1] of this bug https://bugs.launchpad.net/neutron/+bug/1881995 ? It maybe can solve your problem.
[1] https://review.opendev.org/c/openstack/neutron/+/734070

[Arjun Baindur (abaindur)]

Liu, this is not for an HA router. Also, it is not for centralized FIPs.

1. This is a compute node, where l3_agent is in dvr_snat mode. We have multiple such nodes where l3-agent is in dvr_snat mode for regular failover

2. Router is a regular DVR router, not HA. We have no centralized FIPs.

3. There are VMs on the same node with and without floating IPs.

So to reproduce, have 2 or more nodes in DVR SNAT mode for l3-agent. These should also be compute nodes, so nova-compute, etc... is on same.

Create a DVR but non-HA router, so that one snat namespace gets scheduled to one of the 2+ nodes. Create a VM and some Floating IPs on each node, so qrouter namespace is created, fip namespace is created, and rfp/fpr link is created on all nodes.

At this point, snat has been scheduled to one of these dvr_snat nodes as well.

Now, restart l3-agent on one of the OTHER nodes.

You will see on init snat namespace gets created on these nodes, then deleted again in the code paths I listed before. The deletion code triggers deletion of gateway which ends up deleting rfp/fpr link between qrouter and FIP.

Prior to the fix, snat was not created then deleted on dvr_snat nodes that did not host snat router

[Brian Haley (brian-haley)]

Hi Arjun,

Do you also have the fix in https://review.opendev.org/c/openstack/neutron/+/775372 ? That was a follow-on that should solve this issue.

So do you run all your compute nodes in dvr_snat mode? I would think you'd only want dvr mode so they only handle floating IP functions and not SNAT, as I don't think there is a guarantee that the snat router will be placed on the local compute node.

[Arjun Baindur (abaindur)]

Depends on the deployment. But we have multiple nodes in dvr_snat mode to handle SNAT failover, so at minimum 2. And since we don't want compute resources to go to waste, we also schedule VMs on them. So it may be a mix of dvr and dvr_snat nodes.

Yes, we don't care that the SNAT node is local, since the namespace should be reachable from any qrouter, and thus a local VM on the same node that SNAT got scheduled to, or a remote VM on any other node (whether it's also in dvr mode or dvr_snat)

[Arjun Baindur (abaindur)]

Don't think we have that fix, we will try that out.

BTW are you saying the recommendation is to NOT run VMs on dvr_snat nodes? It seems like a waste otherwise to dedicate nodes to run SNAT (especially when we run at least 2 or 3 for failover), especially when usually a majority of VMs use floating IP traffic or are on provider networks.

[Brian Haley (brian-haley)]

Regarding running VMs on dvr_snat nodes, I'll just say that in our environment we follow the "old school" way of having Network Nodes in dvr_snat mode that have no VMs, and Compute Nodes in dvr mode hosting VMs. They're slightly different hardware and tuning configurations since a NN is very busy with l3/dhcp-agent work, with CNs being more stable for running workloads. It also allows us to restart services on, or reboot, the NN without affecting tenant VMs as much. There's no wrong way, just my opinion.

[Slawek Kaplonski (slaweq)]

Is it duplicate of https://bugs.launchpad.net/neutron/+bug/1894843 which actually was already fixed?

[Brian Haley (brian-haley)]

Is this another case of using dvr_snat + compute as not supported:
https://meetings.opendev.org/meetings/networking/2021/networking.2021-07-20-14.00.log.html#l-113
https://review.opendev.org/c/openstack/neutron/+/801503