Routed subnets cannot use snat

[Alban PRATS]

Hi,

We have the following network configuration:
- Subnet A: 10.20.0.0/24
- Subnet B: 10.20.10.0/24
- Router Ext:
    > With a external gateway
    > linked to Subnet A (10.20.0.1)
    > Route to 10.20.10.0/24 via 10.20.0.254
- Router Lan: Linked to both networks (10.20.0.254 and 10.20.10.1)
    > linked to Subnet A (10.20.0.254)
    > linked to Subnet B (10.20.10.1)
    > Route to 0.0.0.0/0 via 10.20.0.1

While instances in the Subnet A can access Internet, instances in Subnet B cannot.
I got an error: From 10.20.0.1 icmp_seq=1 Destination Net Unreachable.

It looks like there is an ip rule in the namespace of the qrouter-RouterExt which route only requests from 10.20.0.1/24 through the Snat.

[root@host ns: qrouter-e17d62be-deea-4317-948c-b1c9eedc93fa ~] ip route
10.20.0.0/24 dev qr-3a2a89c7-53 proto kernel scope link src 10.20.0.1
10.20.10.0/24 via 10.20.0.254 dev qr-3a2a89c7-53
169.254.110.154/31 dev rfp-e17d62be-d proto kernel scope link src 169.254.110.154
192.168.1.0/24 dev qr-0b3f2d0d-8b proto kernel scope link src 192.168.1.1
[root@host ns: qrouter-e17d62be-deea-4317-948c-b1c9eedc93fa ~] ip rule list
0: from all lookup local
32766: from all lookup main
32767: from all lookup default
169082881: from 10.20.0.1/24 lookup 169082881
[root@host ns: qrouter-e17d62be-deea-4317-948c-b1c9eedc93fa ~] ip route show table 169082881
default via 10.20.0.173 dev qr-3a2a89c7-53

Is there a reason why we only route directly attached networks through the snat ?

I've come up with a fix that adds an ip rule for each network specified in the router's routes. Can you give me some feedback on this ? I'm mainly concerned about the reasons why this was this way in the first place and about the cleanup of the unused rules.

Best regards,

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/890459

[Brian Haley (brian-haley)]

So this is a secondary subnet (B) attached via a router to the first subnet (A) if I'm understanding correctly.

Does this only happen with DVR? That's my question after looking at the patch.

I don't see a reason why this shouldn't work, I'll bring it up in our weekly meeting next Tuesday, but you will hopefully get some responses in the change before that.

[Arnaud Morin (arnaud-morin)]

Hey Brian,

Here is a schema of what we try to achieve.
The Router Ext is missing the ip rule for 10.20.10 network (what alban patch is providing).

And yes, we are running DVR, which means, all of these routers are collocated with the instance.
However, the network node (for snat / 10.20.0.173 on this schema) is on a different hardware.

We will do some tests without DVR to see how it behave.

One of our concern is: does anyone know/remember why the ip rule is added? Why not adding the gateway in the default route table?

[Arnaud Morin (arnaud-morin)]

So we did check, with distributed=false, everything works fine.

The ip rule is not there.

[Alban PRATS (al26p)]

A snippet of my deployment can be found in the attachment

Whatever the order of the resources creation, the dvr only contains ip rule for directly attached subnet:
 169082881: from 10.20.0.1/24 lookup 169082881

With my patch, rules to routed subnets are added. In the case of the described configuration, ip rules now contains:
 169082881: from 10.20.0.1/24 lookup 169082881
 169082881: from 10.20.10.0/24 lookup 169082881