[Routed networks] Router routes to other segment CIDRs should have a gateway IP

[Rodolfo Alonso]

The router namespace routes for other segment CIDRs, should have the "via" parameter to the gateway IP address.

The environment I'm using has HA routers; I didn't test yet with DVR or legacy routers.

For example, in my env [1]:
- Network: public
- Subnets:
  (s1) aebc3fc1-d984-40aa-8405-db51df03f60d: 10.46.54.0/26, gwip: 10.46.54.62
  (s2) 55a66c34-3643-49ba-ab8c-326ee76e5f35: 10.46.54.64/26, gwip: 10.46.54.126
- Router namespace:

[root@controller-0 ~]# ip netns exec $r ip r
default via 10.46.54.62 dev qg-7847fd00-72 proto 112
default via 10.46.54.62 dev qg-7847fd00-72 proto static
10.1.0.0/16 dev qr-d224ca69-77 proto kernel scope link src 10.1.0.1
10.46.54.0/26 dev qg-7847fd00-72 proto kernel scope link src 10.46.54.34
10.46.54.64/26 dev qg-7847fd00-72 proto 112 scope link
10.46.54.64/26 dev qg-7847fd00-72 proto static scope link
169.254.0.0/24 dev ha-91b6c056-57 proto kernel scope link src 169.254.0.233
169.254.192.0/18 dev ha-91b6c056-57 proto kernel scope link src 169.254.193.151

This router is attached to the segment of the subnet aebc3fc1-d984-40aa-8405-db51df03f60d (s1).

The route to 10.46.54.64/26 (the other subnet/segment this router is NOT attached to), should have the gateway IP address of the subnet that belongs to the segment the router is attached to; in this case, as commented, (s1).

[1]http://paste.openstack.org/show/804429/

Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1945061

[Bence Romsics (bence-romsics)]

Could you please elaborate what exactly is the topology you expect to be working?

I'm in now way an expert on routed networks, but so far I believed:

1) there should be a router that's attached to all segments of a (routed) network and
2) this router is not managed by Neutron:

"The Networking service does not provide layer-3 services between segments."
from here:
https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html

Are these assumptions correct? If no, please correct them. If yes, then this could be an RFE.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hi Bence:

Exactly, the Neutron router (that will be connected to one of those segments), will not provide any routing between the segments/subnets CIDRs. But the packets coming out from this Neutron router to the external router should point to the GW IP.

In the example provided: there are two segments, each one with a subnet. The router is connected to the first segment and has connectivity to subnet 1 CIDR (10.46.54.0/26). The GW IP for this subnet is 10.46.54.62. The route installed in the router namespace routing table (sorry for the alliteration), is in charge of sending the packets with destination to the subnet 2 CIDR out of the router namespace. But those packets should have, as destination MAC address, the GW IP device MAC address. That means the following route:

# 10.46.54.64/26 dev qg-7847fd00-72 proto static scope link

should be:

# 10.46.54.64/26 dev qg-7847fd00-72 via 10.46.54.62 proto static scope link

Regards.

[Bence Romsics (bence-romsics)]

Hi Rodolfo,

Let's see if I understand (sorry for being slow):

On a routed network you already have the non-neutron-managed router which connects all segments.
However you also want to create a neutron-managed router that's only connected to one of the segments, but not the others.
In this case traffic on the neutron-managed from the connected segment to the non-connected segment has the wrong route installed.

I'm not sure if I see the use case for the neutron-managed router, but the api clearly allows it, so in that case it should behave correctly.

Cheers.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

I think I didn't explain the problem in a correct way. The scenario presented here is something that maybe is not supported or not covered in the Routed Provider Networks feature.

In this scenario, multiple tenant networks are connected to a RPN using a Neutron router. This is an intermediate routing step between the VM and the RPN.

When a VM port is attached to a RPN segment, the routes (subnet.host_routes) are injected into the VM via the DHCP. In this case, the router routes to the other segment/subnet CIDRs are written in the routing table, but the GW IP is missing.

If this scenario should not be supported, then I'll close the bug and document it.

If this scenario is supported, I'll push a patch to fix it.

Regards.

[Bence Romsics (bence-romsics)]

I'm definitely not against such a change. At first I was slow to understand whether you consider this a bug or a missing a feature. But now I see it's a missing feature. However it's tiny, so probably we don't need to bring this to the drivers meeting. On the other hand if you push a patch for this, please also include a bit of documentation, because I'm not aware of any doc saying this is a supported model.

[Oskari Lemmelä (olemmela)]

Hi,

In case VM is directly attached to RPN segment correct other segment routes are injected via config-drive. I haven't tested DHCP case, but might be that it has same problems with gateway info in some cases.

Tripleo is using network segments in routed network deployment. In that case at least provisioning is working via DHCP. I haven't check if there only default route or correct routes propagated via DHCP. This also might be related to DHCP client how it handles extra routes in DHCP reply.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Oskari:

The problem is not in the VM configuration or the routes configured. The "subnet.host_routes" are injected into the VM correctly, in my case via DHCP.

The problem is, in the case I present, when we introduce an extra step between the RPN and the VM. In the case I'm presenting, the VM is not attached to the RPN, as the feature was implemented. In this scenario, the VM has directly communication to the GW configured in the "subnet.host_routes" that are injected, because the VM port is in the same broadcast domain.

Regards.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

After investigating this issue, I found that the router external interface (gateway port) configures the external network "extra subnets" as onlink routes. The "extra subnets" are the additional subnets apart from the gateway port IP CIDR. In this case, the extra subnets are those RPN subnets that are not connected to the router port.

By default, when a router gateway port is enabled, the extra subnets CIDRs are set as onlink routes in the router namespace. E.g.:
$ ip netns exec qrouter-8b56474d-027c-4d96-805d-869ceb92cd96 ip r
...
10.222.144.0/22 dev qg-f5b03925-e4 proto 112 scope link
10.222.144.0/22 dev qg-f5b03925-e4 proto static scope link
10.222.152.0/22 dev qg-f5b03925-e4 proto 112 scope link
10.222.152.0/22 dev qg-f5b03925-e4 proto static scope link
10.222.156.0/22 dev qg-f5b03925-e4 proto 112 scope link
10.222.156.0/22 dev qg-f5b03925-e4 proto static scope link

The justification for those onlink routes is described in the patch that added them [1].

The case reported in this bug is a particular condition where the gateway router is connected to a RPN, specifically to a segment (one subnet) of this network. That means this gateway port has broadcast connectivity to only this subnet CIDR. As commented, to have connectivity to the other subnets, the admin should manually add those extra routes on the router. E.g.:
  $ openstack router set --route destination=192.168.50.0/24,gateway=192.168.20.10 router1

That will overwrite the link routes and create global ones with (1) the destination IP, (2) the GW IP and (3) the device. E.g.:
...
192.168.50.0/24 via 192.168.20.10 dev qg-1d690261-35 proto 112
192.168.50.0/24 via 192.168.20.10 dev qg-1d690261-35 proto static
192.168.60.0/24 via 192.168.20.10 dev qg-1d690261-35 proto 112
192.168.60.0/24 via 192.168.20.10 dev qg-1d690261-35 proto static

This bug should be closed.

Regards.

[1]https://review.opendev.org/c/openstack/neutron/+/90250

[Oskari Lemmelä (olemmela)]

Hi,

In segmented network case only locally connected subnets should be added to router namespace. Not all of them as they are not reachable.

Manually adding static routes is not good approach in large openstack installation. If you add one segment or subnet to RPN, you would need to modify all routers connected to RPN.

I would say https://review.opendev.org/c/openstack/neutron/+/90250 does not handle segmented network case correctly. It should only add locally connected subnets, not all of them.

BR,
Oskari

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Oskari:

https://review.opendev.org/c/openstack/neutron/+/90250 was never meant to work with a RPN (was implemented time before) and RPN were designed to skip any L3 routing inside Neutron, externalizing this task to the underlying network infrastructure.

I'm going to propose a patch that could make legacy routers to work with RPN. But the community will decide if this is valid or not.

Regards.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/791178

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/795581

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791178
Committed: https://opendev.org/openstack/neutron/commit/f450886ff93a201c6520d84088c3a97330814140
Submitter: "Zuul (22348)"
Branch: master

commit f450886ff93a201c6520d84088c3a97330814140
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Thu May 13 06:13:57 2021 +0000

    Allow the use of legacy routers within RPN segments

    This patch adds to legacy routers (no HA nor DVR) to be connected
    to a router provider network segment through the gateway interface.
    The router will be connected to one single segment of the RPN; that
    means the router will have L2 connectivity to one single subnet.
    The gateway router port will have an IP address on the subnet CIDR;
    that will provide connectivity to the broadcast domain of this CIDR
    (as usual, that doesn't change).

    The router, in other scenarios, adds the other subnet CIDRs to the
    router namespace routing table. That allows to SNAT any packet to
    those CIDRs through the gateway port.

    In the RPN case those routes are not added because there is no
    broadcast connectivity with the other subnets. Any packet that needs
    to reach these other subents, should go through the local segment
    gateway IP address. This default route is added always into the
    router namespace.

    Closes-Bug: #1923592

    Change-Id: Ib66b1d7b60eb0ac0a9e3dfd08aae29cb03abde34

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/795581
Committed: https://opendev.org/openstack/neutron/commit/c0846c6518811c02435ba32cabd32592cea0018f
Submitter: "Zuul (22348)"
Branch: master

commit c0846c6518811c02435ba32cabd32592cea0018f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Wed Jun 9 15:37:48 2021 +0000

    Fix "config-routed-networks" document errors

    Fixed "config-routed-networks" document errors reported in [1].

    Related-Bug: #1923592

    [1]https://review.opendev.org/c/openstack/neutron/+/791178

    Change-Id: I3a0e83c1ca273582233fb7788c81e4ba298bc6d0

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.