[RFE] Allow multiple external gateways on a router

[Bence Romsics]

I'd like to bring the following idea to the drivers' meeting. If this still looks like a good idea after that discussion, I'll open a spec so this can be properly commented on in gerrit. Until then feel free to comment here of course.

# Problem Description

A general router can be configured to connect and route to multiple external networks for higher availability and/or to balance the load. However the current Neutron API syntax allows exactly one external gateway for a router.

https://docs.openstack.org/api-ref/network/v2/?expanded=create-router-detail#create-router

{
    "router": {
        "name": "router1",
        "external_gateway_info": {
            "network_id": "ae34051f-aa6c-4c75-abf5-50dc9ac99ef3",
            "enable_snat": true,
            "external_fixed_ips": [
                {
                    "ip_address": "172.24.4.6",
                    "subnet_id": "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
                }
            ]
        },
        "admin_state_up": true
    }
}

However consider the following (simplified) network architecture as an example:

R3 R4
 |X|
R1 R2
 |X|
C1 C2 ...

(Sorry, my original, nice ascii art was eaten by launchpad. I hope this still conveys what I mean.)

Where C1, C2, ... are compute nodes, R1 and R2 are OpenStack-managed routers, while R3 and R4 are provider edge routers. Between R1-R2 and R3-R4 Equal Cost Multipath (ECMP) routing is used to utilize all links in an active-active manner. In such an architecture it makes sense to represent R1 and R2 as 2 logical routers with 2-2 external gateways, or in some cases (depending on other architectural choices) even as 1 logical router with 4 external gateways. But with the current API that is not possible.

# Proposed Change

Extend the router API object with a new attribute 'additional_external_gateways', for example:

{
   "router" : {
      "name" : "router1",
      "admin_state_up" : true,
      "external_gateway_info" : {
         "enable_snat" : false,
         "external_fixed_ips" : [
            {
               "ip_address" : "172.24.4.6",
               "subnet_id" : "b930d7f6-ceb7-40a0-8b81-a425dd994ccf"
            }
         ],
         "network_id" : "ae34051f-aa6c-4c75-abf5-50dc9ac99ef3"
      },
      "additional_external_gateways" : [
         {
            "enable_snat" : false,
            "external_fixed_ips" : [
               {
                  "ip_address" : "172.24.5.6",
                  "subnet_id" : "62da64b0-29ab-11eb-9ed9-3b1175418487"
               }
            ],
            "network_id" : "592d4716-29ab-11eb-a7dd-4f4b5e319915"
         },
         ...
      ]
   }
}

Edited via the following HTTP PUT methods with diff semantics:

PUT /v2.0/routers/{router_id}/add_additional_external_gateways
PUT /v2.0/routers/{router_id}/remove_additional_external_gateways

We keep 'external_gateway_info' for backwards compatibility. When additional_external_gateways is an empty list, everything behaves as before. When additional_external_gateways are given, then the actual list of external gateways is (in Python-like pseudo-code): [external_gateway_info] + additional_external_gateways.

Unless otherwise specified all non-directly connected external IPs are routed towards the original external_gateway_info. However this behavior may be overriden by either using (static) extraroutes, or by running () routing protocols and routing towards the external gateway where a particular route was learned from.

# Alternatives

1) Using 4 logical routers with 1 external gateway each. However in this case the API misses the information which (2 or 4) logical routers represent the same backend router.

2) Using a VRRP HA router. However this provides a different level of High Availability plus it is active-passive instead of active-active.

3) Adding router interfaces (since their number is not limited in the API) instead of external gateways. However this creates confusion by blurring the line of what is internal and what is external to the cloud deployment.

[LIU Yulong (dragon889)]

One router has one external gateway is a fundamental mechanism for neutron, IMO it will be too complicated to refactor the router to support multiple external gateway, because we have DVR, floating IPs, extensions, routes and NAT and so one.

[Bence Romsics (bence-romsics)]

Would it make sense to allow multiple gateways in an incremental fashion? For example as a first step for routers that do no NAT, no floating IPs, no DVR? The actual use case I heard about is exactly like that. Hopefully we'd have less work that way, and we could move on to the rest when we have a use case for them. What do you think?

[Slawek Kaplonski (slaweq)]

Hi Bence,

Can You elaborate more about routers which "do no NAT, no floating IPs, no DVR"? What routers would it be and why it would even need external gateway if it's not doing NAT nor Floating IPs?

[Bence Romsics (bence-romsics)]

Hi Slawek,

In this design all routers are speaking BGP. Therefore all subnets can be publicly routable, and that's why NAT and floating IPs are not needed.

On the other hand what's not so clear to me is this: What are the consequences of Alternative (3) (using router interfaces instead of external gateways). In my current understanding for neutron-dynamic-routing it matters which network is added as an external gateway to a router therefore (even with no NAT) an external gateway cannot just simply be replaced by a router interface.

But maybe this means that a change to neutron-dynamic-routing (allowing deeper control of what routes are advertized) in combination with Alternative (3) here could still be an alternative to having multiple external gateways.

I'm definitely happy to hear about other alternatives (that I did not list above probably because I did not think about them :-)) to represent/support multiple BGP speaking ECMP routers - that's what creates the multiple legs to the North in the above picture.

If we had a way to support that without a big change like this, that just would make our work easier.

[Slawek Kaplonski (slaweq)]

Thx for info Bence. Lets discuss it on next drivers meeting on 4.12.2020.

[Brian Haley (brian-haley)]

I think a neutron router won't do NAT if the internal and external addresses are of the same scope, right? And for IPv6 there is no NAT :)

As far as the RFE goes, my first thought is this would be implemented as a single external interface of the router with >1 default route? But it looks like above you're talking about a second external interface since there is an additional network and subnet. As Liu mentioned, this does complicate things, especially from an IP address consumption viewpoint, I just haven't wrapped my head around it yet.

[Akihiro Motoki (amotoki)]

I think we need to clarify the problem this RFE would like to address.

The current problem description mentions ECMP between R1/R2 (neutron logical router) and R3/R4 (provider router) but the proposed change just discusses multiple external gateways per neutron router. I think it solves a case like below. Is my understanding correct?

 R3 R4
  |X|
   R1
  /|\
  CCC

I also have more questions. I am not sure they are part of this RFE.
- How is ECMP configured? To make it work, ECMP needs to be configured in VMs, neutron routers and provider routers. Perhaps some routing protocol is needed.
- How about NAT and floating IP? If we don't consider active-active case for R1/R2 and only consider multiple external gateways, NAT and floating IP might work.

Regarding NAT (SNAT and floating IP), as Brian said, a neutron router does not do NAT when an external and internal networks belong to a same address scope. This mechanism is used for IPv6 networking and it also works for IPv4 network.

[Slawek Kaplonski (slaweq)]

On the last drivers meeting we were discussing that rfe: [Bug #1905391 “[RFE] VPNaaS support for OVN” : Bugs : neutron](https://bugs.launchpad.net/neutron/+bug/1905391)
We decided about couple of things which should be done next:

1. clarify the RFE and doubts raised by Liu, Brian and Akihiro above,
2. Propose some PoC which will show how complex it will be to do such change e.g. for legacy routers (and if it's possible at all) and explore alternative solution with announcing through BGP IPs from internal networks.

After that we will get back to that discussion in the drivers meeting.

[Bence Romsics (bence-romsics)]

Thanks for all the questions and input, both here and on the l3 and drivers meetings.

I hope to answer all questions, but I will need multiple takes to get there.

Let me start with exploring alternative (3) first, that is: Can we use router interfaces instead of external gateway networks?

For a long time I believed that you can only add an external network into a router's external_gw_info. And I also believed that you can only add an internal network's subnet as a router interface.

Now it seems to me I was wrong in the latter. (So now I'm careful with not using a router's external gw and external net interchangeably.) I tested it with l3-agent and it is actually possible:

* to add an external network's subnet as a router interface
* to add multiple external networks' subnets as router interfaces
* to have connectivity from an internal network to all of these external networks

For me this is quite in contrast with our documentation. Or at least I don't understand what "internal" means below:

https://docs.openstack.org/api-ref/network/v2/index.html?expanded=add-interface-to-router-detail#routers-routers

"""
PUT /v2.0/routers/{router_id}/add_router_interface
Adds an internal interface to a logical router. This means a specified subnet is attached to a router as an internal router interface.
"""

So before I dive into alternative (3) deeper, I must ask: Do we consider this (adding an external net's subnet as a router interface) a feature or an accident of the neutron API?

If it is a feature then alternative (3) may be realistic for me given that:

* the number of router interfaces to external nets are not limited to 1
* my use case does not require Floating IPs or SNAT

However there are multiple further considerations:

* neutron-dynamic-routing does not advertise a single route in a configuration like this. I checked the implementation and it looks for ports with device_owner='router_gateway':

https://opendev.org/openstack/neutron-dynamic-routing/src/commit/4212b705386153c5ff9986068c78ee9c7b4fbe4c/neutron_dynamic_routing/db/bgp_db.py#L839

Of which we don't have any of course, that's why there are no routes advertised. If we take the path of alternative (3) can we consider this a bug of neutron-dynamic-routing? Changing neutron-dynamic-routing to advertise additional routes sounds significantly smaller work and has way less impact than extending core l3 APIs. That sounds good to me.

* Can we allow a router to have the same external network in its external_gw_info and added as a router interface? Can we ever allow/implement an external net as a router interface on a distributed router? My use case does not require any of these. However at the moment I don't have good answers here.

* Shall we change our api-ref to explicit support external networks as router interfaces?

That's all I have about alternative (3) now.

For the record this is what I tested (it all worked):

openstack address scope create scope0
openstack subnet pool create --address-scope scope0 --pool-prefix 10.0.0.0/8 --default-prefix-length 24 pool0

openstack router create router0

devstack # sudo ip link set up dev br-physnet0
devstack # sudo ip address add 10.0.0.1/...

[Bence Romsics (bence-romsics)]

To clarify the use case (for which I'm trying to find a proper representation in the Neutron API):

Please see this etherpad for a better figure (lines 28-46):
https://etherpad.opendev.org/p/neutron-multiple-external-gateways

C-s are computes.
R1 and R2 are two sides of a Neutron-managed active-active HA router (the backend is an HA L3 switch).
R3 and R4 are two sides of an HA provider router outside the cloud.

The links between the computes and R1-R2 are a kind of an MLAG. Traffic is hashed on the compute, roughly half of it is sent to R1, the rest to R2. R1 and R2 runs an active-active router. There's one gateway IP, that's alive on both sides. When there's no fault both sides of R1-R2 can directly forward its share of traffic without sending it over an Inter-Switch-Link to the other side.

The links between R1-R2 and R3-R4 are point-to-point links between the two HA routers. Active-active use of these links via ECMP is required.

Address scopes are to be used, so NAT is not needed as Brian said.

Floating IPs are not required - made unnecessary by the use of BGP.

To Brian's question in comment #6: I need to find a way to represent the 4 links between the cloud and provider routers.

To Akihiro's questions in comment #7: I hope the above use case description answers your questions. Let me know please if I still missed something.

I know I did not answer some of the questions from the driver's meeting yet. However I hope there's enough information to decide which alternative is better suited to be explored in depth:

* allowing multiple external gateways or
* using router interfaces to subnets of external networks and making the implied changes in neutron-dynamic-routing (so routes to tenant networks can still be advertised in this setup).

[Akihiro Motoki (amotoki)]

Thanks Bence for the clarification.
Regarding #10, you need active-active router interfaces in neutron. You also would like to have multiple (external) networks ("external" here means "outgoing" networks for a cloud).
What I am not sure is whether multiple routers on a single network (for "outgoing") does not satisfy your need.

The reason I asked this is because I previously heard a bit different need that operators would like to make external routes redundant (i.e. a single neutron router would like to have multiple external connection). This is a similar need, but your requirements sounds different from two points: active-active neutron router instances AND multiple external networks.

[Akihiro Motoki (amotoki)]

I agree the definition of "external" network is ambiguous and confusing.

In my understanding, "external" network is used as a kind of flags.
I see several meanings in "external".
- Traffic to "external" network is NAT-ed (including floating IPs)
- "external" is a pool of floating IPs
- A router interface connected to an external network is considered as "default route"

"internal" is just used as NOT "external". I think there is nothing more than that.
An external network can be a router gateway but there is no rule that an external network cannot be a router interface.

That's my current understanding.

[Slawek Kaplonski (slaweq)]

On the last drivers meeting we decided to approve that RFE and continue in the "allowing multiple external gateways" direction. Please propose spec and we will discuss details of that proposal there.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/779511
Committed: https://opendev.org/openstack/neutron-specs/commit/dceafdb9880ab4c11100a79a8b4bcd32694d9ca9
Submitter: "Zuul (22348)"
Branch: master

commit dceafdb9880ab4c11100a79a8b4bcd32694d9ca9
Author: Bence Romsics <email address hidden>
Date: Tue Mar 9 10:25:21 2021 +0100

    Allow multiple external gateways

    Change-Id: I4c66627e6c27be5d9e7e54862788e09403befdbc
    Related-Bug: #1905295

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/802029

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/802029
Committed: https://opendev.org/openstack/neutron-lib/commit/75057767eb98c302524e566c79e251ab334af8a1
Submitter: "Zuul (22348)"
Branch: master

commit 75057767eb98c302524e566c79e251ab334af8a1
Author: Bence Romsics <email address hidden>
Date: Thu Jan 14 15:12:13 2021 +0100

    multi-ext-gw: api-def and api-ref

    New router attribute:
        external_gateways

    New router actions:
        PUT add_external_gateways
        PUT update_external_gateways
        PUT remove_external_gateways

    Change-Id: I46381b70f770ccd32943644341388b09f7e14556
    Partial-Bug: #1905295
    Related-Change (spec): https://review.opendev.org/c/openstack/neutron-specs/+/779511

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

RFE implementation not attended, reopen if needed.