[RFE] Active-active L3 Gateway with Multihoming

[Dmitrii Shcherbakov]

Some network designs include multiple L3 gateways to:

* Share the load across different gateways;
* Provide independent network paths for the north-south direction (e.g. via
  different ISPs).

Having multi-homing implemented at the instance level imposes additional burden
on the end user of a cloud and support requirements for the guest OS, whereas
utilizing ECMP and BFD at the router side alleviates the need for instance-side
awareness of a more complex routing setup.

Adding more than one gateway port implies extending the existing data model
which was described in the multiple external gateways spec (https://specs.openstack.org/openstack/neutron-specs/specs/xena/multiple-external-gateways.html). However, it left
adding additional gateway routes out of scope leaving this to future
improvements around dynamic routing. Also the focus of neutron-dynamic-routing
has so far been around advertising routes, not accepting new ones from the
external peers - so dynamic routing support like this is a very different
subject. However, manual addition of extra routes does not utilize the default
gateway IP information available from subnets in Neutron while this could be
addressed by implementing an extra conditional behavior when adding more than
one gateway port to a router.

ECMP routes can result in black-holing of traffic should the next-hop of a
route becomes unreachable. BFD is a standard protocol adopted by IETF
for next-hop failure detection which can be used for route eviction. OVN
supports BFD as of v21.03.0 (https://github.com/ovn-org/ovn/commit/6e0a69ad4bcdf9e4cace5c73ef48ab06065e8519) with a data model that allows enabling
BFD on a per next-hop basis by associating BFD session information with routes,
however, it is not modeled at the Neutron level even if a backend supports it.

From the Neutron data model perspective, ECMP for routes is already a supported
concept since ECMP support spec got implemented (https://specs.openstack.org/openstack/neutron-specs/specs/wallaby/l3-router-support-ecmp.html) in Wallaby (albeit the
spec focused on the L3-agent based implementation only).

As for OVN and BFD, the OVN database state needs to be populated by Neutron
based on the data from the Neutron database, therefore, data model changes to
the Neutron DB are needed to represent the BFD session parameters.

---

The previous work on multiple gateway ports did not get completed and the neutron-lib changes were reverted. Likewise, the scope of this RFE is bigger with some overlap and augmentation compared to prior art. The spec will follow for this RFE with more details as to how the data model and API changes are proposed to be made.

Upd: https://review.opendev.org/c/openstack/neutron-specs/+/870030

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-specs/+/870030

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/870887

[Lajos Katona (lajos-katona)]

Hi, interesting proposal, thanks for it.
There was a spec for BFD also:
https://specs.openstack.org/openstack/neutron-specs/specs/xena/bfd_support.html, not sure it is useful as we aimed OVS originally.

The dynamic-routing/BGP spec perhaps also partially covered the things you need:
https://specs.openstack.org/openstack/neutron-specs/specs/xena/bgpaas-enhancements.html

[Dmitrii Shcherbakov (dmitriis)]

Hi Lajos, thanks a lot for the references! I can definitely use the prior art on the BFD API and data model.

The BGP peering support is the next logical step so it's quite useful too.

[Brian Haley (brian-haley)]

Let's talk about this at the drivers meeting next week.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/873698

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/873699

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/874199

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/874760

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/878527

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/878531

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/870030
Committed: https://opendev.org/openstack/neutron-specs/commit/9763752c73644be92f9a2597c198968d88c1d220
Submitter: "Zuul (22348)"
Branch: master

commit 9763752c73644be92f9a2597c198968d88c1d220
Author: Dmitrii Shcherbakov <email address hidden>
Date: Thu Jan 12 22:50:52 2023 +0300

    Active-active L3 Gateway with Multihoming

    The aim of the RFE is to add multihoming support to routers along with
    automatic management of ECMP for default routes and BFD next-hop
    reachability verification.

    https://bugs.launchpad.net/neutron/+bug/2002687

    Co-Authored-By: Frode Nordahl <email address hidden>
    Related-Bug: #2002687
    Change-Id: I95a0d5f1b7aef985df5625cd83222799db811f2b

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/879462

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/870887
Committed: https://opendev.org/openstack/neutron-lib/commit/e52a9372f76aa104543ab04cf1723e58da17d379
Submitter: "Zuul (22348)"
Branch: master

commit e52a9372f76aa104543ab04cf1723e58da17d379
Author: Dmitrii Shcherbakov <email address hidden>
Date: Wed Jan 18 02:45:03 2023 +0300

    ext-gw-multihoming: api-def and api-ref

    API additions for [1].

    * Added a new router attribute: external_gateways;
    * Added new API definitions for:
        PUT add_external_gateways
        PUT update_external_gateways
        PUT remove_external_gateways
    * Added extensions for each of the new router-level attributes:
      * enable_default_route_ecmp
      * enable_default_route_bfd
    * Combined the validation logic for the external_gateway_info type
      across extensions (l3_ext_gw_mode, qos_gateway_ip and the new
      extension called l3_ext_gw_multihoming).

    [1] https://review.opendev.org/c/openstack/neutron-specs/+/870030/

    Change-Id: I2618475636b2bb9bfd743a62f5d4859d4f68a547
    Related-Bug: #2002687

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/877831
Committed: https://opendev.org/openstack/neutron/commit/d67d1c273668ad4bb0d6906b7684f4a7b095c8d4
Submitter: "Zuul (22348)"
Branch: master

commit d67d1c273668ad4bb0d6906b7684f4a7b095c8d4
Author: Frode Nordahl <email address hidden>
Date: Fri Mar 17 18:42:53 2023 +0100

    [ovn] Drop use of OVN_GW_PORT_EXT_ID_KEY

    At present Neutron maintains an external_id on the
    Logical_Router (LR) representing the gw port (singluar). This
    is problematic when introducing multiple gateway ports.

    Instead we can find Logical Router Port (LRP) that act as
    gateways for the LR at runtime by looking for configuration
    present on all GW ports.

    Partial-Bug: #2002687
    Signed-off-by: Frode Nordahl <email address hidden>
    Needed-By: I95a0d5f1b7aef985df5625cd83222799db811f2b
    Change-Id: I8a915dca1410c70bdfe7a2d72931921d2a1a265e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/877712
Committed: https://opendev.org/openstack/neutron/commit/e5d4499672fe4e4e57a24ae3194d3adaefe7be15
Submitter: "Zuul (22348)"
Branch: master

commit e5d4499672fe4e4e57a24ae3194d3adaefe7be15
Author: Frode Nordahl <email address hidden>
Date: Thu Mar 16 09:48:02 2023 +0100

    [ovn] Drop use of LR OVN_GW_NETWORK_EXT_ID_KEY

    An update to the OVN QoS driver to support the `qos_gateway_ip`
    QoS extension [0] introduced adding the GW network id as an
    external_id on the Logical_Router (LR). This is problematic
    when introducing multiple gateway ports, because a single LR
    can have gateways in multiple networks.

    The external_id key was presumably added because at the point in
    time when a LR is deleted, the code had no other source of this
    information. However, it turns out this step is redundant and
    not neccessary.

    To prove this I include a excerpt of a stack trace when deleting
    a router in the commit message:

        File "services/ovn_l3/plugin.py", line 210, in delete_router
          super(OVNL3RouterPlugin, self).delete_router(context, id)
        File "db/l3_db.py", line 612, in delete_router
          self._delete_current_gw_port(context, id, router, None)
        File "db/l3_db.py", line 452, in _delete_current_gw_port
          self._core_plugin.delete_port(
        File "plugins/ml2/drivers/ovn/mech_driver/mech_driver.py",
            line 886, in delete_port_postcommit
          self._ovn_client.delete_port(context.plugin_context, port['id'],
        File "plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py",
            line 830, in delete_port
          self._delete_port(port_id, port_object=port_object)

    Essentially, a routers GW port(s) will be removed prior to
    deleting the router itself.

    The `ovn_client.delete_port` method will call on the QoS extension
    to remove rules matching the GW port, and that will be the same
    rules as has previously been added for the router.

    I also added a functional test that confirms this fact [1].

    0: I46864b9234af64f190f6b6daebfd94d2e3bd0c17
    1: Ic92a7b3bd73920d08dee41974bfe3aeb1c64b557

    Partial-Bug: #2002687
    Signed-off-by: Frode Nordahl <email address hidden>
    Needed-By: I95a0d5f1b7aef985df5625cd83222799db811f2b
    Change-Id: If7c22bc8a95fa13e746c86a1e9d4a6fa25496e1f

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/878527
Committed: https://opendev.org/openstack/neutron/commit/5510cdab92d1d3eac080053a51779d34a8a19614
Submitter: "Zuul (22348)"
Branch: master

commit 5510cdab92d1d3eac080053a51779d34a8a19614
Author: Frode Nordahl <email address hidden>
Date: Fri Mar 24 12:06:39 2023 +0100

    [ovn] OVNClient._get_router_ports: Drop unused parameter

    The `get_gw_port` parameter is currently unused, the
    implementation hiding behind it also contains a bug, remove it.

    Partial-Bug: #2002687
    Signed-off-by: Frode Nordahl <email address hidden>
    Change-Id: Ie0ba5a478fabe9880746f892ef9c00d7e5660195

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/879462
Committed: https://opendev.org/openstack/neutron/commit/b1cc242faddd4c378dfee7b65ed72b4b190b2467
Submitter: "Zuul (22348)"
Branch: master

commit b1cc242faddd4c378dfee7b65ed72b4b190b2467
Author: Dmitrii Shcherbakov <email address hidden>
Date: Tue Apr 18 20:04:06 2023 +0300

    Add a method to retrieve router gateway ports

    A method is added as opposed to having a synthetic field on a router for
    performance reasons: gateways will only be queried when needed to use
    the external gateways feature API calls.

    Partial-Bug: #2002687
    Change-Id: Iddde9d986b024109bdb7c2aa777a1b017b6a35ab

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/873593
Committed: https://opendev.org/openstack/neutron/commit/a221764751de05e42069f1c097b1025bd9c4fc52
Submitter: "Zuul (22348)"
Branch: master

commit a221764751de05e42069f1c097b1025bd9c4fc52
Author: Dmitrii Shcherbakov <email address hidden>
Date: Mon Feb 13 18:30:15 2023 +0300

    Allow Multiple External Gateways

    * Add a new API for adding/updating/removing multiple gateway ports
      on routers;

    * Implement the necessary backend changes.

    Partial-Bug: #2002687
    Depends-On: I2618475636b2bb9bfd743a62f5d4859d4f68a547
    Change-Id: Id885565e88f6f1898ca5cfac709a24dd62605d1a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/874797
Committed: https://opendev.org/openstack/neutron/commit/89702218db2476a17f6ff36cf51b909db563d887
Submitter: "Zuul (22348)"
Branch: master

commit 89702218db2476a17f6ff36cf51b909db563d887
Author: Dmitrii Shcherbakov <email address hidden>
Date: Wed Feb 22 21:11:31 2023 +0300

    Add extra router attributes for ECMP and BFD

    * enable_default_route_ecmp
    * enable_default_route_bfd

    Partial-Bug: #2002687
    Change-Id: I3fcd0458d20f20ce40378f90f073f37c41400865

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/893023

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/893025

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/893026

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/893181

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/893182

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/893184

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/893185

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/893181

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/893182

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/893184
Committed: https://opendev.org/openstack/neutron/commit/c6b6ecc75196bd47ebcea0284e7d714c861fa630
Submitter: "Zuul (22348)"
Branch: master

commit c6b6ecc75196bd47ebcea0284e7d714c861fa630
Author: Frode Nordahl <email address hidden>
Date: Wed Aug 30 15:28:54 2023 +0200

    Drop release notes for l3-ext-gw-multihoming and adjacent features

    We did unfortunately not make it into Bobcat, and will try again
    to get it into Caracal.

    Partial-Bug: #2002687
    Change-Id: I5b4579a96152b8bdb1d34e59fb492c6f2a01b71e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/893023
Committed: https://opendev.org/openstack/neutron/commit/113f3f668952d4bd2b78ef08c04fb606ca8fc0a5
Submitter: "Zuul (22348)"
Branch: master

commit 113f3f668952d4bd2b78ef08c04fb606ca8fc0a5
Author: Frode Nordahl <email address hidden>
Date: Mon Aug 28 16:38:31 2023 +0200

    Add missing extension classes for router BFD/ECMP extra attributes

    Change I3fcd0458d20f20ce40378f90f073f37c41400865 added the
    implementation for router BFD/ECMP extra attributes, but omitted
    the APIExtensionDescriptor classes that are required for loading
    the extension.

    Partial-Bug: #2002687
    Change-Id: I5f59087a1ff8d37f136ac88e50e0246de68455a8

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/893025
Committed: https://opendev.org/openstack/neutron/commit/8219f3f61b17b8fa62cc8ae20f5e8c0307b71cff
Submitter: "Zuul (22348)"
Branch: master

commit 8219f3f61b17b8fa62cc8ae20f5e8c0307b71cff
Author: Frode Nordahl <email address hidden>
Date: Tue Aug 29 08:18:10 2023 +0200

    Get default value for BFD/ECMP extra attributes from config

    The default value for 'enable_default_route_bfd' and
    'enable_default_route_ecmp' is configurable.

    Ensure the default comes from configuration if a request to create
    router does not contain a value for these attributes.

    Partial-Bug: #2002687
    Change-Id: I581f4c5e5cfa275f8a6f0adec405f205e877ac55

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/893026
Committed: https://opendev.org/openstack/neutron-lib/commit/a29e4a513eac3c48c0399a3e9b65509b87df260e
Submitter: "Zuul (22348)"
Branch: master

commit a29e4a513eac3c48c0399a3e9b65509b87df260e
Author: Frode Nordahl <email address hidden>
Date: Tue Aug 29 11:46:24 2023 +0200

    Update default for BFD/ECMP router extra attributes

    During review of the series it was requested to make the default
    values for 'enable_default_route_bfd' and
    'enable_default_route_ecmp' configurable.

    In order to determine the correct value at runtime, Neutron needs
    the attributes to be filled with something other than 'True' or
    'False' when not provided in the request.

    Needed-By: I581f4c5e5cfa275f8a6f0adec405f205e877ac55
    Related-Bug: #2002687
    Change-Id: I9096685fb79a84e11a8547a5aaa16f7f2df48a56

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/874199
Committed: https://opendev.org/openstack/neutron/commit/0bc9a713872d9639b234eaffce9f43c2fd4be98f
Submitter: "Zuul (22348)"
Branch: master

commit 0bc9a713872d9639b234eaffce9f43c2fd4be98f
Author: Frode Nordahl <email address hidden>
Date: Tue Mar 21 08:54:22 2023 +0100

    [ovn] Implement support for external-gateway-multihoming extension

    The general approach is to update the ovn_client and ovn_db_sync
    code to handle multiple gateway ports wherever a single gateway
    port is handled today.

    In this patch set multiple static routes for default gateway will
    be added by default when multiple gw ports are present. Support
    for the `enable_default_route_ecmp` attribute to control this
    behavior will be added in subsequent patch set to avoid making
    this change too large.

    Partial-Bug: #2002687
    Change-Id: I00b1f29172be5a0034b921b11af3a8d502273766

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/878531
Committed: https://opendev.org/openstack/neutron/commit/4ccc296f56853fdea0e71d02db96a9a2fc3aa882
Submitter: "Zuul (22348)"
Branch: master

commit 4ccc296f56853fdea0e71d02db96a9a2fc3aa882
Author: Frode Nordahl <email address hidden>
Date: Fri Mar 24 14:38:16 2023 +0100

    [ovn] Honor `enable_default_route_ecmp` attribute

    Only create multiple default routes when the router has the
    `enable_default_route_ecmp` attribute set to True.

    Partial-Bug: #2002687
    Signed-off-by: Frode Nordahl <email address hidden>
    Change-Id: I17512ec494661151e46282f0bd08abc55bef1df8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/910332

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/910435

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/874760
Committed: https://opendev.org/openstack/neutron/commit/0bae4b70b6fc53cf5502622677a95ee50cc319a1
Submitter: "Zuul (22348)"
Branch: master

commit 0bae4b70b6fc53cf5502622677a95ee50cc319a1
Author: Frode Nordahl <email address hidden>
Date: Wed Jun 21 08:59:24 2023 +0200

    [ovn] Make scheduling of unhosted gateways aware of current transaction

    At present, whenever multiple additions/updates are made to LRPs
    with gateway_chassis, each update is put in separate transactions
    in an attempt to ensure the scheduler operates on updated
    information for each iteration.

    This is problematic because we don't have the luxury of creating
    separate transactions for updates in all parts of the code base,
    and it is also not very efficient.

    The OVSDBapp library wraps the OVS Python IDL and provides
    different semantics. Most notably the OVSDBapp represents a
    Transaction as a list of command objects with `run_idl` methods
    for execution at some point in the future. The main loop and the
    command objects are not aware of changes made in the current
    transaction until it is committed.

    Fortunately, as an ovsdbapp transaction is committed, the
    underlying OVS Python IDL is kept up to date during the course of
    the transaction [0][1][2].

    Move implementation of scheduling of unhosted gateways into an
    ovsdbapp command, using a plugin reference to the Neutron
    OVNClient class for any calls into the Neutron code, allowing
    scheduling decisions to be made on up to date data as the
    transaction is applied.

    0: https://github.com/openvswitch/ovs/blob/e3ba0be48ca4/python/ovs/db/idl.py#L1316
    1: https://github.com/openvswitch/ovs/blob/e3ba0be48ca4/python/ovs/db/idl.py#L1400
    2: https://github.com/openvswitch/ovs/blob/e3ba0be48ca4/python/ovs/db/idl.py#L2083

    Partial-Bug: #2002687
    Co-Authored-By: Terry Wilson <email address hidden>
    Co-Authored-By: Brian Haley <email address hidden>
    Signed-off-by: Frode Nordahl <email address hidden>
    Change-Id: I83bcf7fe838c0d6b0617c43439643e8443b2bdae

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/873698
Committed: https://opendev.org/openstack/neutron/commit/898498ca3ba268122dd5de43f2a0dbf2eb3c4058
Submitter: "Zuul (22348)"
Branch: master

commit 898498ca3ba268122dd5de43f2a0dbf2eb3c4058
Author: Frode Nordahl <email address hidden>
Date: Mon Feb 13 14:01:08 2023 +0100

    [ovn] Add helper for retrieving LR associated with LRP

    This will be used in the next patch set to implement anti-affinity
    scheduling for routers with multiple LRPs.

    Partial-Bug: #2002687
    Signed-off-by: Frode Nordahl <email address hidden>
    Change-Id: Iff958195f229f7e0714f1285bb3d53497aeec9aa

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/873699
Committed: https://opendev.org/openstack/neutron/commit/8df5ee61d980e955a46381ff9745b0585d846ef5
Submitter: "Zuul (22348)"
Branch: master

commit 8df5ee61d980e955a46381ff9745b0585d846ef5
Author: Frode Nordahl <email address hidden>
Date: Mon Feb 13 14:03:35 2023 +0100

    [ovn] Apply soft anti-affinity for LRs with multiple LRPs when scheduling

    Move scheduling logic from OVNClient._create_lrouter_port to an
    ovsdbapp command so that scheduling decisions are made on up to
    date information as the transaction applies.

    One of the main use cases for routers with multiple gateway
    ports are resiliency. Whenever there are multiple LRPs present in
    a LR, we want to ensure diverse placement of the ports to
    minimize impact of chassis failure.

    Partial-Bug: #2002687
    Signed-off-by: Frode Nordahl <email address hidden>
    Change-Id: I36860b739a8cb99ba0e7fc65950ea252ad6803c4

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/910435
Committed: https://opendev.org/openstack/neutron/commit/12ed429e7b41c7f1ce0901fdc4215216a80b9374
Submitter: "Zuul (22348)"
Branch: master

commit 12ed429e7b41c7f1ce0901fdc4215216a80b9374
Author: Frode Nordahl <email address hidden>
Date: Tue Feb 27 20:36:13 2024 +0100

    [ovn] Document the OVN L3 scheduler: Anti-Affinity

    Document the ``OVNGatewayLeastLoadedScheduler`` behavior when
    there are multiple gateway ports.

    Partial-Bug: #2002687
    Change-Id: I99cf269e35619a2e1fb680d8decbc613267ca62e
    Signed-off-by: Frode Nordahl <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/910332
Committed: https://opendev.org/openstack/neutron/commit/ae9749a5e3118fd5b69fd483705848328ab03211
Submitter: "Zuul (22348)"
Branch: master

commit ae9749a5e3118fd5b69fd483705848328ab03211
Author: Frode Nordahl <email address hidden>
Date: Tue Feb 27 08:45:34 2024 +0100

    [ovn] Ensure all routes are deleted when deleting ExtGw

    The current DeleteLRouterExtGwCommand stops iterating over static
    routes once the first route with a ovn_const.OVN_ROUTER_IS_EXT_GW
    external_id is found.

    There can be multiple static routes with this external ID, so
    this patch continues iteration until all external routes are
    removed and adds a unit test for this condition.

    Trivial-Fix
    Partial-Bug: #2002687
    Change-Id: Ie43abd8bf511e12a0f64c10bafeaafc0823a2076
    Signed-off-by: Frode Nordahl <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/909955
Committed: https://opendev.org/openstack/neutron/commit/743bd1ccefa0735a0b9460d1c231a66b372c99d0
Submitter: "Zuul (22348)"
Branch: master

commit 743bd1ccefa0735a0b9460d1c231a66b372c99d0
Author: Frode Nordahl <email address hidden>
Date: Fri Feb 23 09:17:30 2024 +0100

    [ovn] Ensure OVN DB update on change of number of GW ports

    If a router is created and a GW port is subsequently added by
    updating the router, the change is not always propagated to the
    OVN DB.

    This patch fixes this and also adds a functional test case.

    Trivial-Fix
    Partial-Bug: #2002687
    Signed-off-by: Frode Nordahl <email address hidden>
    Change-Id: I9455678d73fb35b77eac7416917200a419abfa84

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/878543
Committed: https://opendev.org/openstack/neutron/commit/cc1ff09b9e2e64eb6abd519a0d4ca3dda7439eb1
Submitter: "Zuul (22348)"
Branch: master

commit cc1ff09b9e2e64eb6abd519a0d4ca3dda7439eb1
Author: Frode Nordahl <email address hidden>
Date: Fri Mar 24 17:07:36 2023 +0100

    [ovn] Add support for enable_default_route_bfd attribute

    When ``enable_default_route_bfd`` is set, maintain BFD records
    along with default route records. Default route records will
    also be fitted with the `output_port` key, which is a requirement
    for the OVN BFD support.

    Partial-Bug: #2002687
    Signed-off-by: Frode Nordahl <email address hidden>
    Change-Id: I34e2453ab206c13c3ca40c4181970c320bdd8e67

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/893185
Committed: https://opendev.org/openstack/neutron/commit/f16310515860beac14ec04dddb97573a69181ecd
Submitter: "Zuul (22348)"
Branch: master

commit f16310515860beac14ec04dddb97573a69181ecd
Author: Frode Nordahl <email address hidden>
Date: Wed Aug 30 15:29:46 2023 +0200

    Add release notes for l3-ext-gw-multihoming and adjacent features

    These features was added to Caracal.

    Partial-Bug: #2002687
    Change-Id: Id5b0b230e342074d8ca27e4fc4ce1c1f0c1b0d34

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-specs/+/911399

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/911399
Committed: https://opendev.org/openstack/neutron-specs/commit/5852b695efde30a727ec3b9a044589c4f0244d66
Submitter: "Zuul (22348)"
Branch: master

commit 5852b695efde30a727ec3b9a044589c4f0244d66
Author: Frode Nordahl <email address hidden>
Date: Wed Mar 6 07:45:15 2024 +0100

    Move Active-active L3 Gateway with Multihoming to 2024.1

    The feature was fully implemented in the 2024.1 cycle.

    Related-Bug: #2002687
    Change-Id: I4e69bdf2ac9da1154d3847f3191b110f09130e02
    Signed-off-by: Frode Nordahl <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/899402
Committed: https://opendev.org/openstack/neutron/commit/0199a8457b4e813fa1eb6235ad68b49a71725b8f
Submitter: "Zuul (22348)"
Branch: master

commit 0199a8457b4e813fa1eb6235ad68b49a71725b8f
Author: Frode Nordahl <email address hidden>
Date: Thu Oct 26 15:52:18 2023 +0200

    Add documentation for aa-l3-gw-multihoming

    Closes-Bug: #2002687
    Depends-On: I4e69bdf2ac9da1154d3847f3191b110f09130e02
    Signed-off-by: Frode Nordahl <email address hidden>
    Change-Id: I717ca97164eb9a34bb1095c6222f9879017af5ca