[RFE] unmanaged dynamic router resources - OVN

[Roberto Bartzen Acosta]

Problem description
--------------------------------------
Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, we are testing the use of the OVN-IC to interconnect workloads in multiple AZs - with different OpenStacks deployments.

The Neutron default design does not allow to interconnect workloads between different OpenStacks natively, and this could be a requirement for a high availability cloud solution (if we are talking about Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud solutions that use OVN as network backend - ovn-kube case.

We tested an OVN interconnect integrated with 3 OpenStack deployments and it worked very well !!! in this case, we are considering direct L3 traffic at the router level between different network infrastructures.

To make it work we need to configure the TS and the LRP manually, as well as examples from the ovn-kube project [2]. The problem with snat (and FIPs) that was reported in the ovn-kube project has already been fixed in OVN version 22.09, and in newer OVN versions it is not necessary to modify anything in Neutron to pass the (--gateway-port) because OVN finds the gateway port automatically.

At the moment the only issue found in Neutron is related to DB sync, and it is natural because the LRP connected to the TS does not exist in the DB. If the operator needs to restore the Neutron DB, the SYNC repair command will remove the unmanaged externally added resources.

Note: The route learning has been tested with IPv4 and IPv6 addresses and is working fine. A detail in the case of IPv6 is related to the filter of routes learned via LLC addresses [3], take care of this case.

SYNC_REPAIR - problem

* Static Routes (learned OVN-IC routes)
* Router Port -> Transit Switches

Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1
Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': '10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron

-------------------------------------

Proposed solution:
--------------------------------------

This RFE intends to implement a filter in the OVN mech_driver to validate the external_ids key and not remove LRP's and static routes present in the OVN backend without Neutron "key" in external_ids register.

sync_routers_and_rports method:

LRPs case:

Filter the port list when iterating over existing OVN LRPs before checking for existence in the Neutron DB.

LRP created by Neutron - example

_uuid : 1266061f-2a0b-4cb7-bcc4-14cb61a85173
enabled : []
external_ids : {"neutron:network_name"=neutron-d5169427-8fa0-4a25-a679-b33c604dbee1, "neutron:revision_number"="3", "neutron:router_name"="68917dd3-de44-465d-bf85-2722ca742ec0", "neutron:subnet_ids"="1ad00870-1efc-4eb0-b189-d69fbfae626f"}
gateway_chassis : []
ha_chassis_group : []
ipv6_prefix : []
ipv6_ra_configs : {}
mac : "fa:16:3e:d4:63:c5"
name : lrp-3581bee6-ea95-4971-862f-378546244487
networks : ["192.168.1.1/24"]
options : {}
peer : []

LRP externally created - example

ovn-nbctl list logical_router_port rt1-admin-tenant1
_uuid : 6a8bbf7b-4bf6-46a8-b0be-631154b87446
enabled : []
external_ids : {}
gateway_chassis : [b4487769-273e-4ba9-abd5-4743ff987f74]
ha_chassis_group : []
ipv6_prefix : []
ipv6_ra_configs : {}
mac : "aa:aa:aa:aa:ab:01"
name : rt1-admin-tenant1
networks : ["169.254.100.11/24", "fd00::1/64"]
options : {}
peer : []

LRPs created manually do not have Neutron keys in external_ids, the idea here is to filter the return list and use the resource only when external_ids contain Neutron keys.

Static routes case:

Additionally, OVN sync_db composes a list of static routes linked to a router (get_all_logical_routers_with_rports). The proposal is to extend the Neutron key filter in the external ids when creating the return list. Similar to the router port case.

ovn-nbctl lr-route-list 078fd69b-f4c7-4469-a900-918d0a229bd1
IPv4 Routes
Route Table <main>:
              10.0.1.0/24 169.254.100.12 dst-ip (learned)
              10.0.2.0/24 169.254.100.13 dst-ip (learned)
                0.0.0.0/0 200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
          2001:db8:1::/64 fd00::2 dst-ip (learned)
          2001:db8:1::/64 fd00::3 dst-ip (learned)
    2801:80:3ea0:822::/64 fd00::2 dst-ip (learned)
    2801:80:3ea0:823::/64 fd00::3 dst-ip (learned)
                     ::/0 2001:db8:1:: dst-ip

With these specific changes, the management of these LRPs and learned routes will be completely disassociated from Neutron perspective, and the resources can be managed by the operator, creating and removing links with remote routers on demand.

--------------------------------------

Regards,
Roberto

[1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.html
[2] https://github.com/kubeovn/kube-ovn/blob/v1.11.0/docs/cluster-interconnection.md

[3] https://github.com/ovn-org/ovn/commit/cb0e2b3f44daeafb2f02f07289e3c410ee6ead28

------------------------------------------------------------------

Additional logs - Interconnect use case:

OpenStack 1

root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498
IPv4 Routes
Route Table <main>:
              20.0.1.0/24 169.254.200.2 dst-ip (learned)
              20.0.2.0/24 169.254.200.3 dst-ip (learned)
                0.0.0.0/0 200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0 fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07
IPv4 Routes
Route Table <main>:
              10.0.1.0/24 169.254.100.2 dst-ip (learned)
              10.0.2.0/24 169.254.100.3 dst-ip (learned)
                0.0.0.0/0 200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0 fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-f931b37c:~#

OpenStack 2

root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc
IPv4 Routes
Route Table <main>:
              20.0.0.0/24 169.254.200.1 dst-ip (learned)
              20.0.2.0/24 169.254.200.3 dst-ip (learned)
                0.0.0.0/0 200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0 fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a
IPv4 Routes
Route Table <main>:
              10.0.0.0/24 169.254.100.1 dst-ip (learned)
              10.0.2.0/24 169.254.100.3 dst-ip (learned)
                0.0.0.0/0 200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0 fc00:ca5a:ca5a:8000:: dst-ip

OpenStack 3

root@os-infra-1-neutron-ovn-northd-container-f237db97:~#
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3
IPv4 Routes
Route Table <main>:
              20.0.0.0/24 169.254.200.1 dst-ip (learned)
              20.0.1.0/24 169.254.200.2 dst-ip (learned)
                0.0.0.0/0 200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0 fc00:ca5a:ca5a:8000:: dst-ip
root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0
IPv4 Routes
Route Table <main>:
              10.0.0.0/24 169.254.100.1 dst-ip (learned)
              10.0.1.0/24 169.254.100.2 dst-ip (learned)
                0.0.0.0/0 200.200.200.1 dst-ip

IPv6 Routes
Route Table <main>:
                     ::/0 fc00:ca5a:ca5a:8000:: dst-ip

OVN-IC Global database

root@ovn-global-db1:~# ovn-ic-sbctl show
availability-zone osp1
    gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5
        hostname: osp1-gwnode1
        type: geneve
            ip: 192.168.200.28
        port admin-rt1-tenant1
            transit switch: admin-tenant1
            address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"]
        port admin-rt1-tenant1_1
            transit switch: admin-tenant1_1
            address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"]
availability-zone osp2
    gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8
        hostname: osp2-gwnode1
        type: geneve
            ip: 192.168.200.128
        port admin-rt2-tenant1
            transit switch: admin-tenant1
            address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"]
        port admin-rt2-tenant1_1
            transit switch: admin-tenant1_1
            address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"]
availability-zone osp3
    gateway 97595af9-7896-40d0-a883-beadbff1aa5b
        hostname: osp3-gwnode1
        type: geneve
            ip: 192.168.200.228
        port admin-rt3-tenant1
            transit switch: admin-tenant1
            address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"]
        port admin-rt3-tenant1_1
            transit switch: admin-tenant1_1
            address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"]

--------------------------------------

Reference design:

# Global database OVN-IC

ovn-ic-nbctl ts-add admin-tenant1

**** OpenStack 1 ***********

# OVN central 1

ovn-nbctl set NB_Global . name=osp1

ovn-nbctl set NB_Global . options:ic-route-adv=true \
                            options:ic-route-learn=true

ovn-nbctl lrp-add NEUTRON_ROUTER rt1-admin-tenant1 aa:aa:aa:aa:aa:01 169.254.100.1/24
ovn-nbctl lsp-add admin-tenant1 admin-rt1-tenant1 -- \
        lsp-set-addresses admin-rt1-tenant1 router -- \
        lsp-set-type admin-rt1-tenant1 router -- \
        lsp-set-options admin-rt1-tenant1 router-port=rt1-admin-tenant1

ovn-nbctl lrp-set-gateway-chassis rt1-admin-tenant1 832b6c0d-13ce-4600-ab37-78516d8ec4c5 1

ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24

# Gateway node - Openstack 1

ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true

**** OpenStack 2 ***********

ovn-nbctl set NB_Global . name=osp2

ovn-nbctl set NB_Global . options:ic-route-adv=true \
                            options:ic-route-learn=true

ovn-nbctl lrp-add NEUTRON_ROUTER rt2-admin-tenant1 aa:aa:aa:aa:aa:02 169.254.100.2/24
ovn-nbctl lsp-add admin-tenant1 admin-rt2-tenant1 -- \
        lsp-set-addresses admin-rt2-tenant1 router -- \
        lsp-set-type admin-rt2-tenant1 router -- \
        lsp-set-options admin-rt2-tenant1 router-port=rt2-admin-tenant1

ovn-nbctl lrp-set-gateway-chassis rt2-admin-tenant1 17ffabdf-cf47-41ab-9539-d326c13c4ca8 1

ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24

# Gateway node

ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true

**** OpenStack 3 ***********

ovn-nbctl set NB_Global . name=osp3

ovn-nbctl set NB_Global . options:ic-route-adv=true \
                            options:ic-route-learn=true

ovn-nbctl lrp-add NEUTRON_ROUTER rt3-admin-tenant1 aa:aa:aa:aa:aa:03 169.254.100.3/24
ovn-nbctl lsp-add admin-tenant1 admin-rt3-tenant1 -- \
        lsp-set-addresses admin-rt3-tenant1 router -- \
        lsp-set-type admin-rt3-tenant1 router -- \
        lsp-set-options admin-rt3-tenant1 router-port=rt3-admin-tenant1

ovn-nbctl lrp-set-gateway-chassis rt3-admin-tenant1 97595af9-7896-40d0-a883-beadbff1aa5b 1

ovn-nbctl set NB_Global . options:ic-route-blacklist=200.200.200.0/24

# Gateway node

ovs-vsctl set open_vswitch . external_ids:ovn-is-interconn=true

[Miguel Lavalle (minsel)]

@Roderto,

If approved, will you implement this RFE?

[Roberto Bartzen Acosta (rbartzen)]

Hello Miguel,

If approved, I will implement it.

Regards

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

This RFE has been approved today [1]. A spec will be required.

Thanks for this proposal.

[1]https://meetings.opendev.org/meetings/neutron_drivers/2023/neutron_drivers.2023-07-21-14.01.log.txt

[Liu Xie (liushy)]

Hi Roberto,
I have some questions:
1.How to deploy the global ovn-ic db(eg.ovn-ic nb/sb) and tunnels between two clouds?
2.Dose it affect the routes of neutron-router? If it is, how to fixed?
3.Is there a new api ?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-specs/+/891204

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/907489

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/907915

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/891204
Committed: https://opendev.org/openstack/neutron-specs/commit/6e5c76fb1b5f9271d6fc9fae32691dbc2973fb4e
Submitter: "Zuul (22348)"
Branch: master

commit 6e5c76fb1b5f9271d6fc9fae32691dbc2973fb4e
Author: Roberto Bartzen Acosta <email address hidden>
Date: Fri Aug 11 11:11:11 2023 -0300

    Add spec for coexistence support for OVN externally managed resources RFE

    Related-bug: #2027742
    Change-Id: I4915d482358f01e81e4b48986f18f733b9d93642

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/909808

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/909808
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.