[RFE] Add support for Node-Local virtual IP

[Ilya Chukhnakov]

This RFE is to add a new type of IP resource to OpenStack Neutron - a virtual IP that can be shared across multiple ports/VMs (similar to anycast IP) and is guaranteed to only be reachable within the same physical server/node boundaries.

E.g. assuming the cloud has the following resources:
- node "Node-A" with 2 virtual machines with corresponding ports "Port-A-Src" and "Port-A-Dest"
- node "Node-B" with 2 virtual machines with corresponding ports "Port-B-Src" and "Port-B-Dest"
- node "Node-C" with 1 virtual machine with port "Port-C-Src"
We would like to be able to create an IP address resource "NodeLocal-IP" and bind it to "Port-A-Dest" and "Port-B-Dest" such that:
- packets originating from "Port-A-Src" will be guaranteed to be only sent to "Port-A-Dest"
- packets originating from "Port-B-Src" will be guaranteed to be only sent to "Port-B-Dest"
- packets originating from "Port-C-Src" will not reach any destination service

Some examples/use-cases for the feature:
- (generic) Side-car proxies, load-balancers or service mesh platforms. For example K8s Istio project runs a side-car Envoy proxies on each node and all packets from client containers are transparently re-routed through those local proxies. If the proposed feature is added to OpenStack Neutron, that could enable and/or simplify the integration and development of such services for OpenStack.
- (large-scale/high-performance) Distributed caches and CDNs. If the proposed feature is added, that could enable or greatly simplify the use of distributed caches on OpenStack clouds, for example running a VM with Docker Registry cache (passthrough mode) on each physical node could greatly speedup the download/startup time for containers running inside OpenStack VMs.
- (OpenStack) The feature can potentially benefit the OpenStack Octavia project to improve the scalability for very-large-scale deployments by applying load-balance-at-source pattern.

Co-Authored-By: Oleg Bondarev

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Is this something like an allowed address pair but only accessible from the node?

[Ilya Chukhnakov (ichukhnakov)]

From the API point of view we are thinking about something similar to floating IP and trunk port API - meaning that ideally Node-Local IP would be represented as a resource that could be attached/detached to multiple ports. We deliberately left it out of RFE to properly discuss it as part of spec review on gerrit.

From the low-level implementation perspective it might be similar to allowed address pairs. And we actually use allowed address pairs to implement our initial prototype. We believe that such solution could be useful and is easy to implement and it might become a first step or an option for this feature. However in the long run we would like to support transparent address translation/redirection similar to how floating IP works from the guest VM perspective - such that this virtual IP would be purely "virtual" without the need to configure that address on the guest VM OS/interfaces.

[Slawek Kaplonski (slaweq)]

We do have "local" network type in Neutron. Can Your use case be maybe solved by plugging such instance to additional "local" network so it can only communicate with nodes on the same compute node?

[Oleg Bondarev (obondarev)]

Hi @slaweq. We don't think local networks are suitable for the use cases this feature is targeting.
Also we'd like it to be transparent to users/workloads and work on common types of networks (vlan, tunnel): for example VMs/containers from different physical nodes should still be in a single L2 segment and be able to communicate, but only go to local resource for certain IPs.

[Slawek Kaplonski (slaweq)]

Ok, I understand now. Let's discuss it on the tomorrows drivers meeting :)

[Akihiro Motoki (amotoki)]

Looking at usecased mentioned in the RFE description, it sounds useful to me.
It looks similar to anycast in the L3 routing layer, but it is limited to local nodes.
One point I am not sure is whether we need some kind of "default" route for a "NodeLocal-IP". In the proposal, access to "NodeLocal-IP" is unreachable (as the "NodeLocal-IP" is not associated in Node-C. I see two cases: "NodeLocal-IP" must be associated in all relevant nodes or we allow some kind of "default" destination. I am not sure which is better. If we allow "default" destination, we can ensure the reachability for "IP-X" (which is configured as "NodeLocal-IP").

[Akihiro Motoki (amotoki)]

Another discussion point I see is whether we need this in the neutron layer (rather than mechanism like side-car proxies).

[Slawek Kaplonski (slaweq)]

We discussed that RFE today and we decided to approve it. Please now propose spec with details of the solution which You propose and we will continue discussion in the review of that spec.

[LIU Yulong (dragon889)]

"NodeLocal-IP" is binding to the VM, then I did not see the actions (behaviors) reflected in packets.
So the "NodeLocal-IP" will be a "match rule" or some "action rule" reflected in final flows?
It's better to use 4/5/6-tuple to describ the packet conversion and transmission detail.

The original cases is not very clear:
- packets originating from "Port-A-Src" will be guaranteed to be only sent to "Port-A-Dest"
- packets originating from "Port-B-Src" will be guaranteed to be only sent to "Port-B-Dest"
- packets originating from "Port-C-Src" will not reach any destination service

So your examples here should be:
- packets [src_mac, dest_mac, src_ip, dest_ip, (src_port, dest_port)] originating from "Port-A-Src" will be guaranteed to be only sent to "Port-A-Dest" with [src_mac, dest_mac, src_ip, dest_ip, (src_port, dest_port)]
- packets [src_mac, dest_mac, src_ip, dest_ip, (src_port, dest_port)] originating from "Port-B-Src" will be guaranteed to be only sent to "Port-B-Dest"
- packets [src_mac, dest_mac, src_ip, dest_ip, (src_port, dest_port)] originating from "Port-C-Src" will not reach any destination service

It will be better to read about the conversion and transmission detail.

It will be easy to send packet to the "Dest" with some simple dest MAC change in flows, so another concern about this is how to ensure the reply packet can be send back to the right (original) VM?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-specs/+/797798

[Kevin Li (liwenhai)]

From the use case in the RFE, it works more likely a attached fip which forward original external flows to a local port. And blocks traffic from outside. Use case and expected behavior should be:

Initial state：

NodeA： VM-A1(portA1) VM-A2(portA2)
NodeB： VM-B1(portB1) VM-B2(portB2)
NodeC： VM-C1(portC1)
NodeService: VM-Service(portService(10.10.10.10))

Expected behavior: All VMs A1,A2,B1,B2,C1 connect to portService(10.10.10.10) and fetch data.

After enable NLVIP, it works similar as fip and attach to local service VMs:

NodeA： VM-A1(portA1) VM-A2(portA2+NLVIPA1(10.10.10.10))
NodeB： VM-B1(portB1) VM-B2(portB2+NLVIPB1(10.10.10.10))
NodeC： VM-C1(portC1)
NodeService: VM-Service(portService(10.10.10.10))

Expected behavior:
VM-A1 connect to NLVIPA1 for local service with high performance. VM-A1 to portService(10.10.10.10) is blocked.
VM-B1 connect to NLVIPB1 for local service with high performance. VM-B1 to portService(10.10.10.10) is blocked.
VM-C1 connect to portService(10.10.10.10) as original with low performance.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/803051

[Oleg Bondarev (obondarev)]

@Kevin, yes, this is also a correct (and more common) scenario, thanks!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/804523

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/804613

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/797798
Committed: https://opendev.org/openstack/neutron-specs/commit/31e452c6ea047ff5413371bfaa7f44c0d1cc86bd
Submitter: "Zuul (22348)"
Branch: master

commit 31e452c6ea047ff5413371bfaa7f44c0d1cc86bd
Author: Ilya Chukhnakov <email address hidden>
Date: Thu Jun 24 06:36:48 2021 +0300

    Add Node-Local Virtual IP Spec

    Adding the spec to support Node-Local Virtual IP for the RFE
    https://bugs.launchpad.net/neutron/+bug/1930200

    Partial-Bug: #1930200
    Co-Authored-By: obondarev <email address hidden>
    Change-Id: If9f137a839b37f8262f4842b34401a13967ed43e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/803051
Committed: https://opendev.org/openstack/neutron-lib/commit/07e3de1d4cde89432c3474330eff30b3c4e412ca
Submitter: "Zuul (22348)"
Branch: master

commit 07e3de1d4cde89432c3474330eff30b3c4e412ca
Author: Oleg Bondarev <email address hidden>
Date: Mon Jul 26 18:01:06 2021 +0300

    Add Local IP API def

    Partial-Bug: #1930200
    Related-Change:
    https://review.opendev.org/c/openstack/neutron-specs/+/797798 (spec)

    Change-Id: Ia259cd60f048f6c070a1d3a3dbb54b892d67ec30

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/807116

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/804613
Committed: https://opendev.org/openstack/neutron-lib/commit/5e9ced2793b11f5bf18584fa9d242f9b7cbafb08
Submitter: "Zuul (22348)"
Branch: master

commit 5e9ced2793b11f5bf18584fa9d242f9b7cbafb08
Author: Oleg Bondarev <email address hidden>
Date: Tue Aug 10 11:42:00 2021 +0300

    Add Local IP constants and exceptions

    Partial-Bug: #1930200
    Change-Id: I785c6c3fa600d9dcf6a41a527f07e1e4b1ddda52

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/804523
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/807116
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/815102

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/816327

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/816435

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/816468

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/817399

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/816468
Committed: https://opendev.org/openstack/neutron-lib/commit/1d4da16374ff378543f03d85dbf5d564c8f38391
Submitter: "Zuul (22348)"
Branch: master

commit 1d4da16374ff378543f03d85dbf5d564c8f38391
Author: Oleg Bondarev <email address hidden>
Date: Wed Sep 29 17:22:38 2021 +0800

    Add Local IP constants needed for policy checks

    Additionally adds required small fixes to API def

    Partial-Bug: #1930200
    Change-Id: I1ec3185becf03bdc208b0af8c00d2794113a88d5

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/818228

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804523
Committed: https://opendev.org/openstack/neutron/commit/cd1d96863ec8f428f03cc10741e1d876b830a5df
Submitter: "Zuul (22348)"
Branch: master

commit cd1d96863ec8f428f03cc10741e1d876b830a5df
Author: Oleg Bondarev <email address hidden>
Date: Tue Aug 3 18:05:33 2021 +0300

    Add Local IP Extension and DB

    This adds Local IP API extension, DB and OVO models, DB mixin,
    migration and service plugin.

    Partial-Bug: #1930200
    Change-Id: I0ab7c5e9bc918f7fad282673ac6e32e1b01985c5

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807116
Committed: https://opendev.org/openstack/neutron/commit/1222962767bf2b9a3620c9e4681f155146701765
Submitter: "Zuul (22348)"
Branch: master

commit 1222962767bf2b9a3620c9e4681f155146701765
Author: Oleg Bondarev <email address hidden>
Date: Mon Oct 25 14:52:08 2021 +0300

    Add Local IP L2 extension skeleton

    OVS agent part of Local IP feature was divided into
    2 parts to make it easier for reviewers:

    1. This patch adds agent extension skeleton and sets
    server <-> agent RPC communication mechanism via
    push notifications of LocalIPAssociation objects
    create/delete. It also shows how the extension would
    treat those changes. It may be called extension "frontend".

    2. Agent extension flows patch (next one) - deals with OVS
    flows and can be called extension "backend".

    Partial-Bug: #1930200
    Change-Id: I31cb4062b6a21b71c739ab202c60aa7002e4d36e

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/820031

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/815102
Committed: https://opendev.org/openstack/neutron/commit/b51d6958f3c93e72a720fcef77ff55e0d5d10277
Submitter: "Zuul (22348)"
Branch: master

commit b51d6958f3c93e72a720fcef77ff55e0d5d10277
Author: Oleg Bondarev <email address hidden>
Date: Thu Oct 28 11:43:27 2021 +0300

    Add Local IP L2 extension flows

    - setup local ARP responder
    - setup local ip translation flows (nat via conntrack)
    - setup local gARP blocker to prevent undesired Local IP ARP updates
      from other nodes (including real IP address owner)

    2 new OF tables added for br-int:

    - LOCAL_EGRESS_TABLE - to save local ports vlans
    - LOCAL_IP_TABLE - for local ip handling

    Partial-Bug: #1930200
    Change-Id: I49923958d1d602e3af4e02fadbec1b17798c49c8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/824363

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/817399
Committed: https://opendev.org/openstack/neutron/commit/3e71a451958e15e7593bf5f32603c7c653c58aff
Submitter: "Zuul (22348)"
Branch: master

commit 3e71a451958e15e7593bf5f32603c7c653c58aff
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 9 16:26:01 2021 +0300

    Allow to use static Local IP openflow rules

    This avoids conntrack which is usable in OVS offload
    and DPDK cases.
    This also allows to use Local IP feature together with
    openvswitch firewall.

    Partial-Bug: #1930200
    Change-Id: Ifa689fb964ea7335d0fafcf1462cbf820604c7c2

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816435
Committed: https://opendev.org/openstack/neutron/commit/200b345429b0d91246a9b5671dcc621c1e262c37
Submitter: "Zuul (22348)"
Branch: master

commit 200b345429b0d91246a9b5671dcc621c1e262c37
Author: Oleg Bondarev <email address hidden>
Date: Wed Sep 29 16:57:14 2021 +0800

    Add Local IP policy rules

    Partial-Bug: #1930200
    Change-Id: If2d56eb680da521e79cec176bf5c4e6959f1fcf5

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816327
Committed: https://opendev.org/openstack/neutron/commit/0ce584bd9af19daf96bc92dce231a7fac5085aac
Submitter: "Zuul (22348)"
Branch: master

commit 0ce584bd9af19daf96bc92dce231a7fac5085aac
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 2 14:08:16 2021 +0300

    Add Local IP fullstack test cases

    The test:
    - creates 3 VMs in the same network on 2 hosts
    - creates Local IP and associates first VM's port with it
    - pings first VM from the second one (same host) by
      local IP address and checks pings are ok
    - pings first VM from the third one (diff host) by
      local IP and verifies local IP is not accessible

    Second test does the same but with openvswitch security groups
    enabled on hosts and with static NAT rules for Local IPs.

    Partial-Bug: #1930200
    Change-Id: I27a57170b88eda951ff3ef1d5320cb8c692a074a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818228
Committed: https://opendev.org/openstack/neutron/commit/66d50cdbd744cfacfa90905333e3b7dd9cb1ba90
Submitter: "Zuul (22348)"
Branch: master

commit 66d50cdbd744cfacfa90905333e3b7dd9cb1ba90
Author: Oleg Bondarev <email address hidden>
Date: Wed Nov 17 12:43:09 2021 +0300

    Add devstack plugin support for Local IP

    This adds ability to configure local_ip service plugin
    and local_ip OVS agent extension.
    Needed to enable corresponding API/CLI and scenario tests.

    Related-Bug: #1930200
    Change-Id: Ia9269b7a73aef116cf8c5bc6ca0a1cca9deaa056

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/820031
Committed: https://opendev.org/openstack/neutron/commit/d5b9a04bc28faa7696f120b3165f8087eafe815f
Submitter: "Zuul (22348)"
Branch: master

commit d5b9a04bc28faa7696f120b3165f8087eafe815f
Author: Oleg Bondarev <email address hidden>
Date: Wed Dec 1 16:08:35 2021 +0300

    Local IP internal documentation and release note

    Closes-Bug: #1930200
    Change-Id: I6745afad159270c5ccd9be2e68f96d8dafc1dc04

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/824363
Committed: https://opendev.org/openstack/neutron/commit/2a41b0e152915ceac66e82defb2b6610c925a76f
Submitter: "Zuul (22348)"
Branch: master

commit 2a41b0e152915ceac66e82defb2b6610c925a76f
Author: Oleg Bondarev <email address hidden>
Date: Wed Jan 12 12:31:21 2022 +0300

    Local IP: skip ports with invalid ofport

    Some time ago a patch was merged that let's OVS agent to not
    skip ports with invalid ofport [1].
    Thus for Local IP processing we need to explicitly skip such
    ports.

    [1] https://review.opendev.org/c/openstack/neutron/+/640258

    Partial-Bug: #1930200
    Change-Id: I43ba007b4813c02b1cf712252b0925e649fa5813

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/831783

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/831783
Committed: https://opendev.org/openstack/neutron/commit/c02916de4ef63c338a13b1626bc3917bcfa34bf6
Submitter: "Zuul (22348)"
Branch: master

commit c02916de4ef63c338a13b1626bc3917bcfa34bf6
Author: Oleg Bondarev <email address hidden>
Date: Thu Mar 3 17:11:25 2022 +0300

    Local IP: use LOCAL_IP_TABLE for back flows if no OVS fw

    When using static NAT for Local IP translation
    ACCEPTED_EGRESS_TRAFFIC_NORMAL_TABLE should be used for back
    (learned) flows only in case openvswitch security groups are used.
    Otherwise need to use LOCAL_IP_TABLE.

    Partial-Bug: #1930200
    Change-Id: Ie08a26fab3b23507085e00b02c7e11e037a44f62

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/836031

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/836031
Committed: https://opendev.org/openstack/neutron-lib/commit/652741d97bde15c4034eedb526bcc0944c3582e1
Submitter: "Zuul (22348)"
Branch: master

commit 652741d97bde15c4034eedb526bcc0944c3582e1
Author: elajkat <email address hidden>
Date: Thu Mar 31 16:41:39 2022 +0200

    doc: add local-ip api-ref to index

    Change-Id: I89dab958c82e442398b2f65632a50284b558c52d
    Related-Bug: #1930200