[RFE] Add support for Node-Local virtual IP

Bug #1930200 reported by Ilya Chukhnakov
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Wishlist
Oleg Bondarev

Bug Description

This RFE is to add a new type of IP resource to OpenStack Neutron - a virtual IP that can be shared across multiple ports/VMs (similar to anycast IP) and is guaranteed to only be reachable within the same physical server/node boundaries.

E.g. assuming the cloud has the following resources:
- node "Node-A" with 2 virtual machines with corresponding ports "Port-A-Src" and "Port-A-Dest"
- node "Node-B" with 2 virtual machines with corresponding ports "Port-B-Src" and "Port-B-Dest"
- node "Node-C" with 1 virtual machine with port "Port-C-Src"
We would like to be able to create an IP address resource "NodeLocal-IP" and bind it to "Port-A-Dest" and "Port-B-Dest" such that:
- packets originating from "Port-A-Src" will be guaranteed to be only sent to "Port-A-Dest"
- packets originating from "Port-B-Src" will be guaranteed to be only sent to "Port-B-Dest"
- packets originating from "Port-C-Src" will not reach any destination service

Some examples/use-cases for the feature:
- (generic) Side-car proxies, load-balancers or service mesh platforms. For example K8s Istio project runs a side-car Envoy proxies on each node and all packets from client containers are transparently re-routed through those local proxies. If the proposed feature is added to OpenStack Neutron, that could enable and/or simplify the integration and development of such services for OpenStack.
- (large-scale/high-performance) Distributed caches and CDNs. If the proposed feature is added, that could enable or greatly simplify the use of distributed caches on OpenStack clouds, for example running a VM with Docker Registry cache (passthrough mode) on each physical node could greatly speedup the download/startup time for containers running inside OpenStack VMs.
- (OpenStack) The feature can potentially benefit the OpenStack Octavia project to improve the scalability for very-large-scale deployments by applying load-balance-at-source pattern.

Co-Authored-By: Oleg Bondarev

Tags: rfe-approved
Revision history for this message
Rodolfo Alonso (rodolfo-alonso-hernandez) wrote :

Is this something like an allowed address pair but only accessible from the node?

Revision history for this message
Ilya Chukhnakov (ichukhnakov) wrote :

From the API point of view we are thinking about something similar to floating IP and trunk port API - meaning that ideally Node-Local IP would be represented as a resource that could be attached/detached to multiple ports. We deliberately left it out of RFE to properly discuss it as part of spec review on gerrit.

From the low-level implementation perspective it might be similar to allowed address pairs. And we actually use allowed address pairs to implement our initial prototype. We believe that such solution could be useful and is easy to implement and it might become a first step or an option for this feature. However in the long run we would like to support transparent address translation/redirection similar to how floating IP works from the guest VM perspective - such that this virtual IP would be purely "virtual" without the need to configure that address on the guest VM OS/interfaces.

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

We do have "local" network type in Neutron. Can Your use case be maybe solved by plugging such instance to additional "local" network so it can only communicate with nodes on the same compute node?

Changed in neutron:
importance: Undecided → Wishlist
Revision history for this message
Oleg Bondarev (obondarev) wrote :

Hi @slaweq. We don't think local networks are suitable for the use cases this feature is targeting.
Also we'd like it to be transparent to users/workloads and work on common types of networks (vlan, tunnel): for example VMs/containers from different physical nodes should still be in a single L2 segment and be able to communicate, but only go to local resource for certain IPs.

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

Ok, I understand now. Let's discuss it on the tomorrows drivers meeting :)

tags: added: rfe-triaged
removed: rfe
Revision history for this message
Akihiro Motoki (amotoki) wrote :

Looking at usecased mentioned in the RFE description, it sounds useful to me.
It looks similar to anycast in the L3 routing layer, but it is limited to local nodes.
One point I am not sure is whether we need some kind of "default" route for a "NodeLocal-IP". In the proposal, access to "NodeLocal-IP" is unreachable (as the "NodeLocal-IP" is not associated in Node-C. I see two cases: "NodeLocal-IP" must be associated in all relevant nodes or we allow some kind of "default" destination. I am not sure which is better. If we allow "default" destination, we can ensure the reachability for "IP-X" (which is configured as "NodeLocal-IP").

Revision history for this message
Akihiro Motoki (amotoki) wrote :

Another discussion point I see is whether we need this in the neutron layer (rather than mechanism like side-car proxies).

Revision history for this message
Slawek Kaplonski (slaweq) wrote :

We discussed that RFE today and we decided to approve it. Please now propose spec with details of the solution which You propose and we will continue discussion in the review of that spec.

tags: added: rfe-approved
removed: rfe-triaged
Revision history for this message
LIU Yulong (dragon889) wrote :

"NodeLocal-IP" is binding to the VM, then I did not see the actions (behaviors) reflected in packets.
So the "NodeLocal-IP" will be a "match rule" or some "action rule" reflected in final flows?
It's better to use 4/5/6-tuple to describ the packet conversion and transmission detail.

The original cases is not very clear:
- packets originating from "Port-A-Src" will be guaranteed to be only sent to "Port-A-Dest"
- packets originating from "Port-B-Src" will be guaranteed to be only sent to "Port-B-Dest"
- packets originating from "Port-C-Src" will not reach any destination service

So your examples here should be:
- packets [src_mac, dest_mac, src_ip, dest_ip, (src_port, dest_port)] originating from "Port-A-Src" will be guaranteed to be only sent to "Port-A-Dest" with [src_mac, dest_mac, src_ip, dest_ip, (src_port, dest_port)]
- packets [src_mac, dest_mac, src_ip, dest_ip, (src_port, dest_port)] originating from "Port-B-Src" will be guaranteed to be only sent to "Port-B-Dest"
- packets [src_mac, dest_mac, src_ip, dest_ip, (src_port, dest_port)] originating from "Port-C-Src" will not reach any destination service

It will be better to read about the conversion and transmission detail.

It will be easy to send packet to the "Dest" with some simple dest MAC change in flows, so another concern about this is how to ensure the reply packet can be send back to the right (original) VM?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-specs (master)
Changed in neutron:
status: New → In Progress
Changed in neutron:
milestone: none → next
Revision history for this message
Kevin Li (liwenhai) wrote :

From the use case in the RFE, it works more likely a attached fip which forward original external flows to a local port. And blocks traffic from outside. Use case and expected behavior should be:

Initial state:

NodeA: VM-A1(portA1) VM-A2(portA2)
NodeB: VM-B1(portB1) VM-B2(portB2)
NodeC: VM-C1(portC1)
NodeService: VM-Service(portService(10.10.10.10))

Expected behavior: All VMs A1,A2,B1,B2,C1 connect to portService(10.10.10.10) and fetch data.

After enable NLVIP, it works similar as fip and attach to local service VMs:

NodeA: VM-A1(portA1) VM-A2(portA2+NLVIPA1(10.10.10.10))
NodeB: VM-B1(portB1) VM-B2(portB2+NLVIPB1(10.10.10.10))
NodeC: VM-C1(portC1)
NodeService: VM-Service(portService(10.10.10.10))

Expected behavior:
VM-A1 connect to NLVIPA1 for local service with high performance. VM-A1 to portService(10.10.10.10) is blocked.
VM-B1 connect to NLVIPB1 for local service with high performance. VM-B1 to portService(10.10.10.10) is blocked.
VM-C1 connect to portService(10.10.10.10) as original with low performance.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-lib (master)
Revision history for this message
Oleg Bondarev (obondarev) wrote :

@Kevin, yes, this is also a correct (and more common) scenario, thanks!

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/804523

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-lib (master)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-specs (master)

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/797798
Committed: https://opendev.org/openstack/neutron-specs/commit/31e452c6ea047ff5413371bfaa7f44c0d1cc86bd
Submitter: "Zuul (22348)"
Branch: master

commit 31e452c6ea047ff5413371bfaa7f44c0d1cc86bd
Author: Ilya Chukhnakov <email address hidden>
Date: Thu Jun 24 06:36:48 2021 +0300

    Add Node-Local Virtual IP Spec

    Adding the spec to support Node-Local Virtual IP for the RFE
    https://bugs.launchpad.net/neutron/+bug/1930200

    Partial-Bug: #1930200
    Co-Authored-By: obondarev <email address hidden>
    Change-Id: If9f137a839b37f8262f4842b34401a13967ed43e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-lib (master)

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/803051
Committed: https://opendev.org/openstack/neutron-lib/commit/07e3de1d4cde89432c3474330eff30b3c4e412ca
Submitter: "Zuul (22348)"
Branch: master

commit 07e3de1d4cde89432c3474330eff30b3c4e412ca
Author: Oleg Bondarev <email address hidden>
Date: Mon Jul 26 18:01:06 2021 +0300

    Add Local IP API def

    Partial-Bug: #1930200
    Related-Change:
    https://review.opendev.org/c/openstack/neutron-specs/+/797798 (spec)

    Change-Id: Ia259cd60f048f6c070a1d3a3dbb54b892d67ec30

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/807116

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-lib (master)

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/804613
Committed: https://opendev.org/openstack/neutron-lib/commit/5e9ced2793b11f5bf18584fa9d242f9b7cbafb08
Submitter: "Zuul (22348)"
Branch: master

commit 5e9ced2793b11f5bf18584fa9d242f9b7cbafb08
Author: Oleg Bondarev <email address hidden>
Date: Tue Aug 10 11:42:00 2021 +0300

    Add Local IP constants and exceptions

    Partial-Bug: #1930200
    Change-Id: I785c6c3fa600d9dcf6a41a527f07e1e4b1ddda52

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/804523
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/807116
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/815102

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/816327

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/816435

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron-lib (master)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/817399

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron-lib (master)

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/816468
Committed: https://opendev.org/openstack/neutron-lib/commit/1d4da16374ff378543f03d85dbf5d564c8f38391
Submitter: "Zuul (22348)"
Branch: master

commit 1d4da16374ff378543f03d85dbf5d564c8f38391
Author: Oleg Bondarev <email address hidden>
Date: Wed Sep 29 17:22:38 2021 +0800

    Add Local IP constants needed for policy checks

    Additionally adds required small fixes to API def

    Partial-Bug: #1930200
    Change-Id: I1ec3185becf03bdc208b0af8c00d2794113a88d5

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/818228

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804523
Committed: https://opendev.org/openstack/neutron/commit/cd1d96863ec8f428f03cc10741e1d876b830a5df
Submitter: "Zuul (22348)"
Branch: master

commit cd1d96863ec8f428f03cc10741e1d876b830a5df
Author: Oleg Bondarev <email address hidden>
Date: Tue Aug 3 18:05:33 2021 +0300

    Add Local IP Extension and DB

    This adds Local IP API extension, DB and OVO models, DB mixin,
    migration and service plugin.

    Partial-Bug: #1930200
    Change-Id: I0ab7c5e9bc918f7fad282673ac6e32e1b01985c5

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/c/openstack/neutron/+/807116
Committed: https://opendev.org/openstack/neutron/commit/1222962767bf2b9a3620c9e4681f155146701765
Submitter: "Zuul (22348)"
Branch: master

commit 1222962767bf2b9a3620c9e4681f155146701765
Author: Oleg Bondarev <email address hidden>
Date: Mon Oct 25 14:52:08 2021 +0300

    Add Local IP L2 extension skeleton

    OVS agent part of Local IP feature was divided into
    2 parts to make it easier for reviewers:

    1. This patch adds agent extension skeleton and sets
    server <-> agent RPC communication mechanism via
    push notifications of LocalIPAssociation objects
    create/delete. It also shows how the extension would
    treat those changes. It may be called extension "frontend".

    2. Agent extension flows patch (next one) - deals with OVS
    flows and can be called extension "backend".

    Partial-Bug: #1930200
    Change-Id: I31cb4062b6a21b71c739ab202c60aa7002e4d36e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/820031

Changed in neutron:
assignee: nobody → Oleg Bondarev (obondarev)
milestone: next → yoga-3
Changed in neutron:
milestone: yoga-3 → yoga-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/815102
Committed: https://opendev.org/openstack/neutron/commit/b51d6958f3c93e72a720fcef77ff55e0d5d10277
Submitter: "Zuul (22348)"
Branch: master

commit b51d6958f3c93e72a720fcef77ff55e0d5d10277
Author: Oleg Bondarev <email address hidden>
Date: Thu Oct 28 11:43:27 2021 +0300

    Add Local IP L2 extension flows

    - setup local ARP responder
    - setup local ip translation flows (nat via conntrack)
    - setup local gARP blocker to prevent undesired Local IP ARP updates
      from other nodes (including real IP address owner)

    2 new OF tables added for br-int:

    - LOCAL_EGRESS_TABLE - to save local ports vlans
    - LOCAL_IP_TABLE - for local ip handling

    Partial-Bug: #1930200
    Change-Id: I49923958d1d602e3af4e02fadbec1b17798c49c8

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/824363

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/817399
Committed: https://opendev.org/openstack/neutron/commit/3e71a451958e15e7593bf5f32603c7c653c58aff
Submitter: "Zuul (22348)"
Branch: master

commit 3e71a451958e15e7593bf5f32603c7c653c58aff
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 9 16:26:01 2021 +0300

    Allow to use static Local IP openflow rules

    This avoids conntrack which is usable in OVS offload
    and DPDK cases.
    This also allows to use Local IP feature together with
    openvswitch firewall.

    Partial-Bug: #1930200
    Change-Id: Ifa689fb964ea7335d0fafcf1462cbf820604c7c2

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816435
Committed: https://opendev.org/openstack/neutron/commit/200b345429b0d91246a9b5671dcc621c1e262c37
Submitter: "Zuul (22348)"
Branch: master

commit 200b345429b0d91246a9b5671dcc621c1e262c37
Author: Oleg Bondarev <email address hidden>
Date: Wed Sep 29 16:57:14 2021 +0800

    Add Local IP policy rules

    Partial-Bug: #1930200
    Change-Id: If2d56eb680da521e79cec176bf5c4e6959f1fcf5

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/c/openstack/neutron/+/816327
Committed: https://opendev.org/openstack/neutron/commit/0ce584bd9af19daf96bc92dce231a7fac5085aac
Submitter: "Zuul (22348)"
Branch: master

commit 0ce584bd9af19daf96bc92dce231a7fac5085aac
Author: Oleg Bondarev <email address hidden>
Date: Tue Nov 2 14:08:16 2021 +0300

    Add Local IP fullstack test cases

    The test:
    - creates 3 VMs in the same network on 2 hosts
    - creates Local IP and associates first VM's port with it
    - pings first VM from the second one (same host) by
      local IP address and checks pings are ok
    - pings first VM from the third one (diff host) by
      local IP and verifies local IP is not accessible

    Second test does the same but with openvswitch security groups
    enabled on hosts and with static NAT rules for Local IPs.

    Partial-Bug: #1930200
    Change-Id: I27a57170b88eda951ff3ef1d5320cb8c692a074a

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/818228
Committed: https://opendev.org/openstack/neutron/commit/66d50cdbd744cfacfa90905333e3b7dd9cb1ba90
Submitter: "Zuul (22348)"
Branch: master

commit 66d50cdbd744cfacfa90905333e3b7dd9cb1ba90
Author: Oleg Bondarev <email address hidden>
Date: Wed Nov 17 12:43:09 2021 +0300

    Add devstack plugin support for Local IP

    This adds ability to configure local_ip service plugin
    and local_ip OVS agent extension.
    Needed to enable corresponding API/CLI and scenario tests.

    Related-Bug: #1930200
    Change-Id: Ia9269b7a73aef116cf8c5bc6ca0a1cca9deaa056

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/820031
Committed: https://opendev.org/openstack/neutron/commit/d5b9a04bc28faa7696f120b3165f8087eafe815f
Submitter: "Zuul (22348)"
Branch: master

commit d5b9a04bc28faa7696f120b3165f8087eafe815f
Author: Oleg Bondarev <email address hidden>
Date: Wed Dec 1 16:08:35 2021 +0300

    Local IP internal documentation and release note

    Closes-Bug: #1930200
    Change-Id: I6745afad159270c5ccd9be2e68f96d8dafc1dc04

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.opendev.org/c/openstack/neutron/+/824363
Committed: https://opendev.org/openstack/neutron/commit/2a41b0e152915ceac66e82defb2b6610c925a76f
Submitter: "Zuul (22348)"
Branch: master

commit 2a41b0e152915ceac66e82defb2b6610c925a76f
Author: Oleg Bondarev <email address hidden>
Date: Wed Jan 12 12:31:21 2022 +0300

    Local IP: skip ports with invalid ofport

    Some time ago a patch was merged that let's OVS agent to not
    skip ports with invalid ofport [1].
    Thus for Local IP processing we need to explicitly skip such
    ports.

    [1] https://review.opendev.org/c/openstack/neutron/+/640258

    Partial-Bug: #1930200
    Change-Id: I43ba007b4813c02b1cf712252b0925e649fa5813

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/831783

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/831783
Committed: https://opendev.org/openstack/neutron/commit/c02916de4ef63c338a13b1626bc3917bcfa34bf6
Submitter: "Zuul (22348)"
Branch: master

commit c02916de4ef63c338a13b1626bc3917bcfa34bf6
Author: Oleg Bondarev <email address hidden>
Date: Thu Mar 3 17:11:25 2022 +0300

    Local IP: use LOCAL_IP_TABLE for back flows if no OVS fw

    When using static NAT for Local IP translation
    ACCEPTED_EGRESS_TRAFFIC_NORMAL_TABLE should be used for back
    (learned) flows only in case openvswitch security groups are used.
    Otherwise need to use LOCAL_IP_TABLE.

    Partial-Bug: #1930200
    Change-Id: Ie08a26fab3b23507085e00b02c7e11e037a44f62

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 20.0.0.0rc1

This issue was fixed in the openstack/neutron 20.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron-lib (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/836031

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron-lib (master)

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/836031
Committed: https://opendev.org/openstack/neutron-lib/commit/652741d97bde15c4034eedb526bcc0944c3582e1
Submitter: "Zuul (22348)"
Branch: master

commit 652741d97bde15c4034eedb526bcc0944c3582e1
Author: elajkat <email address hidden>
Date: Thu Mar 31 16:41:39 2022 +0200

    doc: add local-ip api-ref to index

    Change-Id: I89dab958c82e442398b2f65632a50284b558c52d
    Related-Bug: #1930200

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.