[RFE] L3 Router support ndp proxy

[yangjianfeng]

As the ipv6 device more and more popularize, we should make our ipv6 VMs more easily connect to external network,but neutron don't support Floating IP and NAT for ipv6. The bgp-dynamic-routing is a optional way to make the ipv6 VMs accessed by external network. But the bgp configuration is more complex, it depend on the external physical router.

So, I propose a eaiser way to make the ipv6 VMs accessed by external network:
In openstack l3 router we set 'proxy_ndp' [1] kernal paramer as '1', like this: 'sysctl -w net.ipv6.conf.all.proxy_ndp=1', then we can add proxied address to gateway tap device, like this: 'ip -6 neigh add proxy 2001:400:1234:567:ffff::8 dev qg-733bd76b-62'.
In external router we just need to add a static direct route, like this: 'ip route add 2001:400:1234:567:ffff::/80 dev fake-gw-port'.
In this way, the external traffic can accurately forward to proper openstack router and then forward to specify VM.

We can implement a plugin to support some APIs, these APIs should support add a single address proxy entry to router external gateway port, in order to that we can control advertise which address to external network. And the iptables can be used to break the trafffic immediately when user delete a address proxy entry.

To guarantee the address is unique, the address scope should be considered.

[1] https://www.geeklab.info/2013/05/ipv6-neighbour-proxy/

[Brian Haley (brian-haley)]

Have you tried using Prefix Delegation? It's supported and works for the entire prefix.

[Miguel Lavalle (minsel)]

Marking as incomplete while waiting for submitter to provide more info

[yangjianfeng (yangjianfeng)]

Hi Brian
I haven't used Prefix Delegation, But I think the IPv6 Prefix Delegation just used to provide automatic allocation of subnet CIDRs. But I proposal is that how to eaisily and flexiblely advertise a ipv6 VM's route. May be we can set the route of the Prefix Delegation's entire prefix ahead of time in external router, but it's not flexible.

Consider the following two scenarios:
One: User create a l3 router then set external gateway and add a ipv6 subnet to it and create a VM on the subnet. How we advertise the VM's route to external router. The BGP is a optional way, But this way need the extranl router's support. The configuretion about it is more complex.

Two: The scene is the same as One. The difference is that user create multiple VMs, but just want to advertise a part of VMs to external network. How do user do?

So, I submit the REF to resolve these problems. I wish neutron to implements some APIs, like these:
`openstack router add ndp proxy <router_id> --address <ipv6 address>`
The api add a ipv6 address to router's external gateway port, then the gateway port can proxy the address's NDP. So the external router can know how to forward packets whose destination address is this ipv6 address.
`openstack router remove ndp proxy <router_id> --address <ipv6 address>`
Remove a ipv6 address from the router's external gateway port. then the gateway port don't proxy the address's NDP. But the external still have the neighber table which contains this ipv6 address for a while. So neutron can set some iptables rules in router namespace to break the traffic immediately.

By this user can control advertise which addresses to external router.

[Miguel Lavalle (minsel)]

Seems worth exploring and discussing. Marking it as confirmed so the conversation can continue

[Brian Haley (brian-haley)]

So in case #1 (User creates a l3 router then sets external gateway and adds an IPv6 subnet to it and create a VM on the subnet), with prefix delegation, since the upstream router has provisioned the prefix, it sets the next-hop for the entire prefix to the neutron router. So all traffic will flow into the subnet automatically, proxy NDP is not required.

For case #2 (advertise a part of VMs to external network), there is no way to do that currently. Can you provide the use case for this? Since if you're trying to not allow incoming traffic that can be accomplished using security groups.

[yangjianfeng (yangjianfeng)]

Hi Brian:
For case #1, I haven't used Prefix Delegation, I don't konw it's mechanism, I will to study it, thanks for your information. But, I still have some questions: 1. If user have multiple neutron routers and add or delete them frequently, will the Prefix Delegation set the next-hops in upstream router automatically for each neutron router? 2. By your comments, I konw the Prefix Delegation need the upstrem router'support, so my question is that whether it's configuretion is complicated to some little companies which haven't senior network engineer?

For case #2, I think the use case is more common. For a common web site, it maybe contains some DB servers and MQ servers, the admin of the web site don't wish they are accessed by external network for security. And for Security Group, I agree that it is a resolve solution optional. But, how about if user disable Security Group for network performance? And some ml2 backends don't support Security Group such as SR-IOV.

[Brian Haley (brian-haley)]

We talked about this in the L3 meeting, just wanted to add some comments.

Regarding Prefix Delegation, I'll just add a quick note on how it works from my memory:

 1) Neutron router requests prefix
 2) Upstream router replies with prefix, adding a route to the /64 it just delegated with the
    next-hop being the Neutron router
 3) Neutron router starts advertising this prefix on the downstream network
 4) When downstream network is removed, the Neutron router releases the prefix

So each IPv6 subnet adds a single route to the upstream router for the entire /64, so
things are happening at L3.

Proxy NDP would work similarly, but at L2:

 1) Upstream router has a prefix, perhaps a /52, on it's downstream interface
 2) Neutron routers configure downstream subnets using prefixes from that pool,
    assumingly from an address pool the admin has configured so there is no overlap
 3) For each instance, the upstream router will do ND on the link, the proxy will
    respond, and the router will add a neighbor entry to it's table

So each IPv6 instance adds a neighbor entry to the upstream router for a /128, so
things are happening at L2 (I know ND is over ICMPv6 so not technically L2). Note
that this can be an issue if the upstream router has a neighbor table limit, since
there is the potential for a lot of proxy entries.

I still think this feature (proxy NDP) could be useful, it might just not be something
an enterprise-class cloud will want to support. It also must not do anything if PD
is enabled, since it could interfere and isn't necessary. So it might be something
the admin has to configure to enable, for example, by making it an extension that must
be added to neutron.conf.

And just to comment on your SR-IOV comment above, I think in that case the instance
will be connected directly to a provider network, so proxy NDP would not apply.

[Brian Haley (brian-haley)]

^^ that wrapped a lot, sorry for that

Just wanted to add an old change related to this, https://review.opendev.org/#/c/143917/

You can probably even run some of those commands by hand to see if they still work.

[yangjianfeng (yangjianfeng)]

Thanks Brian, I entirely agree with your comments #7. The proxy NDP solution just suit some small envtironment, but it is simaple and flexible. If the neutron team think this solution is worthy, I will to submit a spec to describe the solution in detail.

[Slawek Kaplonski (slaweq)]

Reading all Your comments (thx Brian for triaging it), I think that we can discuss that on our next drivers meeting.
IMHO it sounds like it could be something what we can potentially implement as e.g. new service plugin/L3 agent extension which can be enabled on demand by users.

Our next drivers meeting will be on Friday, 15-05-2020: http://eavesdrop.openstack.org/#Neutron_drivers_Meeting - You're welcome to join it yangjianfeng do discuss that proposal and Your use case with us.

[yangjianfeng (yangjianfeng)]

Thanks for slaweq's invitation. I am not sure if I have free time to attend this meeting. And I agree with you that we can potentially implement as e.g. new service plugin/L3 agent extension which can be enabled on demand by users.

So, I write my demand and use case here firstly:
I am working in Troila, we are a private cloud provider. Our majority of customers is some little companies. For them the IPv6 is a fresh thing. So I wish to provide a simple method to them in order to they can rapidly publish IPv6 service.

[YAMAMOTO Takashi (yamamoto)]

is this something like rfc4389?

[Slawek Kaplonski (slaweq)]

After our discussion on the drivers meeting: http://eavesdrop.openstack.org/meetings/neutron_drivers/2020/neutron_drivers.2020-05-15-14.00.log.html#l-22 we all agreed to approve this RFE.
However we have few questions/doubts related to the implementation of this:

* will it require some ndp daemon to be run in router's namespace? Or kernel settings are enough to make it working?
* does the router need to send Neighbor Advertisement when it adds the proxy entry?
* how it will work in case of L3HA during e.g. failover?
* In description You wrote "In external router we just need to add a static direct route, like this: 'ip route add 2001:400:1234:567:ffff::/80 dev fake-gw-port'" - does it mean that neutron-l3 agent will have to do something on the upstream routers?

As a next step we would like to see proposed spec which describes in details how this will be implemented to be able to discuss that.

Also, Miguel Lavalle volunteered that he wants to help with PoC and implementation of this in Neutron, so if You need any help, please sync with Miguel about that :)

[yangjianfeng (yangjianfeng)]

Hi slaweq:
About the last question, It's my nits, I just want to express that the upstream router need to set a dirct route before enable the feature. The command looks a little weird, this is result of I have no physical router then I do the POC with the linux namespace as a fake upstream rotuer, the l3-agent needn't to do anything on the upstream router. The other questions, to be speak frankly, I have not considered them before. I will to make some extra exexperiment, then commit a spec for it.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/728628

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/744815

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/747523

[Slawek Kaplonski (slaweq)]

This bug has had a related patch abandoned and has been automatically un-assigned due to inactivity. Please re-assign yourself if you are continuing work or adjust the state as appropriate if it is no longer valid.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/744815
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/728628
Committed: https://opendev.org/openstack/neutron-specs/commit/6e6ad985f1403299eae125338185c725d023f030
Submitter: "Zuul (22348)"
Branch: master

commit 6e6ad985f1403299eae125338185c725d023f030
Author: Yang JianFeng <email address hidden>
Date: Wed Jan 20 08:56:08 2021 +0000

    L3 router support ndp proxy

    Change-Id: I1b57cbd45513295f80df96efca917476c4fa5033
    Related-Bug: #1877301

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/801181

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/747523
Committed: https://opendev.org/openstack/neutron-lib/commit/2b46c1f74cbf076fca1a100a978a3a8a23d0338c
Submitter: "Zuul (22348)"
Branch: master

commit 2b46c1f74cbf076fca1a100a978a3a8a23d0338c
Author: Yang JianFeng <email address hidden>
Date: Sat Aug 22 09:26:12 2020 +0000

    Adds l3-ndp-proxy extension api definition

    Change-Id: I26f7e642214bd09a6d4608c9cfdf8c5ef2537489
    Partial-Bug: #1877301

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/804846

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/804846
Committed: https://opendev.org/openstack/neutron/commit/e4c168b1fcc3bd79e7708c63533e3caf83ba7f1b
Submitter: "Zuul (22348)"
Branch: master

commit e4c168b1fcc3bd79e7708c63533e3caf83ba7f1b
Author: yangjianfeng <email address hidden>
Date: Tue Aug 17 02:41:15 2021 -0400

    Improve Router callback system's publish events

    When I writing 'ndp_proxy' service plugin, I found I couldn't get enough
    informations about router from the callback system (Such as: the origin
    request body of user send). So, for write service plugin that related
    router plugin more concisely I commit this patch.

    This patch proposal two changes about router callback publish events:
    1. Add 'request_body' parameter to some event's payload
    2. add 'BEFORE_UPDATE' event for router gateway

    Related-bug: #1877301
    Change-Id: I5f6a4e6f0b7c5feb794ddb7efbd07d01bad91af8

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/822253

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/823000

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/743142
Committed: https://opendev.org/openstack/neutron/commit/a0a25cb15c4f04baeed7319b7cdaea328d043388
Submitter: "Zuul (22348)"
Branch: master

commit a0a25cb15c4f04baeed7319b7cdaea328d043388
Author: Yang JianFeng <email address hidden>
Date: Sat Nov 14 08:45:19 2020 +0000

    [Server Side] L3 router support ndp proxy

    Change-Id: I9b92702af8a235443a2fa1aea3997f3d40a03fc3
    Partial-Bug: #1877301

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/827791

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/828022

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/801181
Committed: https://opendev.org/openstack/neutron-lib/commit/c7a2f76cdbabc65d1dcebb76822d664f4c4de354
Submitter: "Zuul (22348)"
Branch: master

commit c7a2f76cdbabc65d1dcebb76822d664f4c4de354
Author: Yang JianFeng <email address hidden>
Date: Sat Jul 17 04:09:50 2021 -0400

    api-ref for ndp proxy

    This patch adds the API documentation for the ndp proxy

    Change-Id: I876405a68cd0f1727350e96eca6fb65433d7247f
    Partial-Bug: #1877301

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/828022
Committed: https://opendev.org/openstack/neutron/commit/5fbd4f1a9d5c5780b6c258c20f336229479a043b
Submitter: "Zuul (22348)"
Branch: master

commit 5fbd4f1a9d5c5780b6c258c20f336229479a043b
Author: yangjianfeng <email address hidden>
Date: Sun Feb 6 12:22:20 2022 +0800

    Add ndp proxy policy rules

    Partial-Bug: #1877301
    Change-Id: I9a815858eb22e1f2ab9fd08ce472ec08b51e8eab

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/744815
Committed: https://opendev.org/openstack/neutron/commit/9b27020a65a5e8d1f6c9dccd352ae00a2c40cf41
Submitter: "Zuul (22348)"
Branch: master

commit 9b27020a65a5e8d1f6c9dccd352ae00a2c40cf41
Author: Yang JianFeng <email address hidden>
Date: Wed Aug 5 00:39:33 2020 +0000

    [Agent Side] L3 router support ndp proxy

    The agent side codes need consider three scenarios:
    1. Non-dvr router. The all related rules are applied in
       qrouter-namespace
    2. Dvr router with the local agent mode is dvr_no_external.
       The all related rules are applied in snat-namespace.
    3. Dvr router with the local agent mode is dvr. In this scenario,
       The all related rules are applied in fip-namespace.

    Change-Id: Ie8729586d318be4a673858021a0116e09e193522
    Partial-Bug: #1877301

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/823000
Committed: https://opendev.org/openstack/neutron/commit/017a098455ee7b0ad0217e84aa613d396eebd032
Submitter: "Zuul (22348)"
Branch: master

commit 017a098455ee7b0ad0217e84aa613d396eebd032
Author: yangjianfeng <email address hidden>
Date: Mon Dec 27 16:28:53 2021 +0800

    Add ndp_proxy to devstack plugin

    This commit adds possibility to configure L3 ndp_proxy service
    plugin and l3 extension with devstack plugin.

    Change-Id: I41713b1cfbf55b0b0e752904c4aea06fe0a00e10
    Partial-Bug: #1877301

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/836581
Committed: https://opendev.org/openstack/neutron/commit/d73ec5000bcb3a6add158eb50057af0a619c7f7f
Submitter: "Zuul (22348)"
Branch: master

commit d73ec5000bcb3a6add158eb50057af0a619c7f7f
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Mar 21 04:48:11 2022 +0000

    [L3] Fix "NDPProxyAgentExtension.ha_state_change" call

    The parameter "data" passed to the method "ha_state_change" is not
    a router but a dictionary with "router_id" info.

    The method "NDPProxyAgentExtension._process_router" requires the
    router ID and the "enable_ndp_proxy" value, stored in the agent
    router cache.

    Closes-Bug: #1967839
    Related-Bug: #1877301
    Change-Id: Iab163e69f7e3641e2e1a451374231b6ccfa74c3e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/822253
Committed: https://opendev.org/openstack/neutron/commit/ad3171c5388567b39112cb439996110d67117f81
Submitter: "Zuul (22348)"
Branch: master

commit ad3171c5388567b39112cb439996110d67117f81
Author: yangjianfeng <email address hidden>
Date: Sun Dec 19 05:55:35 2021 +0000

    [docs] L3 router support ndp proxy

    Change-Id: I2b8642b6830d3e1e1ef86c779c55e9ac1d0f7568
    Partial-Bug: #1877301

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/837552

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/837552
Committed: https://opendev.org/openstack/neutron/commit/79de6969fa4734385552928cdd6676d7f6fb5473
Submitter: "Zuul (22348)"
Branch: master

commit 79de6969fa4734385552928cdd6676d7f6fb5473
Author: yatinkarel <email address hidden>
Date: Tue Apr 12 16:38:23 2022 +0530

    Drop vzctl dep from bindep package list

    This was added with [1] and broked arm64 unit test
    jobs as vzctl is only available from amd64[2].

    [1] https://review.opendev.org/c/openstack/neutron/+/822253
    [2] https://packages.ubuntu.com/focal/vzctl

    Related-Bug: #1877301
    Change-Id: Ie2f8ef9645c5c5331f5ff0da468a9eea508fd246

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/841810

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/845662

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/845662
Committed: https://opendev.org/openstack/neutron/commit/81a330113b5ae3d9b163eb2a92bfa0a628a31dd1
Submitter: "Zuul (22348)"
Branch: master

commit 81a330113b5ae3d9b163eb2a92bfa0a628a31dd1
Author: yangjianfeng <email address hidden>
Date: Tue Jun 14 11:15:02 2022 +0800

    Return default value when get router's enable_ndp_proxy from `request_body`

    In some cases, the creation of router was called by `l3 plugin`
    directly, didn't call API. In this way, the router's request_body
    of the `ndp_proxy` service plugin received has no `enable_ndp_proxy`,
    this will result in `ndp_proxy` service plugin raise error.

    Closes-Bug: #1978519
    Related-Bug: #1877301
    Change-Id: I413c33970002532d11de5bc2afb145704a8488a9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/827791
Committed: https://opendev.org/openstack/neutron-tempest-plugin/commit/2936a2952c9ea7ce3ee6f50c08e64abd2efed3e0
Submitter: "Zuul (22348)"
Branch: master

commit 2936a2952c9ea7ce3ee6f50c08e64abd2efed3e0
Author: yangjianfeng <email address hidden>
Date: Fri Feb 4 11:22:11 2022 +0800

    Add ndp proxy API tests

    Depends-on: https://review.opendev.org/845662
    Change-Id: I7393b74b8a715de92b5aebccf11b30971ea7747a
    Partial-Bug: #1877301

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/841810
Committed: https://opendev.org/openstack/neutron/commit/9dd06e9c4424c7201cb65420f73309c545349055
Submitter: "Zuul (22348)"
Branch: master

commit 9dd06e9c4424c7201cb65420f73309c545349055
Author: yangjianfeng <email address hidden>
Date: Sat May 14 17:06:21 2022 +0800

    Forbid create ndp proxy on same router with same ip address

    Create multiple ndp proxies with same ip address within one router
    is invalid. The related database constraint was missed in previous
    patchsets. The patch add some codes fixed this error.

    Additionally, Fixed two typo errors.

    Related-Bug: #1877301
    Change-Id: Iab24ad78a3d4d9b0ee584cf0986328c9ae2bd16a

[Dr. Jens Harbott (j-harbott)]

Are there any outstanding tasks left or can this be closed as completed?

[Brian Haley (brian-haley)]

I think it's done, anything else can be treated as a bug.