[RFE] Add TCP/UDP port forwarding extension to L3

[Gal Sagie]

I have searched and found many past efforts to implement port forwarding
in Neutron. I have found two incomplete blueprints [1], [2] and an
abandoned patch [3].

There is even a project in Stackforge [4], [5] that claims to implement
this, but the L3 parts in it seems older then current master.

I have recently came across this requirement for various use cases, one
of them is providing feature compliance with Docker port-mapping feature
(for Kuryr), and saving floating IP's space.

There has been many discussions in the past that require this feature,
so i assume there is a demand to make this formal, just a small examples
[6], [7], [8], [9]

The idea in a nutshell is to support port forwarding (TCP/UDP ports) on
the external router leg from the public network to internal ports, so
user can use one Floating IP (the external gateway router interface IP)
and reach different internal ports depending on the port numbers. This
should happen on the network node (and can also be leveraged for
security reasons).

I think that the POC implementation in the Stackforge project shows that
this needs to be implemented inside the L3 parts of the current
reference implementation, it will be hard to maintain something like
that in an external repository. (I also think that the API/DB
extensions should be close to the current L3 reference implementation)

I would like to renew the efforts on this feature and propose a spec for
this to the next release. And of course if any of the people interested
or any of the people that worked on this before want to join the effort,
you are more then welcome to join and comment.

[1] https://blueprints.launchpad.net/neutron/+spec/router-port-forwarding
[2] https://blueprints.launchpad.net/neutron/+spec/fip-portforwarding
[3] https://review.openstack.org/#/c/60512/
[4] https://github.com/stackforge/networking-portforwarding
[5] https://review.openstack.org/#/q/port+forwarding,n,z
[6] https://ask.openstack.org/en/question/75190/neutron-port-forwarding-qrouter-vms/
[7] http://www.gossamer-threads.com/lists/openstack/dev/34307
[8] http://openstack.10931.n7.nabble.com/Neutron-port-forwarding-for-router-td46639.html
[9] http://openstack.10931.n7.nabble.com/Neutron-port-forwarding-from-gateway-to-internal-hosts-td32410.html

Some more descriptions:
https://review.openstack.org/#/c/224727/2/specs/mitaka/port_forwarding.rst
https://review.openstack.org/#/q/status:abandoned+project:openstack/neutron+branch:master+topic:bp/router-port-forwarding

[Armando Migliaccio (armando-migliaccio)]

https://review.openstack.org/#/c/224727

[Armando Migliaccio (armando-migliaccio)]

I like Port forwarding don't get me wrong, but (as mentioned in the spec), providing this in a DVR context is challenging if not impossible under current circumstances, and therefore this becomes a hurdle for us if we want to provide a consistent user experience. Furthermore, this spec/feature has received very little attention both from reviewers and submitters, reflecting the fact that perhaps there isn't much appetite in solving this right now. Finally, the container use case presented is interesting, but I would imagine that port-forwarding is not exactly top priority (but I may be mistaken).

For these reasons, and until more clarity can be provided on some of the aspects raised in the spec, I'd say this may end up being deferred.

[Wenxin Wang (stieizc-33)]

I've came across an environment with a shortage of ipv4 addresses, where port forwarding with fixed ips would really help. It seems from the spec that port forwarding on fixed ips in DVR is easier to implement than forwarding on floating ips (just like the SNAT part on the network node). Maybe this could be done first. I may be wrong, though.

I had no experience with DVR, but it seems from the Networking Guide[1] that port forwarding on floating ips have to be done in the relevant compute node.

[Wenxin Wang (stieizc-33)]

Sorry for the extra comment, I forgot to give the reference link.

[1] http://docs.openstack.org/networking-guide/scenario_dvr_ovs.html#architecture

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]

[Armando Migliaccio (armando-migliaccio)]

This expired because it lacks volunteers both review and development side, and that may fundamentally stem from the fact that no-one thinks it's a high priority use case.

[Armando Migliaccio (armando-migliaccio)]

We are not processing RFEs at this point during the release cycles

[ilnurgi (ilnurgi87)]

Hello,

I implemented it.

neutron portforwarding-list ...
neutron portforwarding-show ...
neutron portforwarding-create ...
neutron portforwarding-delete ...

I'm ready to contribute this

[Ihar Hrachyshka (ihar-hrachyshka)]

@ilnurgi, you said you 'implemented it'. Do you have patches to refer to? Are you going to revive the spec?

More importantly, who is going to work on the code *and* drive the feature to completion from review side? Is L3 subteam onboard with the idea for Ocata and is willing to trade some other ongoing work to that feature?

[Kevin Benton (kevinbenton)]

@Ihar, perhaps comment 8 *is* the implementation. :)

[Armando Migliaccio (armando-migliaccio)]

in #11, it looks like we just lost our committer. Again, nice feature, not enough traction.

[Reedip (reedip-banerjee-deactivatedaccount)]

If possible, can I take it for Pike ?

[brenda (tian-mingming)]

Hi, Reedip. I am also intersted in this feature. I have posted a message in mail list, but no response. I don't know why.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/470596

[Kevin Benton (kevinbenton)]

The RFE is approved, but the spec needs to clearly define how this will work with DVR and HA routers. We don't want an implementation that only works with certain types of routers.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/470596
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=66eb7e172912acbeb7670d7154a9686bb0726c47
Submitter: Zuul
Branch: master

commit 66eb7e172912acbeb7670d7154a9686bb0726c47
Author: Reedip <email address hidden>
Date: Sat Jun 3 06:07:38 2017 +0000

    Spec for Port Forwarding

    The following spec specifies the Port Forwarding extension
    for Floating IPs. And intro a new sub resource into floatingip for port
    forwarding support.

    Depends-On: If40305044c9dfe0024b64bd3921232bb0a6c9372
    Change-Id: Ib2c47b585538bbc067a488e34fd0fc8097314f98
    Partial-Bug: #1491317

[Szabolcs Gelencsér (h3yduck)]

hey folks,

What is the expected date of releasing this enhancement in neutron? We would really need this, is there anything I can help with?

[Reedip (reedip-banerjee-deactivatedaccount)]

Hi Szabolcs ,
I updated the patches which are resolving this bug.
The list of patches are in https://review.openstack.org/#/q/topic:bp/port-forwarding+(status:open+OR+status:merged)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/535638
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=0de474f396d5bba9aeb37e774f56e30d72334837
Submitter: Zuul
Branch: master

commit 0de474f396d5bba9aeb37e774f56e30d72334837
Author: reedip <email address hidden>
Date: Mon Apr 23 16:39:50 2018 +0800

    Introduce API definition of Floating IP Port Forwarding

    This patch introduces the port forwarding API definition which extends
    the current Floating IP API.

    This patch partially implements the following spec:
    https://specs.openstack.org/openstack/neutron-specs/specs/rocky/port-forwarding.html

    Change-Id: I86adb014c5c9b55b00085849c3ebdd38adfed85b
    Co-Authored-By: zhaobo <email address hidden>
    Partial-Bug: #1491317

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/574673

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/575326

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/578764

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/579910

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/578764
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=e863e8f1cd25a786fe416e94185f250eb1191d2f
Submitter: Zuul
Branch: master

commit e863e8f1cd25a786fe416e94185f250eb1191d2f
Author: ZhaoBo <email address hidden>
Date: Thu Jun 28 18:41:00 2018 +0800

    Extend port_forwardings field in Floatingip response

    This patch introduces a new API extension for exposing
    port_forwardings field in Floatingip response.

    This extension requires the router and port_forwarding service plugins.

    This change implements the effect of port forwarding on
    the existing Floatingip API, as specified in [1]

    [1]
    https://specs.openstack.org/openstack/neutron-specs/rocky/port-forwarding.html

    Change-Id: Ia35b57fb26a0cf277f31816e90d24f91ca1f63d1
    Partial-Bug: #1491317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/535647
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5bd6281f9cf2362cd2964b2e7c2e7cbc479d9461
Submitter: Zuul
Branch: master

commit 5bd6281f9cf2362cd2964b2e7c2e7cbc479d9461
Author: ZhaoBo <email address hidden>
Date: Tue Jul 3 15:45:44 2018 +0800

    [server side] Floating IP port forwarding OVO and db script

    This patch implements the port forwarding OVO and db layer code.
    Such as:
    * Introduces a new OVO named 'PortForwarding'.
    * Introduces a new db model for OVO.
    * A migration db script for port forwarding function.

    Partially-Implements: blueprint port-forwarding
    This patch partially implements the following spec:
    https://specs.openstack.org/openstack/neutron-specs/specs/rocky/port-forwarding.html

    The race issue fix in:
    https://review.openstack.org/#/c/574673/

    Fip extend port forwarding field addition in:
    https://review.openstack.org/#/c/575326/

    Partial-Bug: #1491317
    Change-Id: If24e1b3161e2a86ccc5cc21acf05d0a17f6856e7

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/579910
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=21ae99d5b3e287480a2d9cfcfdc2571671d047c4
Submitter: Zuul
Branch: master

commit 21ae99d5b3e287480a2d9cfcfdc2571671d047c4
Author: ZhaoBo <email address hidden>
Date: Tue Jul 3 17:05:36 2018 +0800

    [server side] Floating IP port forwarding plugin

    This patch implements the plugin.
    This patch introduces an new service plugin for port forwarding resources,
    named 'pf_plugin', and supports create/update/delete port forwarding
    operation towards a free Floating IP.

    This patch including some works below:
    * Introduces portforwarding extension and the base class of plugin
    * Introduces portforwarding plugin, support CRUD port forwarding
    resources
    * Add the policy of portforwarding

    The race issue fix in:
    https://review.openstack.org/#/c/574673/

    Fip extend port forwarding field addition in:
    https://review.openstack.org/#/c/575326/

    Partially-Implements: blueprint port-forwarding
    Change-Id: Ibc446f8234bff80d5b16c988f900d3940245ba89
    Partial-Bug: #1491317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/575326
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d00a1558a5a51ad8d189ad3d72aabae6447a88b5
Submitter: Zuul
Branch: master

commit d00a1558a5a51ad8d189ad3d72aabae6447a88b5
Author: ZhaoBo <email address hidden>
Date: Tue Jul 3 22:47:04 2018 +0800

    [server side] Expose port forwardings in FIP API

    This patch introduces a new API extension named 'extend-fip-port-forwarding'
    for exposing 'port_forwardings' field in floatingip responses.

    Partially-Implements: blueprint port-forwarding
    Change-Id: I9016abb6eb650c86c570a0ee78ee12361f4632e4
    Partial-Bug: #1491317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/533850
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=de9b39ed2c4423cfb1ea8b3b92f2b0b40e9c9d62
Submitter: Zuul
Branch: master

commit de9b39ed2c4423cfb1ea8b3b92f2b0b40e9c9d62
Author: ZhaoBo <email address hidden>
Date: Thu May 31 16:49:18 2018 +0800

    [agent side] L3 agent side Floating IP port forwarding

    This patch contains the l3 agent extension and agent part code.
    This patch introduce a new l3 agent extension named "port_forwarding",
    to process the binding of the port forwarding resources, manage its own
    floatingip configuration on router interface and floatingip status.
    Currrently, we support all Neutron Router reference implementations.

    This extension uses the period router sync task and PortForwarding OVO
    rpc.

    * The main idea about this new extension is using the generic router sync
      rpc to maintain the host port forwarding resources,
    * For a single port forwarding create/update/delete, process it one by one
      in smaller scope for forbidding refresh the iptables with a larger
      scope frequently.

    Partially-Implements: blueprint port-forwarding
    Partial-Bug: #1491317
    Change-Id: Ic56e67d428f6177099c285a9d1bccabc1e710f2b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/574673
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4088461ed6cdbc3b1a6896cb03596dd4c3a64d05
Submitter: Zuul
Branch: master

commit 4088461ed6cdbc3b1a6896cb03596dd4c3a64d05
Author: ZhaoBo <email address hidden>
Date: Tue Jul 3 22:12:47 2018 +0800

    [server side] Fix race issue for port forwarding plugin

    This patch fixes the race condition with update/delete neutron
    serveral resources, such as port forwarding conflict with
    floatingip and port forwarding conflict with port.

    Also this approach need the revision function, so need to fix in port
    forwarding model to aware relationship revision update.

    As the port forwarding resource associated with 2 resources,
    one is floatingip, the other is neutron internal port.
    So floatingip update/delete maybe in a conflict situation with
    port forwarding creation. But for port, we just lack the logic to
    process port forwarding during update port's fixed_ip and delete
    port.

    So the approach here is adding logic to let l3 plugin and port
    forwarding plugin know each other when both sides may process the same
    floatingip resource. Based on the existing revision_number feature,
    if one side fail as db staleError, the api layer will retry the whole
    operation for this resource, so there must be a failure on one side in
    this case. This patch just adds the association logic for l3 plugin and
    port forwarding plugin, also adds a event receiver for port update/delete.

    Then the behavior about the port forwarding associated resources would
    be:
    * For fip resource, I introduce one function in that patch.
      _check_floatingip_request
    So during floatingip update/delete, the function will process
    fip and check by rpc callback from l3_plugin, if port forwarding plugin
    also creates a port forwarding with the same fip at this moment. The
    success side would be the one who update the fip_db first, the other side
    would be failure after db retry.

    * For port resource, during update port fixed_ip or delete port, we will
    delete the associated port forwarding resources for free the
    fip:external_port socket.

    Partially-Implements: blueprint port-forwarding
    Change-Id: I637ebcb33b91d899a077bded5ca10097a830a847
    Partial-Bug: #1491317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585731
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=35d945e92f77af1a1051c13158c67ef3c59e8f6b
Submitter: Zuul
Branch: master

commit 35d945e92f77af1a1051c13158c67ef3c59e8f6b
Author: ZhaoBo <email address hidden>
Date: Wed Jul 25 20:07:18 2018 +0800

    Add ext_parent policy check

    Add common parent owner check for the resources which introduced by
    service plugin.

    Then port forwarding resource will share the same tenant_id with
    floatingip. That means only the fip owner can create/update/get/delete
    the associated port forwarding resource.

    Partially-Implements: blueprint port-forwarding
    Partial-Bug: #1491317
    Change-Id: I450c674e55ca15e1d9a6a6224138f3305427da68

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/588079

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/588996

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/588997

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/589071

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/588996
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=473280422a6213d8bee1607b2491ac0d8dc82b69
Submitter: Zuul
Branch: master

commit 473280422a6213d8bee1607b2491ac0d8dc82b69
Author: Miguel Lavalle <email address hidden>
Date: Sun Aug 5 18:13:04 2018 -0500

    api-ref for floating IPs port forwardings

    This patch adds the API documentation for the floating IPs port
    forwardings

    Change-Id: Ib3d223220785959aebee29e9a001d985cce0920c
    Partially-Implements: blueprint port-forwarding
    Partial-Bug: #1491317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/588079
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=66c8ed9c973a9b6df727517f96df40d2ecf467d6
Submitter: Zuul
Branch: master

commit 66c8ed9c973a9b6df727517f96df40d2ecf467d6
Author: Miguel Lavalle <email address hidden>
Date: Wed Aug 1 16:15:09 2018 -0500

    Add release note for port forwardings.

    This patch adds a release note for the floating IPs port forwarding
    functionality.

    Change-Id: I2e0069148cd2551d4534d8a2fd8799f432ffb5e3
    Partially-Implements: blueprint port-forwarding
    Partial-Bug: #1491317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/588997
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ef8e437312f9e6596d2f88b2599c66b9818046c2
Submitter: Zuul
Branch: master

commit ef8e437312f9e6596d2f88b2599c66b9818046c2
Author: Miguel Lavalle <email address hidden>
Date: Sun Aug 5 19:06:31 2018 -0500

    Add FIP port forwarding to Networking Guide

    This patch adds and introduction and configuration instructions to the
    Networking Guide for floating IPs port forwarding

    Change-Id: I821b242f4ba58d92b8e9491db65232ec0a85f73b
    Partially-Implements: blueprint port-forwarding
    Partial-Bug: #1491317

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/589071
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=6e4abc9422cdb197b003061a0fe4b060b4ee4c33
Submitter: Zuul
Branch: master

commit 6e4abc9422cdb197b003061a0fe4b060b4ee4c33
Author: Slawek Kaplonski <email address hidden>
Date: Mon Aug 6 10:20:00 2018 +0200

    Add 'is_filter' and 'is_sort_key' to fip pf attributes

    This patch adds flags to indicate which of the FIP port forwarding's
    atributes can be used to filtering and as a sort key.

    Change-Id: Ie3b50c7f3e3a82b565e8e16acc6ba3e0ea9dc471
    Partially-Implements: blueprint port-forwarding
    Partial-Bug: #1491317

[LIU Yulong (dragon889)]

Maybe we can mark this as implemented?

[Slawek Kaplonski (slaweq)]

Blueprint is marked as implemented already: https://blueprints.launchpad.net/neutron/+spec/port-forwarding so I think that this can be also marked as so. But let's wait for Miguel's decision as he wa approver of the BP

[Abdallah (paltel)]

hello all,

Im new here and working on train release, can i get this fix patch to do port forwarding on route gateway ip and how can i get it ?

many thanks in advance