VPNaaS: Allow multiple local subnets for IPSec

[Paul Michali]

Problem: Currently VPN IPSec site-to-site connections can only specify one local subnet. The user must create multiple tunnels, if more than one subnet is desired on the local end.

Solution: Allow the user to specify more than one local subnets for a single IPSec connection.

Advantages: This simplifies the user experience for setting up VPN for multiple subnets. There is the obvious reduction in number of connections and their associated resources needed. The change can also be used for future VPN flavors (e.g. DM VPN).

[Paul Michali (pcm)]

I can work on this, if/when approved by drivers team.

[Armando Migliaccio (armando-migliaccio)]

Guidelines, details on how to write RFE's, and the process for handling features if you have already submitted specs in the past, but are yet to be complete. can be found here:

https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst

For more details, please reach out on #openstack-neutron or openstack-dev ML [neutron].

[venkata anil (anil-venkata)]

Paul Michali,Feel free to assign this bug on your name when u start working on this.(I am also happy to work on this bug :) )

[Paul Michali (pcm)]

Armando, I'm (already) trying to follow the guidelines for RFE with this bug. Am I missing anything? This is not a previously submitted request and there is currently no spec for it (following the steps...create RFE, get approval, see if spec with new format is requested, assign to RFE, and work on it).

Venkata, I plan on working on this, but did not sign up, as I'm trying to follow the new RFE process (see Armando's link above). I removed your assignment.

[Kyle Mestery (mestery)]

Please change the RFE to reflect the verbage from this link [1].

[1] https://github.com/openstack/neutron/blob/master/doc/source/policies/blueprints.rst#rfe-submission-guidelines

[Paul Michali (pcm)]

Description has been updated. Let me know if there is additional information needed.

[Paul Michali (pcm)]

Team consensus was to add validation logic to require:

- all local subnets for a connection are of the same IP version
- local and peer subnets are of the same IP version

Implication is that, for now, dual-stack would need two tunnels to support. We can remove that restriction, once we have tested this scenario, but the thought was to initially add the restriction and keep things simple.

[Kyle Mestery (mestery)]

Seems like a usability enhancement. Moving to triaged and marking a priority. I think we can move ahead with this one.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/191944

[Paul Michali (pcm)]

Implementing endpoint-groups API to support multiple local subnets (and have an API that can be used for other VPN types), and then will use the new APIs to implement the multiple subnets.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/215717

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/216248

[OpenStack Infra (hudson-openstack)]

Change abandoned by Paul Michali (<email address hidden>) on branch: master
Review: https://review.openstack.org/215717
Reason: Will do all changes under one commit - review is 212692

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/216248
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=2f85ddf2e8b2e5d14554f5dc210254fdeb87598e
Submitter: Jenkins
Branch: master

commit 2f85ddf2e8b2e5d14554f5dc210254fdeb87598e
Author: Paul Michali <email address hidden>
Date: Mon Aug 24 13:02:14 2015 +0000

    VPNaaS: Splitting out models from database class

    SInce additional models are going to be added to VPNaaS for the endpoint
    groups and multiple local subnets feature, this is an opportune time to
    split out the models into a new module.

    Change-Id: Ia729bd0c6967fa2b8c698495aa360f340b42d98a
    Related-Bug: 1459423

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/191944
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=c24391cb5ce2f759b8a029044baa1b083bb11c7a
Submitter: Jenkins
Branch: master

commit c24391cb5ce2f759b8a029044baa1b083bb11c7a
Author: Paul Michali <email address hidden>
Date: Mon Jun 15 15:21:12 2015 -0400

    VPNaaS: DevRef for multiple local subnets

    This provides design information to go along with the RFE.
    Selected approach that is the most flexible, permitting
    future reuse.

    Provided details on how to handle backwards compatibility.

    Change-Id: I943ed13d3d43017a86d709eeae33967f25515937
    Partial-Bug: 1459423

[Paul Michali (pcm)]

Endpoint groups (infrastructure for the multiple local subnets) is out for review. Need the code (212692) and CLI client (219455) reviewed and upstreamed.

Coding is proceeding on the multiple local subnet implementation.

[Kyle Mestery (mestery)]

I don't think this is gonna make RC1 at this point.

[Paul Michali (pcm)]

Kyle, should the RFE be broken into two bugs, one for Endpoint Groups and one for Multiple Local Subnets for IPSec? The former is completed, out for review, a foundational piece for the latter, can be used by BGP VPN and other flavors, and can be treated independently.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/212692
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=56b0bca500b83644e54a4493dcee795200fe3aa7
Submitter: Jenkins
Branch: master

commit 56b0bca500b83644e54a4493dcee795200fe3aa7
Author: Paul Michali <email address hidden>
Date: Tue Aug 4 11:21:54 2015 +0000

    VPNaaS: Provide Endpoint groups capability.

    This commit implements the new tables for VPN endpoint-groups, and the
    CRUD REST API. Validation logic has been added for the reference
    implementation. Migration files are created, so that the needed tables
    are created. Unit tests have been added for the API and database.

    The neutron client and Horizon support for this new API will be handled
    separately, as will API test cases for the new API.

    This new API is a prerequisite for providing the multiple local subnet
    feature, which will use this API and make changes to existing APIs. At
    that time, migration and backward compatibility support will be provided,
    so that users can migrate smoothly to the new API (since there is no
    support for micro-versioning).

    APIImpact

    Depends-On: Ia729bd0c6967fa2b8c698495aa360f340b42d98a
    Change-Id: I6e10590a988312eafca076a14be38b19e2d44a87
    Partial-Bug: 1459423

[Paul Michali (pcm)]

Endpoint groups and CLI client are upstreamed. Multiple local subnet code, will be placed for review soon, and followed by CLI client support.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/230164

[Paul Michali (pcm)]

https://review.openstack.org/#/c/223049/5

[Armando Migliaccio (armando-migliaccio)]

Some stuff already merged, so this effectively approved.

[Paul Michali (pcm)]

There are some related follow on commits that will be needed, once this upstreams...

- API test for new endpoint group and multiple local subnet APIs (once API tests available in neutron-vpnaas repo).
- Horizon support for multiple local subnet feature.
- Check when changing subnet in use by endpoint group (https://bugs.launchpad.net/neutron/+bug/1503862)
- Akihiro would like to see endpoint group API placed in an extension shim to make the API discoverable.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/230164
Committed: https://git.openstack.org/cgit/openstack/neutron-vpnaas/commit/?id=7ba17a3155a7ab69032e4be9f17777818aa977e5
Submitter: Jenkins
Branch: master

commit 7ba17a3155a7ab69032e4be9f17777818aa977e5
Author: Paul Michali <email address hidden>
Date: Mon Sep 28 20:00:58 2015 +0000

    VPNaaS: Multiple Local Subnets feature

    Implements support for multiple local subnets for IPSec site to site
    connections, using the new Endpoint Group API.

    The implementation supports backwards compatibility as follows. If
    a VPN service is created with a subnet, then the older API is assumed
    and the user must specify the peer CIDRs for any IPSec connections
    and cannot specify multiple local subnets.

    If a subnet is not provided for the VPN service, then the user must
    use the newer API and provide a local and peer endpoint group IDs for
    the IPSec connection (and cannot specify the peer CIDRs in the IPSec
    connection API).

    Implication here is that the subnet will be an optional argument for
    the VPN service API.

    With this feature, when an endpoint group is deleted, a check is made
    to ensure that there are no IPSec connections using the group.

    Migration will move the subnet from VPN service API to a new endpoint
    group, and specify the group in any connections using the service.
    The peer CIDR(s) will be moved from each connection to an endpoint
    group, with the group ID specified in the connection.

    Note: As part of testing the database methods for this feature, several
    more tests were created to test database access only, instead of doing
    a round trip test. In a separate commit, these tests can be expanded,
    and the existing round trip tests removed.

    Note: Tests for building the dict used in sync requests was enhanced,
    as part of supporting this feature. The previous tests didn't check
    that the peer CIDR information was correct, so were enhanced as well.

    Note: The service driver passes the local CIDR(s) in a new field,
    called local_cidrs. This field is used for the older API, as well,
    passing the subnet's CIDR, and allowing consistent consumption by the
    device driver. The IP version is also passed, rather than obtaining
    it from the subnet info (so both new and old API use the same fields).

    Note: to support rolling upgrades, where an agent may be using the older
    release, after the server has been updated, the subnet CIDR field passed
    from the service driver to device driver, will be populated from the
    first local endpoint from the first connection (there has to be at least
    one connection, when sending data to the agent).

    Note: In the device driver, I noticed that the local CIDR's IP version
    can change the config file output, so I added test cases for IPv6, as
    part of enhancing the tests for multiple local CIDRs.

    Change-Id: I7a011e3170d7db463a6561e550b2ead3e3311125
    Partial-Bug: 1459423

[Paul Michali (pcm)]

Armando, the endpoint groups portion of the feature upstreamed in Liberty. The multiple local subnets part upstreamed early in Mitaka, along with functional tests. The outstanding part is the neutron-client changes to support this, which are out for review and should upstream shortly.

It was an operator desired feature, when asked on the openstack-dev mailing list.

[Paul Michali (pcm)]

Review https://review.openstack.org/#/c/231133/ just upstreamed yesterday, so this should complete this bug.

[Armando Migliaccio (armando-migliaccio)]

Everything targeting this bug has merged:

https://review.openstack.org/#/q/message:%22'1459423'%22,n,z

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/python-neutronclient 4.0.0 release.