[RFE] Add support for external vxlan encapsulation to neutron router

[Steve Ruan]

Problem Description

In my deployment, neutron router will be connected to its upstream router with vxlan tunnel.
But when setting neutron router gw as vxlan network, it can't work. Because there is no way to configure the vxlan peer address or vxlan multicast group.
This configure will be also useful for other scenarios like use vxlan to connect 2 openstack directly.
To achieve this, we propose to add a flood list to external vxlan network.

Proposed Change

1. CLI
After change the CLI will looks like:
(neutron) net-create -h
--provider:underlay_flooding_list ip_address Remotes IP addresses of flood list tunnel for VXLAN networks. You can repeat this option.

2. When to create/remove tunnel
When router-gateway-set is called, it will create a qg- interface.
We need set "external_network_bridge" to empty, then qg- will attached to br-int. Why?
Because br-ex did not provide vlan isolation of network.

Then ovs agent will detect the new qg- interface, and check:
1. qg- interface is router gw interface
2. qg- interface' network type is vxlan
3. qg- interface' network have flood list
Then it will create vxlan tunnel, and add flows.
Similar case for remove tunnel.

3. How to create tunnel
The flood list tunnel will be create in br-tun.

Please help review:
https://review.openstack.org/#/q/topic:vxlan_vteps

[Armando Migliaccio (armando-migliaccio)]

If you move this to INPROGRESS, I assume you started to work on this. You're doing it at your own risk, because it's not guaranteed that this will be accepted.

[Kyle Mestery (mestery)]

I think this is something which is useful. Currently, we're assuming external networks are all VLAN networks, which doesn't make sense all the time.

[Mickey Spiegel (emspiege)]

I agree that this is useful. A few comments on the "how" part described above.

The underlay IP multicast group needs to be consistent across all nodes using a particular VNI. Specifying this per router means specifying the same value multiple times, and it also leaves open the possibility of misconfiguration, with inconsistent values on different routers. There are two options that seem cleaner:

1) Add the attribute to network, something like "provider:underlay_multicast_ip". This would allow arbitrary IP multicast values to be specified per VNI.

2) Specify underlay IP multicast groups in the configuration, as is currently done for LinuxBridge:
https://review.openstack.org/#/c/204725/.
Note that with this approach, the provider would have to know the hard coded algorithm to determine which IP multicast address goes with which VNI, so that it could configure upstream physical switches to be consistent with the Neutron routers.
Note also recent discussion on openstack-neutron IRC (12/7/15) about moving this configuration from linuxbridge_agent.ini to ml2_conf.ini (no bug open yet).

It seems like the underlay peer IP addresses that trigger tunnel creation should be a list of external underlay peer IP addresses. For example, if the upstream physical routers use VRRP, there would be two different underlay peer IP addresses.

There are many possibilities for where to specify the underlay peer IP addresses, with different pros and cons: networks, subnets, router external gateway info, or in a configuration file. What you need to end up with at a minimum is a set of VXLAN tunnels to the underlay peer IP addresses where all upstream gateway IPs are located (union over all routers on the node, following the external gateway fixed IPs to subnets to gateway IPs). I guess having more VXLAN tunnels than you need would not be harmful, especially if underlay IP multicast groups are used for broadcasts and unknowns.

[Armando Migliaccio (armando-migliaccio)]

This is an interesting use case.

Today when a router's gateway is set, the uuid of the external network is specified and nothing else. Then, the L3 agent picks that up and does its magic in the _external_gateway_added operation. I believe that supporting this use case should require the most of the user facing changes to be during the creation of the external provider network. The router-gateway-set operation should be oblivious as to whether the external network is a vxlan tunnel or not; therefore to Mickey's point I think option 1 makes the most sense.

That said, today we can already create vxlan provider networks. Adding extra attributes to the object should not be hard. The hard part is in making the L3 agent's gateway business logic handle the vxlan case, and that's where the tricky part is: the logic itself is not very pluggable and some work may be required.

The complexity of this use case makes me believe this needs a spec and full vetting process. Let's bring this up to the drivers meeting to make sure I am not talking nonsense.

[Steve Ruan (ruansx)]

Our team did some investigation. In our change, it's quite simple and will not need to change L3 agent's gateway business logic.

How to do it?
1. Set "external_network_bridge" to empty.
   Router gw port "qg-*" will connect to br-int.
   If not, "qg-*" will connect to external_network_bridge(for example: br-ex), it will be complex.
   For example, 2 "qg-*" in 2 different vxlan networks need vlan isolation, but br-ex did not have yet.

2. L2 agent will detect the qg-* port, it will found it's gw port and gw network is vxlan network.
   It will create tunnel in br-tun, and install the flow which can reuse current APIs.

We almost finish the test on ovs agent, if necessary, we can share the code and write a function spec.

[Kevin Benton (kevinbenton)]

Yes, you shouldnt be deploying the l3 agent with the external_network_bridge option set to br-ex because they is deprecated. Leave it set to an empty string and then the external gateway port is handled like any other l2 port. With that in mind the only required changes should be in the L2 data models to record these vxlan targets and in the l2 agents to setup tunnels to them.

[Armando Migliaccio (armando-migliaccio)]

Status update:

http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-02-11-22.00.log.html#l-193

[Armando Migliaccio (armando-migliaccio)]

Please file a spec to outline your proposal. More details in the logs.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/285986

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/293815

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/293819

[OpenStack Infra (hudson-openstack)]

Change abandoned by Na Zhu (<email address hidden>) on branch: master
Review: https://review.openstack.org/293819
Reason: will submit a new one

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/299191

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/299480

[OpenStack Infra (hudson-openstack)]

Change abandoned by Na Zhu (<email address hidden>) on branch: master
Review: https://review.openstack.org/293815
Reason: will submit a new one soon.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/299492

[OpenStack Infra (hudson-openstack)]

Change abandoned by Na Zhu (<email address hidden>) on branch: master
Review: https://review.openstack.org/285986
Reason: will submit a new one soon.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Na Zhu (<email address hidden>) on branch: master
Review: https://review.openstack.org/299191
Reason: Will submit it in another patch set #299480

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/315896

[OpenStack Infra (hudson-openstack)]

Change abandoned by Na Zhu (<email address hidden>) on branch: master
Review: https://review.openstack.org/299480
Reason: re-submit https://review.openstack.org/315896

[OpenStack Infra (hudson-openstack)]

Change abandoned by Na Zhu (<email address hidden>) on branch: master
Review: https://review.openstack.org/299492
Reason: re-submit https://review.openstack.org/315896

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/316596

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/317309

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/315896
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/316596
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/317309
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Armando Migliaccio (armando-migliaccio)]

It looks like we lost steam here.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/282180
Reason: Comments on bug report seems to hint this effort has been abandoned. Please resume, if you intend to proceed.

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]