[RFE]Support routing traffic based on subnet

[zhaobo]

Use case
==========
In real data center, it may contains several external gateways to access external network. The VM instances can access all of them in L3 layer. Each gateway may from different network service providers, and different network performance(such as stability, bandwidth, speed). Cloud users may want to make the VMs in the specified subnet of network to routing the traffic which from some specified source IP/CIDR to the site which own the best connection line. But the other VMs'traffic which locate on other subnets in the same network pass through the original gateway as default. So we need to route the specified traffic to other gateway.

Also another use case, if cloud user A launched 2 VMs in different subnets, One VM X is used by the customers, and the other VM Y is used for accessing the private env which deployed by cloud user A and contains some security process(private FW, etc..), so there is a requirement that cloud user A need some traffic from VM X(maybe the traffic from VMX which want to access the internet) to pass through the cloud user A's private env to access the internet firstly. Also, this private env can not expose to the customer of user A, user A dont allow that the route table in VM inject the specified routing rules.
So we need another way to process this traffic, like PBR on router. The traffic from each subnets which attached on this router, may need a individual route table for each subnet now.

requests/propose
===========
For openstack, neutron. we need neutron to provide the ability to control the tenant network traffic based on subnet. That means we need the ability of forward this kind traffic based on source IP/CIDR. It may be called "routing traffic based on subnet". Each subnet may has individual route table, each subnet may have mutil different nexthops for supporting more flexible routing.For a subnet, the different traffic of VMs in this subnet may routing to different nexthop.

[Armando Migliaccio (armando-migliaccio)]

Whenever I see the word policy I fear that the topic can be turned into a huge discussion and we could easily go and attempt to boil the ocean.

To this aim, we should try and shed some clarity on the problem scope:

* What do you mean by "Policy"
* How do you intend to alter existing routing rules
* What the impact is for IPv6, and distributed routing
* Whether you intend to expose this functionality in a plugin agnostic fashion.

At the same time, I wonder if this could potentially be solved with L3 flavors, where each one implements its own specific routing policy and then the tenant connects the subnet of choice to the flavor of choice in order to comply with a specific policy. Does that sound crazy?

[zhaobo (zhaobo6)]

Thanks for concern, Armando. I will show the answer the questions as below.

* What do you mean by "Policy"
That's my mistake that wrote a "Policy" in the RFE title, and make you confused. Sorry about that. All we want to accomplish is to set up routing rules in L3 routers based on the CIDR of the source subnet or a specific source IP address from a subnet with an interface on the router. These rules would have a higher priority than normal rules. That means we want to implement it with PBR(policy-based-routing), so I wrote the wrong RFE name with a "Policy" word.

* How do you intend to alter existing routing rules
The less change is better. We want to preserve the current behavior for the most part. The only difference will be the presence of new rules that will route based on certain source CIDRs/IPs. For source CIDRs/IPs that have not been setup by the user in the new way, the router will behave exactly the same as the current routers.

* What the impact is for IPv6, and distributed routing
Yeah, this may effect both of them, so we want to implement this also for IPv6 and DVR parts.

* Whether you intend to expose this functionality in a plugin agnostic fashion.
Yes, that's right, implement it as the fashion is perfectly possible.

Also you raise a question about the L3 flavors/mutil-l3-backends, yeah, we want to follow the L3 flavors approach to implement.

[Armando Migliaccio (armando-migliaccio)]

Thanks for providing further details. There's still more to understand to better scope the problem. For instance:

* What kinds of rules can the user specify (e.g. source routing, type of traffic)?
* How these rules can be specified via the API?
* Whether or not the 'rule language' is extensible?

I suppose much of this can be presented in the form of a spec.

I'll leave linux implementation details for now, but from an L3-agent based implementation, I could envision a service plugin that exposes an API for defining the policy rules and a router flavor 'PBR-aware' that will allow an L3 agent to enforce the rules being defined for the particular router implementation (i.e. HA and/or DVR), should the subnet/router association match.

[zhaobo (zhaobo6)]

Thanks for quick return. :)
As your suggestion&quesions, I answer them below.
* What kinds of rules can the user specify (e.g. source routing, type of traffic)?
For this point now, we are very interested on source based routing, such as subnet CIDR or specific IP address. Of course we could implement this feature for add other criteria either, such as ip address range, type of traffic, etc.

* How these rules can be specified via the API?
We could have a "language" similar to security groups that contain security group rules. In this case, we could have routing policies that contain routing rules. For the first implementation, the routing rules will be simple for only support source based routing, as indicated in the previous point. When creating a router of flavor "PBR-aware", one of its attributes will be a routing policy containing routing rules. This routing policy will complement the normal performed by current Neutron routers.

* Whether or not the 'rule language' is extensible?
Yeah, like point 2 we described, we could use the approach like that.

For your kind envision, we completely agree with you that the service plugin approach.

[Armando Migliaccio (armando-migliaccio)]

This most likely needs to be elaborated in the form of spec, but let see if amongst the drivers team, there's any comment on the direction of this proposal.

[zhaobo (zhaobo6)]

@Armando,
Thanks for return. I will search the proposal in the driver meeting and post the spec.

[YAMAMOTO Takashi (yamamoto)]

i'm not sure if i understand the use case. does it really require source routing? isn't having multiple routers enough?

will this likely be an extension of extraroute extension?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/523658

[zhaobo (zhaobo6)]

Oh, I'm sorry for refresh the status, I will move back.

[Miguel Lavalle (minsel)]

Spec is here: https://review.openstack.org/#/c/523658

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: master
Review: https://review.opendev.org/523658
Reason: According to what we agreed during Shanghai PTG, I abandon this patch for now due to no activity. If You would be interested in continue work on this, feel free to restore the patch.