[RFE] Create host-routes for routed networks (segments)

Bug #1766380 reported by Harald Jensås on 2018-04-24
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Wishlist
Harald Jensås

Bug Description

When using routed networks[1] on an instance connected to multiple networks the traffic from a segment_a to segment_b within a L3 network might be routed via a different network if the default router/gateway is not on the interface connecting to the routed network.

It would be good to (at-least have an option to) automatically configure host_routes on the subnets in a routed L3 network. In such a way that traffic with a destination on a different segment within the same L3 network is routed via the instance interface connecting to the same L3 network.

Example:
 instance_a:
   - port_a: some_net, segmentX, some_net_subnet <-- default gateway
   - port_b: net1, segmentA, subnetA

 instance_b:
   - port_a: other_net, segmentY, other_net_subnet <-- default gateway
   - port_b: net1, segmentB, subnetB

Unless a host-route is in place, traffic from instance_a to instance_b will use some/other-net, not net1 which both is connected to.

This RFE is to have the host_routes property on the subnets withing net1 populated, so that clients are aware of neighbour L3 networks.

An example configuration:
-------------------------

First Create some_net and other_net:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ openstack network create \
  --provider-network-type flat \
  --provider-physical-network some_phys_net some_net
$ openstack network create \
  --provider-network-type flat \
  --provider-physical-network other_phys_net other_net
$ SOME_NET_SEGMENT_ID=$(openstack network segment list \
                          --network some_net -f value -c ID)
$ OTHER_NET_SEGMENT_ID=$(openstack network segment list \
                           --network other_net -f value -c ID)
$ openstack subnet create \
  --network-segment $SOME_NET_SEGMENT_ID \
  --network some_net \
  --ip-version 4 \
  --subnet-range 192.168.20.0/24 \
  --dhcp \
  --gateway 192.168.20.1 \
  some_net_subnet
$ openstack subnet create \
  --network-segment $OTHER_NET_SEGMENT_ID \
  --network other_net \
  --ip-version 4 \
  --subnet-range 192.168.30.0/24 \
  --dhcp \
  --gateway 192.168.30.1 \
  other_net_subnet

Create net1, with two segment (segmentA and segmentB):
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ openstack network create \
  --provider-network-type flat \
  --provider-physical-network net1_segmentA \
  net1
$ NET1_SEGMENTA_ID=$(openstack network segment list \
                       --network net1 -f value -c ID)
$ openstack network segment set --name net1_segmentA $NET1_SEGMENTA_ID
$ openstack network segment create \
  --network-type flat \
  --physical-network net1_segmentB \
  --network net1 \
  net1_segmentB

Create two subnets associated with different segments on net1:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
$ openstack subnet create \
  --network-segment net1_segmentA \
  --network net1 \
  --ip-version 4 \
  --subnet-range 192.168.100.0/25 \
  --dhcp \
  subnetA
$ openstack subnet create \
  --network-segment net1_segmentB \
  --network net1 \
  --ip-version 4 \
  --subnet-range 192.168.100.128/25 \
  --dhcp \
  subnetB

<pseudo-command> $ openstack server create \
                     --network some_network \
                     --netowork net1 \
                     instance_a

<pseudo-command> $ openstack server create \
                     --network other_network \
                     --netowork net1 \
                     instance_b

NOTE: Assume instance_a is scheduled on a compute with bridge mapping to physical network net1_segmentA, and instance_b is scheduled to a compute with bridge mapping to physical network net1_segmentB.

What we end up with is traffic within net1 (the routed network) is routed via the default gateway on some_net/other_net.

The cloud user most likely does not know that net1 is a routed network, and expect traffic within net1 _not to be_ routed via some/other-net.

To solve this problem neutron should do the equivalent of:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

$ openstack subnet set \
  --host-route destination=192.168.100.128/25,gateway=192.168.100.1 \
  subnetA
$ openstack subnet set \
  --host-route destination=192.168.100.0/25,gateway=192.168.100.129 \
  subnetB

^^ Would make DHCP agents advertise the routes within net1 to clients, making traffic destined for other segments on net1 take a path via a route within net1 (within the routed network).

I believe it would make sense to automate this, so that when additional subnets on additional segments are added the new destination is appended to the host routes.

[1] https://docs.openstack.org/neutron/latest/admin/config-routed-networks.html

description: updated
description: updated
description: updated
description: updated
description: updated
Pawel Suder (pasuder) wrote :

I am wondering if it is intended to have net1 in segmentA on instance_a and net1 in segmentB on instances_b.

I would like to ask few questions:

- is subnet0 the same for all some_net, other_net and net1 nets?
- is net1/subnet0 a connected route for instances?
- is it correct that traffic between instances will be exchanged over segmentA in prior to net1? does it it mean that segment takes priority over network?
- does it mean that instances should be aware about routing things? i.e via DHCP agent
- how to configure mentioned example? would you like to provide example set of command, please?

description: updated
description: updated
Harald Jensås (harald-jensas) wrote :

@Pawel, I have updated the Bug Description. I hope it can clear up you questions.

description: updated
description: updated
description: updated
description: updated
Pawel Suder (pasuder) wrote :

Harald, thank you for your update.

Issue is that two instances are having IPs assigned from two different subnets and are not aware about network where those subnets are included. As you wrote, if default GW is configured and instance does not have explicit configured static route OR connected route, traffic to different subnet (even in the same network, as per Neutron nomenclature) will go through default GW. That's normal network behavior.

I went through it and I did a small research: instead of using default gateway, it might be possible to use static route option - https://tools.ietf.org/html/rfc3442

As I know udhcpd is as an DHCP agent and it seems to support it: https://git.busybox.net/busybox/commit/?id=ee0f444f11504e47fc0f3ab6bc2bf95defb6e6ae and https://git.busybox.net/busybox/commit/?id=7b57ff4436f3e672076e6f5440954a958ba04ab3 but I did not find information about configuration option for that.

With other DHCP server it should be easier I guess. What is more DHCP client needs to handle this DHCP option static route, to correctly configure it.

IMO from Neutron site, it should be possible to somehow configure DHCP server to announce information about static route, but it would be tricky to have it done auto-magically. Questions which I have:

- which static routes should be announced within one Neutron network over all subnets?
- what with segments? static routes should be aware about those correlations?

Harald Jensås (harald-jensas) wrote :

We already support setting static routes via host_routes on a neutron subnet.
The DHCP implementations sets up the options:
https://github.com/openstack/neutron/blob/master/neutron/agent/linux/dhcp.py#L918-L970

For neutron networks, prior to routed provider networks (segments), we even set link-local host routes so that clients do not communicate via the gateway if the two subnets are on the same neutron network.
See: https://github.com/openstack/neutron/blob/master/neutron/agent/linux/dhcp.py#L942-L948

The whole idea of a routed provider network in neutron is to define a L3 network (L3 domain) that to the users’ point of view it looks like booting to a single shared network. Ref Routed networks spec: https://specs.openstack.org/openstack/neutron-specs/specs/newton/routed-networks.html

- which static routes should be announced within one Neutron network over all subnets?
I'll try with another example:

my_routed_provider_network:
  my_segment_a:
    my_subnet_a: 192.168.20.0/26
  my_segment_b:
    my_subnet_b: 192.168.20.64/26
  my_segment_c:
    my_subnet_c: 192.168.20.128/26
  my_segment_d:
    my_subnet_d: 192.168.20.192/26

That is effectively 192.168.20.0/24 split into 4 smaller ipv4 subnets.
In the physical topology:
   rack1: my_subnet_a
   rack2: my_subnet_b
   rack3: my_subnet_c
   rack4: my_subnet_d

As a user I just want to spin up 4 instances on network: my_routed_provider_network. I have no clue that my two instances might end up being scheduled onto compute nodes in different racks, and thus be on different L2 networks with a router boundary between them.
As a user I expect traffic within the network: my_routed_provider_network to stay within that network.

The host routes (static routes) that need to be configure for this to happen are:
 my_subnet_a:
   - dest=192.168.20.64/26,via=192.168.20.1
   - dest=192.168.20.128/26,via=192.168.20.1
   - dest=192.168.20.192/26,via=192.168.20.1
 my_subnet_b:
   - dest=192.168.20.0/26,via=192.168.20.65
   - dest=192.168.20.128/26,via=192.168.20.65
   - dest=192.168.20.192/26,via=192.168.20.65
 my_subnet_c:
   - dest=192.168.20.0/26,via=192.168.20.129
   - dest=192.168.20.64/26,via=192.168.20.129
   - dest=192.168.20.192/26,via=192.168.20.129
 my_subnet_d:
   - dest=192.168.20.0/26,via=192.168.20.193
   - dest=192.168.20.64/26,via=192.168.20.193
   - dest=192.168.20.128/26,via=192.168.20.193

NOTE: In this case, we could also use a supernet and summarize to a single route dest=192.168.20.0/24

- what with segments? static routes should be aware about those correlations?
Yes, static routes should only be configured for subnets associated with _other_ segments on the same network.

If you have multiple subnets associated with the same segment on a network they get link-local routes by the dhcp implementation. These link-local routes ensure that traffic stays within the network on L2 (non-routed) neutron networks, despite the two subnets using different subnet ranges.

Pawel Suder (pasuder) wrote :

Thank you for your explanation and providing detailed information about current implementation.

As I understand your RFE is for having updates of host routes for existing pairs subnet/segment, if a new pair of subnet/segment will be created within the same network. Am I right?

Harald Jensås (harald-jensas) wrote :

> As I understand your RFE is for having updates of host routes
> for existing pairs subnet/segment, if > a new pair of
> subnet/segment will be created within the same network. Am I right?

Correct. That would bring parity with what we do with the link-local routes for multiple subnets on a single segment. I.e traffic between end-points in a neutron network should stay inside the network.

Pawel Suder (pasuder) wrote :

Thank you for your update. It seems to be clarified. PTL will have a look into it and validate it.

Miguel Lavalle (minsel) wrote :

This makes sense to me. Let's bring it up to the drivers team

tags: added: rfe-triaged
removed: rfe
Changed in neutron:
status: New → Confirmed
Miguel Lavalle (minsel) on 2018-05-11
Changed in neutron:
importance: Undecided → Wishlist
Miguel Lavalle (minsel) wrote :

@Harald,

We discussed this RFE in the drivers meeting today. There is a consensus that it a very sensible RFE. There was a minor concern though. We want to make sure that the host routes generated are added to the subnets involved at the API level, so the users have visibility of the routes they are getting. Is the API change included in your plan?

Harald Jensås (harald-jensas) wrote :

@Miquel,

What I want to do is calculate the routes and merge them to any user defined host-routes and store them on the subnet. So, yes, I want to do this in such a way that there should be visibility at the API level.

Miguel Lavalle (minsel) wrote :

@Harald,

Thanks for the follow up. This RFE was approved during today's Drivers meeting. Please carry on

tags: added: rfe-approved
removed: rfe-triaged

Fix proposed to branch: master
Review: https://review.openstack.org/570405

Changed in neutron:
assignee: nobody → Harald Jensås (harald-jensas)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/573896
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=19dccbf07e5234734dee7317bbb681413a1590cd
Submitter: Zuul
Branch: master

commit 19dccbf07e5234734dee7317bbb681413a1590cd
Author: Harald Jensås <email address hidden>
Date: Fri Jun 8 21:10:32 2018 +0200

    Add BEFORE notification event for subnet in ml2 plugin

    Implements the BEFORE_CREATE notification in the
    _before_create_subnet method.

    Related-Bug: #1766380
    Change-Id: I2ad73054b3b870314e8047e9a84ca60285c852e1

Reviewed: https://review.openstack.org/570405
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8d580dc0379842a6a83e922d71f6bc1ffbc85d23
Submitter: Zuul
Branch: master

commit 8d580dc0379842a6a83e922d71f6bc1ffbc85d23
Author: Harald Jensås <email address hidden>
Date: Tue May 22 01:47:36 2018 +0200

    Routed Networks - peer-subnet/segment host-routes (1/2)

    Ensure that host routes are maintained for each subnet within
    a network. Subnets associated with different segments on the
    same network get's host_routes enties added/removed as
    subnets are created, deleted or updated.

    This change handle the host_routes for the subnet that is
    created or updated.

    Partial-Bug: #1766380
    Change-Id: If6792d121e7b8e1ab4c7a548982a42e69023da2b

Reviewed: https://review.openstack.org/573897
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8361b8b5aebad4df3c1012952d9a87b936fef326
Submitter: Zuul
Branch: master

commit 8361b8b5aebad4df3c1012952d9a87b936fef326
Author: Harald Jensås <email address hidden>
Date: Sat Jun 9 02:46:56 2018 +0200

    Routed Networks - peer-subnet/segment host-routes (2/2)

    Ensure that host routes are maintained for each subnet within
    a network. Subnets associated with different segments on the
    same network get host_routes entries added/removed as subnets
    are created, deleted or updated.

    This change handle the host_routes for the peer subnets on the
    same network when a subnet is created or deleted.

    Also adds a shim api extension.

    APIImpact: Host routes are now calculated for routed networks.
    Closes-Bug: #1766380
    Change-Id: Iafbabe6352283e7f1a535a7b147bd81fb32f0ed1

Changed in neutron:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/574706
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=51bb43045ad6866f0078c607c7311db81cd3bab5
Submitter: Zuul
Branch: master

commit 51bb43045ad6866f0078c607c7311db81cd3bab5
Author: Harald Jensås <email address hidden>
Date: Tue Jun 12 13:49:11 2018 +0200

    Shim extension - segments peer subnet host routes, and api-ref

    Adds shim extension for segment peer-subnet host routes.
    Adds api-ref documentation of the new extension.

    Related-Bug: 1766380
    Depends-On: Iafbabe6352283e7f1a535a7b147bd81fb32f0ed1
    Change-Id: Ibd1b565a04a6d979b6e56ca5469af644894d6b4c

This issue was fixed in the openstack/neutron 13.0.0.0b3 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers