New neutron subnet pool support breaks multinode testing.

[Clark Boylan]

The new subnet pool support in devstack breaks multinode testing bceause it results in the route for 10.0.0.0/8 being set to via br-ex when the host has interfaces that are actually on 10 nets and that is where we need the routes to go out. Multinode testing is affected because it uses these 10 net addresses to run the vxlan overlays between hosts.
For many years devstack-gate has set FIXED_RANGE to ensure that the networks devstack uses do not interfere with the underlying test host's networking. However this setting was completely ignored when setting up the subnet pools.

I think the correct way to fix this is to use a much smaller subnet pool range to avoid conflicting with every possible 10.0.0.0/8 network in the wild, possibly by defaulting to the existing FIXED_RANGE information. Using the existing information will help ensure that anyone with networks in 10.0.0.0/8 will continue to work if they have specified a range that doesn't conflict using this variable.

In addition to this we need to clearly document what this new stuff in devstack does and how people can work around it should they conflict with the new defaults we end up choosing.

I have proposed https://review.openstack.org/379543 which mostly works except it fails one tempest test that apparently depends on this new subnet pool stuff. Specifically the V6 range isn't large enough aiui.

[Dr. Jens Harbott (j-harbott)]

It seems that the tests for network auto allocation need at least four subnets within each subnet pool (v4+v6).

The issue with reusing the FIXED_RANGE* settings as subnet pools is that the former were intended to be used as a single network, while the subnet pools will be split up into multiple subnets. So in particular FIXED_RANGE_V6 is defined as a /64, which is the proper size for a single network, but using that for SUBNETPOOL_PREFIX_V6 would need to chop it into pieces smaller than /64, which will not work with SLAAC.

So my proposal has two parts:

1. Update devstack-gate to set the SUBNETPOOL* variables in a way that the behaviour in the gate will be similar to what was happening before. It will not be identical, since we need to split the /20 into four /22s, but that should resolve the current failures. https://review.openstack.org/379521

2. Update devstack to have more sensible defaults for the subnet pools. Using all of 10/8 seems pretty greedy and should be changed. Using the documentation prefix as a real world default value also seems broken, probably setting SUBNETPOOL_PREFIX_V6 to some random prefix from fd00::/8 like we do for FIXED_RANGE_V6 would be more sensible.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/380023

[Dr. Jens Harbott (j-harbott)]

In the long run, any prefix from RFC1918 or other private-use networks is going to clash with something somewhere.

So as a permanent solution, how about having someone (like the OpenStack Foundation) become a LIR, so that it could get a dedicated network assigned that is guaranteed to be not in use anywhere else. I don't know which RIR would be appropriate, but if we go to RIPE (european RIR), one could get an IPv4 /22 and an IPv6 /32. The latter should be large enough for anything, while a /22 may be a bit small, but well, IPv4 is doomed anyway.

An alternative approach might be asking one (or multiple) of the OpenStack supporting companies, whether they might be able to donate a larger prefix. It wouldn't have to be routed or formally assigned, they would just need to guarantee that it will not be in use on anything where a devstack instance might be running.

[Clark Boylan (cboylan)]

A little bit more info. This is breaking general ipv4 connectivity in OSIC for our test nodes because we don't get public IPv4 addresses there so any IPv4 connections to the internet are NATed with the local address living in 10.0.0.0/8. This breaks the route for that then no more IPv4 connectivity.

Also I would very strongly prefer that we address this without making any changes to devstack-gate. If we fix this with changes to devstack-gate then that means that people using this must run devstack-gate rather than just devstack to have a working devstack installation.

[Jeremy Stanley (fungi)]

Ideally, all address blocks used by DevStack (including via plugins) should inherit a common v4 /16 prefix which is set by a _single_ configuration option. Subnets of that /16 ought to then be registered with a data structure early in setup and DevStack should fail out with a comprehensible error message as soon as it spots a conflict/overlap between any of these subnets (bonus points for checking whether they also overlap any preexisting local nets for the system). This would not only make it less likely to cause spontaneous conflicts due to picking excessively-sized address ranges from all over RFC 1918 space, but would also make the network configuration trivially portable (such that we could even signal DevStack on a provider-by-provider basis in our jobs as to a base prefix we've deemed safe in that particular environment).

As for asking an RIR for a (fairly large) offline v4 test allocation on the grounds that avoiding conflicts with RFC 1918 space is "hard," I have serious doubts you'll convince them. I've had a hard enough time over the years justifying _legitimate_ connected uses for new v4 allocations. This is something that can be solved in software, it just needs some thought and planning.

[Goutham Pacha Ravi (gouthamr)]

If CIs don't need to use subnetpools allocation, they should be able to turn it off entirely. I like the fix proposed as part of this bug: https://bugs.launchpad.net/devstack/+bug/1628267

[Sean M. Collins (scollins)]

Clark and I believe this is the correct fix: https://review.openstack.org/#/c/379543/

[Ihar Hrachyshka (ihar-hrachyshka)]

It doesn't seem like a neutron issue. Removed neutron from the list of affected projects.

[Armando Migliaccio (armando-migliaccio)]

Let me provide some context: I think this stems from patches [1,2]. Perhaps we can strike a balance between my overly conservative and Assaf's overly aggressive range. I intentionally chose different defaults from FIXED_RANGE_* because I felt the two were not quite the same as pointed out in comment #1. For this reason, I am slightly reluctant to embrace fix [3], even if it does the trick.

Ultimately these values can be overridden at user's leisure, and we can ultimately choose sensible defaults in devstack-gate if need be. Bear in mind, there is some documentation available in [4].

[1] https://review.openstack.org/#/c/282559
[2] https://review.openstack.org/#/c/283289
[3] https://review.openstack.org/#/c/379543
[4] http://docs.openstack.org/draft/networking-guide/config-subnet-pools.html

[Armando Migliaccio (armando-migliaccio)]

Talked to Clarkb on IRC, perhaps we may need to tweak how routes are installed by Neutron when subnetpools are adopted, but I would not hastily rule out any responsibility from Neutron just yet. A build failure example in [2].

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2016-10-04.log.html#t2016-10-04T00:44:50
[2] http://logs.openstack.org/78/381278/2/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/7f82862/console.html#_2016-10-03_21_14_28_763192

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/381965

[Miguel Angel Ajo (mangelajo)]

The routes we install are just for testing, we seem to be routing 10.0.0.0/8 to the
router GW external leg.

I have sent a patch to propose a reduced address range of 10.0.0.0/16 which still allows
for 256 subnets built out of the pool (opposed to 2048)

[Miguel Angel Ajo (mangelajo)]

https://review.openstack.org/381965

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/381940
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=45f071e36eab4f3d20cacbc9cd610e536e1dd2b9
Submitter: Jenkins
Branch: master

commit 45f071e36eab4f3d20cacbc9cd610e536e1dd2b9
Author: Spyros Trigazis <email address hidden>
Date: Tue Oct 4 17:48:28 2016 +0200

    devstack: Fix neutron configuration to run in OSIC

    We can merge this temporary fix and track the following
    related bugs to revert. For details for the issue, please
    refer to the bug descriptions.

    Related-Bug: #1628267
    Related-Bug: #1629133
    Change-Id: I51feb6ccdc0fab91a591568866e6801f2bbb319b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/381934
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=183d1d5ff162a353a7d90c10c57ddf9ca0359fa1
Submitter: Jenkins
Branch: master

commit 183d1d5ff162a353a7d90c10c57ddf9ca0359fa1
Author: Jim Rollenhagen <email address hidden>
Date: Tue Oct 4 11:39:45 2016 -0400

    Set SUBNETPOOL_PREFIX_V4 to FIXED_RANGE

    The default for SUBNETPOOL_PREFIX_V4 in devstack causes external
    connectivity to be mostly broken in osic-cloud1 nodes. IPA relies on
    external activity to build images, and osic-cloud1 is pretty prevalent
    for running CI jobs, therefore this breaks IPA CI.

    Setting this to the same range as FIXED_RANGE should eliminate this, or
    at least reduce the probability of it happening from 1:(size of /8) to
    1:(size of /24).

    This should be reverted when the actual problem is fixed in devstack.

    Related-Bug: #1629133
    Change-Id: I8f0e5405f3e2d6e35418c73f610ac6b779dd75e5

[Armando Migliaccio (armando-migliaccio)]

I was looking at logs once more and I spotted [1]. Then I guess the culprit is [2]. Subnetpool per se has nothing wrong.

[1] http://logs.openstack.org/78/381278/2/check/gate-tempest-dsvm-neutron-multinode-full-ubuntu-xenial/7f82862/logs/devstacklog.txt.gz#_2016-10-03_21_12_03_768
[2] https://review.openstack.org/#/c/356026/4/lib/neutron_plugins/services/l3

[Armando Migliaccio (armando-migliaccio)]

And line [1] more precisely.

[1] https://review.openstack.org/#/c/356026/4/lib/neutron_plugins/services/l3@345

[Armando Migliaccio (armando-migliaccio)]

After the latest finding, I think it's evident that there's nothing we can do to fix this from the Neutron side.

[John L. Villalovos (happycamp)]

This now has a StackOverflow question :(

http://stackoverflow.com/questions/39826272/devstack-installation-failing-tempest-test-requirements

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/381965
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=ca89d071b3c249fba55a824f7f4fc247b7c22948
Submitter: Jenkins
Branch: master

commit ca89d071b3c249fba55a824f7f4fc247b7c22948
Author: Miguel Angel Ajo <email address hidden>
Date: Tue Oct 4 18:17:44 2016 +0200

    Reduce the scope of the subnet pool prefix in neutron

    Some of the clouds used for CI use the 10.2xx.0.0/16 range
    for VMs, and collide with the wider 10.0.0.0/8.

    This setting allows for creation of 256 subnets out of the pool.

    Change-Id: I48c86f94098f1501f0e7f90a265dda7e81440eb0
    Closes-Bug: 1629133

[Monty Taylor (mordred)]

I have added neutron back to this bug because it is still not fixed.

I'm going to restate the problem:

In the gate, a 'safe' network range is added to the devstack config. This is a range that, in terms of the gate, can be used for things and is known to not conflict with existing network ranges in the various gate clouds.

As part of setting itself up, neutron ignores that setting.

The reason it ignores it seems to be based in a technically valid viewpoint that FIXED_RANGE and SUBNETPOOL_PREFIX_V4 mean different things. That's fine.

What's not fine is for the solution (or lack of one) to break the networking of nodes in the gate. Especially since we just flipped the switch on neutron-by-default.

So unless there is actually a solution to OpenStack developers not being able to run jobs in the gate without broken networking, let's not call this one either fixed or not-relevant. Setting SUBNETPOOL_PREFIX_V4=$FIXED_RANGE seems fine for the gate. If it's not, let's find a solution that is.

[Monty Taylor (mordred)]

I've pushed up this:

https://review.openstack.org/397294

Which reflects my admittedly poor understanding of all of the issues at hand.

To be clear - I'm not trying to suggest changes to defaults - only that things like the gate (specifically inclusive of the actual gate) need to be able to sanely express "this network range is safe for the cloud to use for networking things". Although the gate is an obvious and very present example, one can easily imagine people wanting to tell devstack to use particular ranges when running at home so as to not have devstack trying to step on IPs that run the TV or the fridge or whatnot.

In the case where no such ranges have been communicated, the current behavior seems fine.

If it needs to be the case that there need to be _2_ ranges set and they need to be different and non-overlapping, then let's be explicit about that and we make sure devstack-gate always does that.

[Armando Migliaccio (armando-migliaccio)]

Monty,

It was my understanding that the issue reported here stemmed from the fact that change [1] recklessly replaced the host's gateway without being cognizant of the networking ranges at hand.

Can you please provide evidence to any recent failure you have observed? Is the failure mode the same?

It is my understanding that this is a setup issue; neutron operates based on the details that are being fed to it, and takes over based on the assumption that the deployer has ensured that underlay and overlay have been decoupled. What sort of solution would you expect to see being provided by neutron itself?

[1] https://review.openstack.org/#/c/356026/4/lib/neutron_plugins/services/l3@345

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/398012

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/398012
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=4bfbc291eefd92d8b7885f36275b7ff541e067ab
Submitter: Jenkins
Branch: master

commit 4bfbc291eefd92d8b7885f36275b7ff541e067ab
Author: Kevin Benton <email address hidden>
Date: Tue Nov 15 17:26:05 2016 -0800

    Derive IP ranges from new ADDRS_SAFE_TO_USE vars

    The switch to using subnetpools caused quite a bit of confusion
    because it didn't respect the value of FIXED_RANGE. This caused
    conflicts in the gate with it's default IPv4 value of 10.0.0.0/8.

    This patch does a few things to address the issue:
    * It introduces the IPV4_ADDRS_SAFE_TO_USE and IPV6_ADDRS_SAFE_TO_USE
      values and adjusts all of the FIXED_RANGE and SUBNETPOOL_PREFIX values
      to dervive from them by default.
      * This addresses the concern that was raised about implying that
        SUBNETPOOL_PREFIX and FIXED_RANGE are equivalent when setting
        SUBNETPOOL_PREFIX=FIXED_RANGE by default. Now we have a new value
        for the operator specify a chunk of addresses that are safe to
        use for private networks without implementation implications.
      * Backwards compatibility is maintained by alloing users to override
        override all of these values.
    * The default for IPV4_ADDRS_SAFE_TO_USE uses /22 instead of /24
      * Because we want to be able to use subnetpools for auto allocated
        topologies and we want to be able to have a large chunk of
        instances on each network, we needed a little more breathing room
        in the default v4 network size.
    * SUBNET_POOL_SIZE_V4 default is changed from 24 to 26
      * In conjuction with this change and the one above, the default
        subnetpool will support up to 16 64-address allocations.
      * This should be enough to cover any regular gate scenarios.
      * If someone wants a bigger/smaller subnet, they can ask for that
        in the API request, change this value themselves, or use a different
        network entirely.
    * FIXED_RANGE_V6 defaults to a max prefix of /64 from IPV6_ADDRS_SAFE_TO_USE
      * This avoids the private subnet in the non-subnetpool case from being
        larger than /64 to avoid issues identified in rfc 7421.
      * Users can still explicitly set this value to whatever they want.
        This 'max' behavior is only for the default.
      * This allows IPV6_ADDRS_SAFE_TO_USE to default to a /56, which leaves
        tons of room for v6 subnetpools.

    Closes-Bug: #1629133
    Change-Id: I7b32804d47bec743c0b13e434e6a7958728896ea

[Jeremy Stanley (fungi)]

It appears this is finally fixed by https://review.openstack.org/398012 in devstack (and enabled by https://review.openstack.org/398611 in devstack-gate).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/405528
Committed: https://git.openstack.org/cgit/openstack/networking-bgpvpn/commit/?id=2021bf4571c34581528e97a8aaa6ce9441ec231e
Submitter: Jenkins
Branch: master

commit 2021bf4571c34581528e97a8aaa6ce9441ec231e
Author: Thomas Morin <email address hidden>
Date: Thu Dec 1 17:00:07 2016 +0000

    Revert "Temporary workaround to our gate issues"

    This reverts commit dd4941459cc146a94c53b1ef7d7460932f8ae017.

    The patch for devstack has merged :

    I48c86f94098f1501f0e7f90a265dda7e81440eb0

    Related-Bug: 1629133
    Change-Id: I69907b256af6218c4cfe7aa2b4ba0508991fe4b9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/388602
Committed: https://git.openstack.org/cgit/openstack/magnum/commit/?id=821bacc4a7f0b7a0f99d5b23da23375c3db41f64
Submitter: Jenkins
Branch: master

commit 821bacc4a7f0b7a0f99d5b23da23375c3db41f64
Author: yatin <email address hidden>
Date: Wed Oct 19 10:36:52 2016 +0000

    Revert "devstack: Fix neutron configuration to run in OSIC"

    This reverts commit 45f071e36eab4f3d20cacbc9cd610e536e1dd2b9.

    The Temporary fix can be reverted as devstack has released
    the fix in following patch:-
    https://review.openstack.org/398012

    Change-Id: I837f4925cf4c797bd1b02a7bf244ca5742159971
    Closes-Bug: #1628267
    Closes-Bug: #1629133

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/magnum 4.0.0 release.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Bug closed due to lack of activity, please feel free to reopen if needed.