Bootstrapping Juju 2.2.x fails on a Openstack cloud with Neutron running L3 Agent HA.

[Tom-Erik Røberg]

Bootstrapping Juju 2.2.x fails on a Openstack cloud with Neutron running in HA. It fails to bootstrap a new version 2.2.2 controller. Upgrading a controller from 2.1.3 to 2.2.2 succeeds, but adding a new model results in an error.

juju bootstrap --metadata-source /home/ubuntu/simplestreams devstack devbackcont1 --no-gui --constraints “mem=4G root-disk=30G” --config network=“ca556f23-4d7a-4439-985c-5b7163526123”
 …
ERROR fetching hosted model spaces: adding subnet “169.254.192.0/18”: subnet “169.254.192.0/18” already exists
ERROR failed to bootstrap model: subprocess encountered error code 1

169.254.192.0/18 is the subnet used for HA communication between Neutron routers.

[John A Meinel (jameinel)]

This sounds like it is less "neutron is HA", and more that there is potentially the same subnet configured in Openstack 2 times. Juju 2.2 started inventorying the provider to find the Subnets that are available, and we record what subnet maps to what provider id. However, we also currently don't allow the same subnet to be registered 2 times.

Can you confirm if you
a) Do have 169.254.192.0/18 associated with more than one Openstack Subnet, and
b) if so, is that intentional and what is the use case for having a repeated range

[Tom-Erik Røberg (tom-erik-roberg)]

Thank you for the quick response!

Yes, we currently have two Openstack subnets associated with 169.254.192.0/18.

This subnet is used for communication between the Neutron HA routers. Neutron
creates one network and a subnet for each tenant/project. This network and
subnet is not associated with a tenant/project, no project_id is recorded in
the database. See attachment with detailed output from openstack.

These subnets and networks was created after we enabled Neutorn HA which we
deploy with neutron-api charm.

https://jujucharms.com/neutron-api/#charm-config-enable-l3ha

I assume that this is correct behavior from Neutron. The HA network is
internal to Neutron and it makes sense that it should not be exposed to the
tenant/project and therefore does not have a project_id.

I have investigated the issue further and it seems that I'm hitting this
problem because my test account has admin access in Openstack, even when
tenant-name is set to the correct tenant/project. I can successfully bootstrap
Juju 2.2.2 with an account with "member" access to the tenant/project.

It seems that there is bug in how Juju filters out subnets by project_id. The
HA networks for Neutron does not have a project_id set. See attached output
from openstack.

[John A Meinel (jameinel)]

Can you also include the content of 'openstack network show' for the various networks as well? I would guess that we ultimately filter by the networks your project is going to be attached to.

[Tom-Erik Røberg (tom-erik-roberg)]

Yes, please see the attached output from openstack network show.

[Sandor Zeestraten (szeestraten)]

As Tom-Erik mentioned, that is the default subnet/CIDR for the L3 HA admin network in OpenStack which will be created everytime you create virtual routers in OpenStack so this is something Juju needs to support.

See the official OpenStack documentation:
* https://docs.openstack.org/mitaka/config-reference/networking/networking_options_reference.html
* https://docs.openstack.org/mitaka/networking-guide/deploy-lb-ha-vrrp.html

[Andrew Woodward (xarses)]

I also ran into problems getting the controller deployed with duplicate subnets, In my case they are not needed and I was able to bypass the problem by removing the duplicates. Reading the above, there are many cases where there may be duplicate subnet cidr.

ERROR fetching hosted model spaces: adding subnet "10.10.0.0/26": subnet "10.10.0.0/26" already exists
2017-09-01 17:52:37 DEBUG cmd supercommand.go:459 error stack:
github.com/juju/juju/state/subnets.go:222: subnet "10.10.0.0/26" already exists
github.com/juju/juju/state/subnets.go:235:
github.com/juju/juju/state/subnets.go:235: adding subnet "10.10.0.0/26"
github.com/juju/juju/state/spacesdiscovery.go:78:
github.com/juju/juju/state/spacesdiscovery.go:51:
github.com/juju/juju/agent/agentbootstrap/bootstrap.go:223: fetching hosted model spaces
github.com/juju/juju/cmd/jujud/agent/agent.go:102:
2017-09-01 17:52:37 DEBUG juju.cmd.jujud main.go:186 jujud complete, code 0, err <nil>
17:52:37 ERROR juju.cmd.juju.commands bootstrap.go:492 failed to bootstrap model: subprocess encountered error code 1
17:52:37 DEBUG juju.cmd.juju.commands bootstrap.go:493 (error details: [{github.com/juju/juju/cmd/juju/commands/bootstrap.go:584: failed to bootstrap model}
{subprocess encountered error code 1}])

[Andrew Woodward (xarses)]

I was thinking, a possible work around would be to bootstrap a controller with OpenStack credentials that can't see the L3 HA or other possible duplicate networks.

[Karl Kloppenborg (karlkloppenborg)]

Hi,

Is there any update on this?
We're also experiencing this issue as our HA L3 Openstack networks require the subnet 169.254.192.0/18.

It doesn't seem far fetching to create a configuration option to override the subnet JuJu uses for communication? Ideally this would be a configurable option. It really depends on how tied into the core orchestration 169.254.192.0/18 really is.

I'm going to take a look and see if it's something I can help.

Thanks,
Karl.

[devep (devepone)]

Hi! Is there any workaround for this available?
I just encountered this on 2 seperate openstack deployments with Neutron DVR. I believe this should be supported.
Thank you

[Nicholas Skaggs (nskaggs)]

Let's look at this for 2.3

[John A Meinel (jameinel)]

AIUI the link-local subnets are only seen by "admin" style users of Openstack. If you create a tenant and give it only access to useful subnets, then we don't have the same problem with duplicate subnets.

We'll try to put something into 2.3 (bug #1710848) as a workaround, but the correct fix for modeling multiple networks with subnet ranges that can overlap is more of a 2.4 fix.

[John A Meinel (jameinel)]

I meant to say bug #1733266