be able to set juju management network

[Alvaro Uria]

Juju environment variables do not allow a "juju-management-network" setting to define a certain network block to be used to manage an environment.

By defining ie: juju-management-network="172.20.0.0/23", we would make sure Juju state servers would only be listening on IP addresses in such IP pool, and nodes' "private-address" would be chosen from 172.20.0.0/23 network, too.

We have an OpenStack scenario with several network blocks reserved for Ceph traffic, OpenStack API (OS-API) endpoints, OpenStack Management (OS-Mgmt) network, etc. and we would like to limit Juju traffic to the OS-Mgmt network.

On the other hand, LXCs created (Juju 1.25; Trusty) are configured on both OS-API and OS-Mgmt, and Juju is choosing OS-API IP as private-address. Ideally (as said), we would prefer OS-Mgmt IP to be chosen as private address.

Finally, when juju ensure-availability is set, not filtering state servers IPs (to only listening on ie OS-Mgmt) would force all state servers to have same network blocks set, or juju-run could potentially timeout.

[James Tunnicliffe (dooferlad)]

This is solved through the use of spaces with Juju 2.0. Please see https://jujucharms.com/docs/devel/network-spaces for more information.

[Alvaro Uria (aluria)]

Hi,

Thank you for the information, James.

On network-spaces documentation, I see Juju 2.0 does not have support for MaaS. Could you please give us an ETA for such feature? Next release cycle will be in the Fall, right? On the other hand, such support would need to set "disable-network-management: false", wouldn't it? We currently configure our own /etc/network/interfaces file with the environment variable enabled (disable-network-management: true).

Please consider reviewing this feature priority. Having a management tool using production networks is something we would like to limit as much as possible.

Thanks again for your help.

-Alvaro.

[Cheryl Jennings (cherylj)]

Hi Alvaro, Juju 2.0 does support both MAAS 1.9 and MAAS 2.0. If you are configuring your own /etc/network/interfaces and you do not want Juju to modify it, then yes, you would need to set disable-network-management to true. Just be aware that you may not be able to deploy to containers if that config option is set.

[Peter Sabaini (peter-sabaini)]

Cheryl, from the link James posted above, network spaces are not supported with MAAS yet.

[Cheryl Jennings (cherylj)]

Yeah, the docs are wrong. The only substrate that supports spaces is MAAS.

Opened an issue against the docs page: https://github.com/juju/docs/issues/1210

Thanks for catching, Peter!

[Mick Gregg (macgreagoir)]

I don't think spaces yet solves this issue, so I'm re-opening this for discussion.

[Peter Sabaini (peter-sabaini)]

I wonder about the impact of juju agents not being reachable by state servers.

In the scenario Alvaro has mentioned above juju agents can reach state servers but not vice versa. One immediate fallout is that juju run and juju actions don't work on those units. Is there any other impact to be expected in this scenario?

[John A Meinel (jameinel)]

The actual way in which this would be implemented would be a binding for the Juju Controller into a given space.
There are a few considerations:
1) "juju" the client also talks to the controller from your laptop. So only exposing OS-MGMT addresses may mean that you could no-longer run Juju commands. We could try to split the values into GUI vs Client vs Agent APIs.

It may be that we advertise all of the values, but give preference to the ones that fit a given subset? (Or at least always listen on all addresses, if we don't advertise them.)

2) For the other question about Controller reaching Agents, agents other than the Controller don't actually listen on a port. Its possible we would try to SSH, but the last checks we've seen said that Agents do the right thing and watch for actions/juju run requests rather than needing to be dialed from the controller.
It seems there might still be a bug there, but it should be considered independent from this bug.

[Alvaro Uria (aluria)]

Hey John, thank you for your reply.

> It may be that we advertise all of the values,
> but give preference to the ones that fit a
> given subset? (Or at least always listen on all
> addresses, if we don't advertise them.)

Ops guys like to have multiple config options with default values.
1)
I would suggest to allow specifying which network juju state servers listen at, having 0.0.0.0 (ANY) by default. As an alternative, jujud could always listen on 0.0.0.0 and sysadmins would set firewall rules to allow jujuclients (over API network) + jujuagents (over MGMT network).

2)
Regarding "juju run" and "juju actions": as you know, juju-db stores the preferred private address of each unit, but it looks like jujud has its own algorithm to determine which network will be used (which looks unfit as it relies on "string'ed IP" [not the integer] sorting - see sample at https://pastebin.canonical.com/177526/).
I would recommend enforcing Juju to use such network (stored in mongo), while it could fallback (retry) on the other networks found. In the past, we've tried to REJECT (tcp-reset) ssh connections from machine-0 to env units over unwanted networks (ie. juju-run via Storage Cluster network), but we found Juju didn't try other networks but quickly returned "connection refused".

[Horacio Durán (hduran-8)]

There is an ugprade path happening from 1.25.x -> 2.x in late April so you should be able to have this fix in your 1.25 deployment since 2.3 is just before the upgrade.

[John A Meinel (jameinel)]

I feel like we need to break this down into several bugs as they'll all need a little bit of a different implementation to fix.

1) Fix 'juju run' and actions to work even when IP addresses are confusing. The rule *should* be that Agents must have a route to some of the Controllers, but controllers shouldn't need direct ingress to the agents.

2) Be able to give specific spaces that you want pieces of the Controller infrastructure to run on. (What IP addresses do you want to advertise for clients, what addresses do you want to advertise for agents, what IP addresses should Mongo use to configure the replica set, etc)

3) Be able to have preferred addresses for SSH access. This feels potentially very machine specific. Is it per-application? Is it a hierarchy of preferences for spaces? If you have host machines that are potentially in 4 spaces, but containers that aren't exposed to all of them, would the preferred address for SSH differ for each? Would the preferred address differ if you are inside the network rather than outside. Would it differ if someone was using --proxy to go via the Controllers rather than going directly to the hosts.

In 2.1 we should be trying to connect to all addresses, seeing which ones respond with an appropriate ssh handshake and appear to have a public key matching what the host reported it was using.

4) There may be something about private addresses for 'juju run' etc, and while the sorting issue is an old problem with 1.25, it is true that Spaces concretely address the idea of what IP address to report for a given application (and if we extend it to Juju controllers/infrastructure), but it doesn't really create a preference for Machines when referenced directly.

[Tim Penhey (thumper)]

This has hit the roadmap for 18.04.

[John A Meinel (jameinel)]

https://github.com/juju/juju/pull/8207

is a step along this path.

[Anastasia (anastasia-macmood)]

This is a feature not a bug.

We are actively working on it right now. There are a lot of PRs going in into 2.4 to deliver this functionality. It is not practical to link all code changes here.

Since the PR linked here has already landed, I'll mark this report as Fix Committed.

However, I am wondering if it's best to track Feature Request as blueprints with Work Items, rather than as a bug report as here....

[Anastasia (anastasia-macmood)]

Marking as Fix Released as 2.4.0 is out.