"spaces" are not comprehensible
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
Undecided
|
Joseph Phillips |
Bug Description
Without reading the test scenarios in bridgepolicy_
And still, after reading bridgepolicy_
I am trying to get this very simple example here to work for days now.
I have been using MAAS 2.4.2 and 2.6.0 and Juju 2.6.8 and 2.6.9 in various combinations.
Apparently I want two networks on my MAAS hosts as well as in the lxd containers. The hosts run ceph-osd, while the ceph-mon units and ceph-fs run in LXD containers on these hosts.
MAAS VLANs, subnets, spaces are configured correctly (well apparently except for the spaces), I have access to the necessary spaces on the host machines and everything works well, if I do not resort to LXD for the ceph-mon units.
But if I try, I always end up with this error:
no obvious space for container "0/lxd/0", host machine has spaces: storage, storagefront, ...
Test scenarios producing the same error are:
func (s *bridgePolicySt
// The host machine will be in 2 spaces, but neither one is 'default',
// thus we are unable to find a valid space to put the container in.
func (s *bridgePolicySt
// The host machine will be in 2 spaces, but neither one is 'default',
// thus we are unable to find a valid space to put the container in.
Looks like I have to go for this scenario to make things work:
func (s *bridgePolicySt
// The host machine has 2 network devices and 2 bridges, but none of them
// are in a known space. The container also has no requested space.
// In that case, we will use all of the unknown bridges for container
// devices.
or this
func (s *bridgePolicySt
// There is a "default" and "dmz" space, and our machine has 2 network
// interfaces, but is not part of any known space. In this circumstance,
// we should try to bridge all of the unknown space devices, not just one
// of them. This is are fallback mode when we don't understand the spaces of a machine.
Judging from the test scenarios, it sounds like it is not allowed to request two spaces for a container, which does not appear to make much sense from my point of view. Consider the bundle below, which I had been biting my teeth on for the last three days.
This is a rather straight forward real life example.
Ceph is best deployed with a dedicated cluster network (space storage) for background data operations and a dedicated network for client requests (space soragefront).
As I want my ceph-mon nodes to reside on the ceph-osd hosts, I got to go with LXD as the two do not mix well (they overwrite each others configs if you put them on the same host directly).
applications:
ceph-mon:
charm: 'cs:~openstack-
num_units: 3
options:
expected-
ceph-
ceph-
pg-autotune: 'true'
series: bionic
annotations:
gui-x: '750'
gui-y: '500'
to:
- 'lxd:0'
- 'lxd:1'
- 'lxd:2'
bindings:
public: storagefront
cluster: storage
ceph-osd:
charm: 'cs:~openstack-
num_units: 3
options:
autotune: true
bluestore: true
bluestore
bluestore-db: /dev/nvme0n1
ceph-
ceph-
osd-devices: /dev/sdb /dev/sdc
series: bionic
annotations:
gui-x: '1000'
gui-y: '500'
to:
- '0'
- '1'
- '2'
- '3'
bindings:
public: storagefront
cluster: storage
ceph-fs:
charm: 'cs:~openstack-
num_units: 1
options:
ceph-
series: bionic
annotations:
gui-x: '488'
gui-y: '511'
to:
- 'lxd:3'
bindings:
public: storagefront
ntp:
charm: 'cs:ntp-35'
series: bionic
annotations:
gui-x: '1000'
gui-y: '0'
relations:
- - 'ceph-osd:mon'
- 'ceph-mon:osd'
- - 'ntp:juju-info'
- 'ceph-osd:
- - 'ceph-mon:mds'
- 'ceph-fs:ceph-mds'
machines:
'0':
series: bionic
constraints: tags=storage
'1':
series: bionic
constraints: tags=storage
'2':
series: bionic
constraints: tags=storage
'3':
series: bionic
constraints: tags=storage
I'll try again, getting rid of all space assignments beforehand. According to the test scenarios, I should be lucky then.
Reading this in bridgepolicy.go sounds like a container can only have ONE single space.
// inferContainerS
// on. This should only be used when the container itself doesn't have any
// valid constraints on what spaces it should be in.
// If ContainerNetwor
// If this machine is in a single space, then that space is used. Else, if
// the machine has the default space, then that space is used.
// If neither of those conditions is true, then we return an error.
As said above it evades me, why a single space for a container should be a requirement? Isn't it a much more real life scenario, that I want to attach different containers to different sets of networks (sets as in multiple)?
Be it as it may, the documentations does not mention any such requirement nor does it explain, what the "space" construct is intended to be used for. At first sight it seems redundant, as it appears to be nothing more than a naming scheme to allow charms to request being placed in certain subnets without having to type numbers.
Or it might be a way for grouping such subnets into a "space" - ergo into networking scenarios. But when I am not allowed to assign certain subnets to more than one space it seems to make the "space" construct quite restricted as well and unfit for most thinkable scenarios.
So what is it's intended purpose?
Judging from the tests, it seems to be only for partitioning the network into domains of ownership, so department A does not interfere with department B.
https:/
"From a security standpoint, with spaces, the Juju environment network topology can be organised in a way such that applications possess only the network connectivity they require."
And https:/
So assigning to spaces to one container should be fine, shouldn't it?
I am offering to update the Juju documentation with a comprehensive explanation if anybody can help me understand it's intended purpose.
From what I know so far, I will probably only have one single space for the time being.
My best regards,
Markus
Changed in juju: | |
status: | New → Triaged |
assignee: | nobody → Joseph Phillips (manadart) |
Changed in juju: | |
status: | Triaged → In Progress |
Changed in juju: | |
milestone: | none → 2.7-beta1 |
Changed in juju: | |
milestone: | 2.7-beta1 → 2.7-rc1 |
Changed in juju: | |
status: | In Progress → Fix Committed |
tags: |
added: maas-provider removed: maas |
Changed in juju: | |
status: | Fix Committed → Fix Released |
Hi Markus,
Thanks for the thorough report.
We're working on Juju networking and spaces right now. The documentation should get some love in the process, but to summarise; a space is intended to be a collection of subnets with common ingress and egress rules.
There is no mandate that a container has access just one space. We should be ensuring that the host has a bridge for every space that the container requires access to based on constraints and endpoint bindings, those then being parents of NIC devices in the container.
The unique space requirement only falls out when we fail to make a determination in the first case and fall through to choosing one from those available. If there a more than one we error.
I am looking into this now, because based on the endpoint bindings in your bundle, the spaces required by the container should be correctly determined without falling back to inference. There might be a race between the bundle processing and the provisioner. I'll try to dig it up.