juju does not support --to <zone> placement directive in bundles

[Dmitrii Shcherbakov]

While it is possible to do something like

juju deploy cs:xenial/ceph-osd --to az1

via CLI, there is no such placement directive like zone:<azname> for bundles, therefore, one cannot say “9 units, 3 into every AZ”.

https://github.com/juju/juju/blob/juju-2.4.3/cmd/juju/application/bundle.go

Likewise, it is not possible to add a machine to a specific AZ using a bundle.

The use-case for that is making sure that a container placement directive refers to a machine in a certain AZ, e.g.:

applications:
  ceph-mon:
  num_units: 3
  # ...
  # not sure if ceph mons will be in the right AZ
  to:
  - lxd:0
  - lxd:1
  - lxd:2

It would also be nice to have:

applications:
  ceph-mon:
  num_units: 3
  # ..
  to:
  - lxd:zone1
  - lxd:zone2
  - lxd:zone3

Something like this is also not supported:

machines:
  0:
    zone: <az-name>

See also: https://discourse.jujucharms.com/t/better-control-for-unit-placement-into-availability-zones/212

[Dmitrii Shcherbakov (dmitriis)]

Added field-high due to relevance for ongoing MAAS and OpenStack Juju provider deployments.

The workaround is usage of manual machine allocation to a given zone(s):

for i in {0..5} ; do juju add-machine zone=az1 ; done
for i in {0..5} ; do juju add-machine zone=az2 ; done
for i in {0..5} ; do juju add-machine zone=az3 ; done

# add units to machines according to their <number>:<availability zone> allocation

juju deploy --map-machines=existing <bundle>

[Richard Harding (rharding)]

Thanks, we're working to get this into the 2.5 release. If the fix is clean I'll see if we can backport to previous ones as well.

[John A Meinel (jameinel)]

Interesting, I would have thought you would prefer:

 for i in {0..5}; do juju add-machine zone=az1; juju add-machine zone=az2;
juju add-machine zone=az3; done

Is there a reason you prefer the machines to be grouped by AZ and not round
robin?

On Tue, Oct 30, 2018 at 2:55 AM Dmitrii Shcherbakov <
<email address hidden>> wrote:

> Added field-high due to relevance for ongoing MAAS and OpenStack Juju
> provider deployments.
>
> The workaround is usage of manual machine allocation to a given zone(s):
>
> for i in {0..5} ; do juju add-machine zone=az1 ; done
> for i in {0..5} ; do juju add-machine zone=az2 ; done
> for i in {0..5} ; do juju add-machine zone=az3 ; done
>
> # add units to machines according to their <number>:<availability zone>
> allocation
>
> juju deploy --map-machines=existing <bundle>
>
> --
> You received this bug notification because you are a member of Canonical
> Field High, which is subscribed to the bug report.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1791715
>
> Title:
> juju does not support --to <zone> placement directive in bundles
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1791715/+subscriptions
>

[Dmitrii Shcherbakov (dmitriis)]

Round-robin makes it a bit harder to place containers in bundles and make
changes.

to:
  - lxd:15
  - lxd:16
  - lxd:17

The above is not very descriptive. If you make any changes to numbering or
accidentally skip one number in a bundle failure domains will shift.

Ideally, container placement decisions should be based on a combination of
constraints (networking, tags, storage, cpu, memory) and zone placement
directives.

num_units: 3
constraints:
  tags: converged
to:
  - lxd:zone:az1
  - lxd:zone:az2
  - lxd:zone:az3
bindings:
  "": oam-space
  someendpoint: somespace

If I understand correctly there is currently no way to say "place into
containers on nodes matching certain constraints and spread containers
across underlying node availability zones" - one has to always specify
where to place a container.

I think both cases are valid:

   1. having --to lxd:zone:<azname> or --to zone:<azname> supported in
   bundles;
      - this is needed in cases where we have some availability zones used
      which should be excluded from use in a given model since we do not have a
      model-config to restrict AZs to be used;
      2. not having it (just num_units + constraints with containers
   allocated on any matching nodes with AZ spread).
   1. something like --to lxd:any

On Wed, Oct 31, 2018 at 10:30 AM John A Meinel <email address hidden>
wrote:

> Interesting, I would have thought you would prefer:
>
> for i in {0..5}; do juju add-machine zone=az1; juju add-machine zone=az2;
> juju add-machine zone=az3; done
>
> Is there a reason you prefer the machines to be grouped by AZ and not round
> robin?
>
> On Tue, Oct 30, 2018 at 2:55 AM Dmitrii Shcherbakov <
> <email address hidden>> wrote:
>
> > Added field-high due to relevance for ongoing MAAS and OpenStack Juju
> > provider deployments.
> >
> > The workaround is usage of manual machine allocation to a given zone(s):
> >
> > for i in {0..5} ; do juju add-machine zone=az1 ; done
> > for i in {0..5} ; do juju add-machine zone=az2 ; done
> > for i in {0..5} ; do juju add-machine zone=az3 ; done
> >
> > # add units to machines according to their <number>:<availability zone>
> > allocation
> >
> > juju deploy --map-machines=existing <bundle>
> >
> > --
> > You received this bug notification because you are a member of Canonical
> > Field High, which is subscribed to the bug report.
> > Matching subscriptions: juju bugs
> > https://bugs.launchpad.net/bugs/1791715
> >
> > Title:
> > juju does not support --to <zone> placement directive in bundles
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/juju/+bug/1791715/+subscriptions
> >
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1791715
>
> Title:
> juju does not support --to <zone> placement directive in bundles
>
> Status in juju:
> Triaged
>
> Bug description:
> While it is possible to do something like
>
> juju deploy cs:xenial/ceph-osd --to az1
>
> via CLI, there is no such placement directive like zone:<azname> for
> bundles, therefore, one cannot say “9 units, 3 into every AZ”.
>
>
> https://github.com/juju/ju...

[John A Meinel (jameinel)]

Don't you have the same problems if you get the numbers wrong in sequence?
1-5, 6-10, 11-15 still ends up in the wrong zone if you do 5 vs 6. But I
suppose it's more obvious with:
  lxd:1
  lxd:6
  lxd:11

(its easier to do divide-by-5 math than mod 3 in your head, though that
would then require N-per-zone to be easy math)

I know we've generally avoided doing any automatic scheduling of
containers, but given the design of putting all the interesting control
plane into containers it pushes harder on wanting scheduling. There is a
lot of difficulty once you get into multiple applications that could be
scheduled together. (with machines we default to 1 primary per machine, but
with containers you don't have that anymore). And Juju doesn't know if 2
applications are both CPU intensive, etc.
John
=:->

On Oct 31, 2018 21:40, "Dmitrii Shcherbakov" <
<email address hidden>> wrote:

Round-robin makes it a bit harder to place containers in bundles and make
changes.

to:
  - lxd:15
  - lxd:16
  - lxd:17

The above is not very descriptive. If you make any changes to numbering or
accidentally skip one number in a bundle failure domains will shift.

Ideally, container placement decisions should be based on a combination of
constraints (networking, tags, storage, cpu, memory) and zone placement
directives.

num_units: 3
constraints:
  tags: converged
to:
  - lxd:zone:az1
  - lxd:zone:az2
  - lxd:zone:az3
bindings:
  "": oam-space
  someendpoint: somespace

If I understand correctly there is currently no way to say "place into
containers on nodes matching certain constraints and spread containers
across underlying node availability zones" - one has to always specify
where to place a container.

I think both cases are valid:

   1. having --to lxd:zone:<azname> or --to zone:<azname> supported in
   bundles;
      - this is needed in cases where we have some availability zones used
      which should be excluded from use in a given model since we do not have a
      model-config to restrict AZs to be used;
      2. not having it (just num_units + constraints with containers
   allocated on any matching nodes with AZ spread).
   1. something like --to lxd:any

On Wed, Oct 31, 2018 at 10:30 AM John A Meinel <email address hidden>
wrote:

> Interesting, I would have thought you would prefer:
>
> for i in {0..5}; do juju add-machine zone=az1; juju add-machine zone=az2;
> juju add-machine zone=az3; done
>
> Is there a reason you prefer the machines to be grouped by AZ and not round
> robin?
>
> On Tue, Oct 30, 2018 at 2:55 AM Dmitrii Shcherbakov <
> <email address hidden>> wrote:
>
> > Added field-high due to relevance for ongoing MAAS and OpenStack Juju
> > provider deployments.
> >
> > The workaround is usage of manual machine allocation to a given zone(s):
> >
> > for i in {0..5} ; do juju add-machine zone=az1 ; done
> > for i in {0..5} ; do juju add-machine zone=az2 ; done
> > for i in {0..5} ; do juju add-machine zone=az3 ; done
> >
> > # add units to machines according to their <number>:<availability zone>
> > allocation
> >
> > juju deploy --map-machines=existing <bundle>
> >
> > --
> > You received this bug notification because you are ...

[Dmitrii Shcherbakov (dmitriis)]

> There is a lot of difficulty once you get into multiple applications that
could be scheduled together.

In a long run this looks quite complicated. Besides az-aware scheduling,
affinity and anti-affinity and constraints there is also device-aware
scheduling (cpu model, PCIe devices) if container profiles contain devices
passed through. Tracking available vs allocated PCIe devices is another
concern.

I think it all leads to the need for a proper scheduler for containers but
this is a large feature considering the support for different providers.

So far we have been mostly placing control plane components into containers
which do not require anything being passed through.

Placing hypervisor components into LXD brings passing devices into the mix
but probably on "one unit per node gets all devices" basis so no usage
tracking needs to happen for a base case. We would then let Nova handle
allocation in case of OpenStack.

For storage components we often use dmcrypt + lvm which is problematic in
unprivileged containers. There is also a problem with the fact that
curtin-generated udev rules for creating
/dev/disk/by-dname/<device-name-in-maas> symlinks have to be copied to
containers for us to use persistent block device names.

https://lists.linuxcontainers.org/pipermail/lxc-users/2014-January/006076.html
https://lists.linuxcontainers.org/pipermail/lxc-users/2014-January/006078.html

To sum up: editing bundles is currently a pain as everything is hard-wired
to machine numbers dynamically assigned to AZs and we mostly care about
"simple" containers as of today.

By the looks of it:

1) machine declarations in bundles with specific AZs would allow us to
remove the dynamic part in initial bundles;
2) --to zone:<zone name> for units would allow bare-metal-only components
to be placeholders in addition to using machine declarations;
3) --to lxd:zone:<zone name> plus evenly spreading units of the same
application would allow us to avoid hard-wired placements in many cases.

On Mon, Nov 5, 2018, 05:55 John Meinel, <email address hidden> wrote:

> Don't you have the same problems if you get the numbers wrong in sequence?
> 1-5, 6-10, 11-15 still ends up in the wrong zone if you do 5 vs 6. But I
> suppose it's more obvious with:
> lxd:1
> lxd:6
> lxd:11
>
> (its easier to do divide-by-5 math than mod 3 in your head, though that
> would then require N-per-zone to be easy math)
>
> I know we've generally avoided doing any automatic scheduling of
> containers, but given the design of putting all the interesting control
> plane into containers it pushes harder on wanting scheduling. There is a
> lot of difficulty once you get into multiple applications that could be
> scheduled together. (with machines we default to 1 primary per machine, but
> with containers you don't have that anymore). And Juju doesn't know if 2
> applications are both CPU intensive, etc.
> John
> =:->
>
> On Oct 31, 2018 21:40, "Dmitrii Shcherbakov" <
> <email address hidden>> wrote:
>
> Round-robin makes it a bit harder to place containers in bundles and make
> changes.
>
> to:
> - lxd:15
> - lxd:16
> - lxd:17
>
> The above is not very descriptive. If you mak...

[Joseph Phillips (manadart)]

Discussions around this surfaced the idea of supporting availability zones as constraints.
There is a patch proposed here:
https://github.com/juju/juju/pull/9481

[Richard Harding (rharding)]

The fix is supported zones as a constraint option vs a placement directive in Juju 2.5.

See the new zones section in the docs https://docs.jujucharms.com/devel/en/reference-constraints

[Dmitrii Shcherbakov (dmitriis)]

I think this should be marked as fix-released as Juju 2.5 supports zone constraints now.

https://github.com/juju/juju/pull/9481