Application deployment onto clustered lxd results in unpredictable placement

[Gábor Mészáros]

I'm deploying on top of maas provider into lxc containers.
Latest juju snap 2.8.1 13324 with agents also matching version; maas 2.8.1-8567-g.c4825ca06 7808.
I have deployed ubuntu on 4 nodes and then installed latest snap lxd (4.3, 16100) onto them, and went through the clustering configuration (lxd init).
Once that's done I've deployed a bundle with the following placement:
machines:
  "0":
  "1":
  "2":

applications:
  ceph-mon:
...
    to:
    - lxd:0

  ceph-osd:
...
    to:
    - lxd:0
    - lxd:1
    - lxd:2

This has resulted in the following container list:
root@edge2-4:~# lxc list
+---------------------+---------+-----------------------+------+-----------+-----------+----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | LOCATION |
+---------------------+---------+-----------------------+------+-----------+-----------+----------+
| juju-410afd-0-lxd-0 | RUNNING | 192.168.30.208 (eth0) | | CONTAINER | 0 | edge2-1 |
| | | 172.40.2.8 (eth1) | | | | |
+---------------------+---------+-----------------------+------+-----------+-----------+----------+
| juju-410afd-0-lxd-1 | RUNNING | 192.168.30.209 (eth0) | | CONTAINER | 0 | edge2-4 |
| | | 172.40.3.8 (eth2) | | | | |
| | | 172.40.2.9 (eth1) | | | | |
+---------------------+---------+-----------------------+------+-----------+-----------+----------+
| juju-410afd-3-lxd-1 | RUNNING | 192.168.30.210 (eth0) | | CONTAINER | 0 | edge2-1 |
| | | 172.40.3.9 (eth2) | | | | |
| | | 172.40.2.10 (eth1) | | | | |
+---------------------+---------+-----------------------+------+-----------+-----------+----------+
| juju-410afd-3-lxd-2 | RUNNING | 192.168.30.211 (eth0) | | CONTAINER | 0 | edge2-2 |
| | | 172.40.3.10 (eth2) | | | | |
| | | 172.40.2.11 (eth1) | | | | |
+---------------------+---------+-----------------------+------+-----------+-----------+----------+

Notice that eg both 0-lxd-0 and 3-lxd-1 landed on edge2-1 (edge2-1 got machine number 0, edge2-2 is machine 1, edge2-4 is machine 3).
Also 3-lxd-1 and 3-lxd-2 are on separate machines.

This is due to the default lxd cluster behavior, where it spreads the containers evenly on the hosting nodes. It can be overridden by supplying the --target to the lxc create command during container creation time.

The only workaround I have found is to move the containers after their deployment is complete:
root@edge2-1:~# lxc stop juju-410afd-3-lxd-2
root@edge2-1:~# lxc move juju-410afd-3-lxd-2 --target edge2-4

But this could be done by juju during container deployment time.

[Joseph Phillips (manadart)]

Hi Gábor.

From what you have described, you are *not* using a LXD cloud, but rather a MAAS cloud on which you have set up a LXD cluster.

The LXD provider does support container placement - we treat each server as an availability zone, so you can use the --to syntax to force where a container goes. But this is not the same as local container management on physical nodes, which does not reason about clustering.

You have two options.

1. After setting up the cluster, add it to the same controller as another cloud, so you can deploy containers on to the machines you want.

2. Simply don't use clustering. By default Juju gets container addresses from MAAS when you deploy something to lxd:x. All the containers should be able to access each other and they end up on the exact node you specify.

I would favour option 2, as it is the simplest.

The main question here is, what are you hoping to gain by using a cluster in this way?

[Gábor Mészáros (gabor.meszaros)]

Thanks.

I am using the lxd:x placement in the maas provider case with clustering.

If I go option 1. and bootstrap a lxd cloud, then how can I specify which physical host I want my container to run on? All I see is that the controller container got created on top of one of the host, but then I still don't see from the juju status where that machine has landed in terms of physical placement.

[Joseph Phillips (manadart)]

You should be able to add the LXD cluster as a cloud to the existing controller; see the help for "add-cloud":

    juju add-cloud --controller mycontroller mycloud --credential mycred

As to what physical machine the containers get created on, that would have to be inferred from the LXD server to MAAS node relationship.

It's a little fiddly, which is why I think the first option is better.

There is an unanswered question about from my previous comment regarding what benefit clustering affords in this set-up. An answer for that might help us with the best path forward.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.