do not apply lxd profiles to (lxd) manual machine in a lxdcloud model

[rillettes]

After adding manual machines to a model in a LXD cloud, it tries to upgrade their profile:

~$ juju status
Model Controller Cloud/Region Version SLA Timestamp
maas localhost-localhost localhost/localhost 2.8.0 unsupported 11:12:14+02:00

App Version Status Scale Charm Store Rev OS Notes
postgresql 12.2 active 1 postgresql jujucharms 207 ubuntu
rackd 20.04 active 2 ubuntu jujucharms 15 ubuntu
regiond 20.04 active 2 ubuntu jujucharms 15 ubuntu

Unit Workload Agent Machine Public address Ports Message
postgresql/0* active idle 0 10.169.193.155 5432/tcp Live master (12.2)
rackd/0 active idle 3 192.168.122.2 ready
rackd/1* active idle 4 192.168.122.3 ready
regiond/0 active idle 1 192.168.2.52 ready
regiond/1* active idle 2 192.168.2.53 ready

Machine State DNS Inst id Series AZ Message
0 started 10.169.193.155 juju-73e0a6-0 focal Running
1 error 192.168.2.52 manual:192.168.2.52 focal cannot upgrade machine's lxd profile: 1: not found
2 error 192.168.2.53 manual:192.168.2.53 focal cannot upgrade machine's lxd profile: 2: not found
3 error 192.168.122.2 manual:192.168.122.2 focal cannot upgrade machine's lxd profile: 3: not found
4 error 192.168.122.3 manual:192.168.122.3 focal cannot upgrade machine's lxd profile: 4: not found

These are LXDs, but manually added to be able to use a specific LXD profile.
Machine 0 is using the default profile, the others use custom profiles to manage network connections.
afaik, there is no way to specify profiles to deploy charms (except very specific cases as described in https://juju.is/docs/lxd-cloud-advanced).

A similar issue was reported in lp:1883178. Juju was fixed to not update profiles for KVM manual machines, I'd find it sensible to not try to update the profile for any manual machine.

[Heather Lanigan (hmlanigan)]

The fix for 1883178 was to disallow profiles for any manual machine. It was in 2.7.7 and will be released with 2.8.1.

@rillettes, please check if this issue still occurs with 2.7.7 or the 2.8/candidate snap (2.8.1)

[rillettes (bliblu)]

Hi,

I performed the upgrade:
On the client side:
$ sudo snap refresh --channel latest/candidate juju
On the controller:
$ juju upgrade-model -m controller --agent-stream proposed
best version:
    2.8.1
started upgrade to 2.8.1

It claims running 2.8.1:
$ juju model-config -m controller agent-version
2.8.1

But When trying to upgrade the model:
$ juju upgrade-model -m maas --agent-stream proposed
best version:
    2.8.1
ERROR model cannot be upgraded to 2.8.1 while the controller is 2.8.0: upgrade 'controller' model first

Indeed on the controller machine (it's not HA-ed):
$ juju ssh -m controller 0 "ls -la /var/lib/juju/tools"
total 12
drwxr-xr-x 3 root root 4096 Jun 27 07:35 .
drwxr-xr-x 7 root root 4096 Jul 6 07:18 ..
drwxr-xr-x 2 root root 4096 Jun 27 07:35 2.8.0-focal-amd64
lrwxrwxrwx 1 root root 17 Jun 27 07:35 machine-0 -> 2.8.0-focal-amd64
Connection to 10.169.193.169 closed.

And in the logs:
$ juju ssh -m controller 0 "sudo grep 'agent binaries' /var/log/juju/machine-0.log | tail -10"
2020-07-06 07:23:37 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: no matching agent binaries available
2020-07-06 07:24:35 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: no matching agent binaries available
2020-07-06 07:25:42 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: no matching agent binaries available
2020-07-06 07:27:09 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: no matching agent binaries available
2020-07-06 07:28:54 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: no matching agent binaries available
2020-07-06 07:30:39 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: no matching agent binaries available
2020-07-06 07:32:28 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: no matching agent binaries available
2020-07-06 07:34:31 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: no matching agent binaries available
2020-07-06 07:36:26 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: no matching agent binaries available
2020-07-06 07:38:33 ERROR juju.worker.dependency engine.go:671 "upgrader" manifold worker returned unexpected error: no matching agent binaries available
Connection to 10.169.193.169 closed.

This particular issue isn't directly relevant for this bug, but blocking the tests so a pointer would be helpful.
I can file a separate bug if needed.

Thanks !

[Heather Lanigan (hmlanigan)]

It looks like you've hit: https://bugs.launchpad.net/juju/+bug/1742342. I've updated it with more info.

There are 2 options depending on your risk level.

No risk: Leave as is until 2.8.1 is release this week, and upgrade of the controller will happen once the released agent is available. Then you can upgrade the model.

Some risk if we need to spin a new 2.8.1 candidate before the released version goes out. You will not able to upgrade from 2.8.1 proposed, to 2.8.1 released.
   Allow the current upgrade to continue by running:
     juju model-config -m controller agent-stream=proposed
   Then before upgrading the model, do the same for each model to be upgraded:
     juju model-config agent-stream=proposed

[rillettes (bliblu)]

Thanks.
The issue reported here is not blocking, so I prefer waiting for 2.8.1 released.

[rillettes (bliblu)]

Hi,

After 2.8.1 was released, the upgrade happened automatically on the controller and I could upgrade the model.
However, the machines are still in error:
1 error 192.168.2.52 manual:192.168.2.52 focal cannot upgrade machine's lxd profile: 1: not found
2 error 192.168.2.53 manual:192.168.2.53 focal cannot upgrade machine's lxd profile: 2: not found
3 error 192.168.122.2 manual:192.168.122.2 focal cannot upgrade machine's lxd profile: 3: not found
4 error 192.168.122.3 manual:192.168.122.3 focal cannot upgrade machine's lxd profile: 4: not found

How can I clear them ?

Thanks.

[Heather Lanigan (hmlanigan)]

The controller should never be upgraded automatically. When 2.8.1 was released to snap, the local juju client was upgraded.

Please double check the controller version by looking at the 'juju status -m controller', it will include the juju version running on the controller. And what version of juju is the model with this manual machines it running?

[rillettes (bliblu)]

Hi,

The controller: https://pastebin.ubuntu.com/p/374Bw6dQ2T/
The model: https://pastebin.ubuntu.com/p/y3bWcPGGYw/

Thanks.

[Pen Gale (pengale)]

I note that there has not been activity on this in a while. Did the situation get resolved? Or do we need to dig into this further?

[Launchpad Janitor (janitor)]

[Expired for juju because there has been no activity for 60 days.]