juju upgrade-machine failed to prepare lxd hosts

[Shunde Zhang]

A user wants to upgrade OS of every host in a juju deployed cluster.
In this cluster, some applications are running as LXD containers on baremetal machines.

A simple reproducer is like below:

Model Controller Cloud/Region Version SLA Timestamp
cdk24 maas-cloud-default maas-cloud/default 2.9.29 unsupported 02:30:20Z

App Version Status Scale Charm Channel Rev Exposed Message
ubuntu 22.04 active 1 ubuntu stable 21 no

Unit Workload Agent Machine Public address Ports Message
ubuntu/0* active idle 6/lxd/0 192.168.1.72

Machine State Address Inst id Series AZ Message
6 started 192.168.1.254 cute-fowl focal default Deployed
6/lxd/0 started 192.168.1.72 juju-2233af-6-lxd-0 focal default Container started

It is Ok to upgrade OS of container 6/lxd/0.

ubuntu@maas:~$ juju upgrade-series 6/lxd/0 prepare jammy
WARNING: This command will mark machine "6/lxd/0" as being upgraded to series "jammy".
This operation cannot be reverted or canceled once started.
Units running on the machine will also be upgraded. These units include:
  - ubuntu/0

Leadership for the following applications will be pinned and not
subject to change until the "complete" command is run:
  - ubuntu

Continue [y/N]?

However this command failed for machine 6.

ubuntu@maas:~$ juju upgrade-series 6 prepare jammy
ERROR units for machine "6" not found

After a discussion [1] with tlmiller in charmhub's mattermost channel, it looks this is caused by a check in juju client [2]. This check is added to the client from Juju 2.9.

On the other hand, the user's model is still 2.7 so downgrading juju client to 2.7 allows the upgrade to move forward since there is no such check in juju client 2.7.

As mentioned by tlmiller, if the model is 2.9 or above, a workaround is to deploy a dummy application to the host machine, like below.

juju deploy ubuntu dummy --to 6

I've tested this in my lab and it works.
After having a dummy application, I was able to upgrade the OS of machine 6.

After all, it might be better to review this check in the code and see if it can be removed in a new version. If this is to be removed, could you please remove it in the next 2.9 release too?

[1] https://chat.charmhub.io/charmhub/pl/mxhd94k7ntd77bwheqwbjkxqky
[2] https://github.com/juju/juju/blob/ff8bddb7e83b90a14d610e877377eea961a676ec/cmd/juju/machine/upgrademachine.go#L286

[Ian Booth (wallyworld)]

The next 2.9.46 candidate release will not include a fix for this bug and we don't plan on any more 2.9 releases. As such it is being removed from its 2.9 milestone.

If the bug is still important to you, let us know and we can consider it for inclusion on a 3.x milestone.