Juju 2.9 failing to deploy lxd containers with lxd latest/stable as lxd version 5.x is promoted to latest/stable

[Diko Parvanov]

As mentioned in https://juju.is/docs/olm/lxd juju 2.9 only works with lxd 4.x and juju 3.x works with lxd 5.x.

Current deployments that haven't upgraded to 3.x will have issues deploying new containers as the default value for juju model-config lxd-snap-channel is "latest/stable"

Workaround to fix this is to change juju model-config lxd-snap-channel="4.0/stable" and sudo snap remove lxd on the affected machines, as snap lxd downgrades are not supported.

Juju should be able to detect that it is running 2.9 and not 3.x and deploy the latest working lxd snap on the new machines, or at least warn (even better error out) if the user tries to deploy containers with lxd-snap-channel pointing to latest/stable that has 5.x promoted.

[Diko Parvanov (dparv)]

Another important point here is, if you set to 4.0/stable and add a container to a machine that already has containers running juju will try to downgrade lxd and break all existing containers:

t=2023-01-09T11:59:33+0000 lvl=eror msg="Failed to start the daemon" err="Error creating database: schema version '43' is more recent than expected '42'"
Error: Error creating database: schema version '43' is more recent than expected '42'
=> LXD failed to start

So the workaround for new machines would be to set to 4.0/stable, snap remove lxd, add new container so that juju can properly configure netplan and then rollback to latest/stable and during this time you should not add any containers to machines already hosting live containers.

[Paul Goins (vultaire)]

For clarity - the LXD issue is a "wontfix" per https://github.com/lxc/lxd/issues/11265.

[Heather Lanigan (hmlanigan)]

There was a PR which landed on 14-Nov to change lxd-snap-channel to default as 5.0/stable for juju 2.9.

https://github.com/juju/juju/pull/14875

However it didn't make it into 2.9.37 (by my testing). 2.9.38 is in candidate right now, and expected to release this week.

Acknowledging that this doesn't match our documentation. I'll double check on what it should be.

[Heather Lanigan (hmlanigan)]

@dparv, please file a new bug related to the different issue raised in #1, where juju changes the lxd snap version if already used on the machine.

[Heather Lanigan (hmlanigan)]

Will update the docs to say juju 2.9 is working with lxd 5.0

[Diko Parvanov (dparv)]

Tried 5.0/stable with juju 2.9.22 and adding new lxd container installs 5.0/stable snap, but netplan doesn't get configured either. It only works with 4.0/stable.

[Harry Pidcock (hpidcock)]

Hi Diko,

What series are you running for the machine and what series is the LXD container?

[Diko Parvanov (dparv)]

Tried adding containers after upgrading juju 2.9.22 to 2.9.37 and now I can successfully do it, so there is a fix already somewhere between 2.9.22 and 2.9.37.

[Trent Lloyd (lathiat)]

It's worth noting that the docs at "https://juju.is/docs/olm/lxd" I believe to be specific to the LXD provider, as opposed to LXD on a MAAS machine. I believe these have largely different code bases and aren't super related.

This issue was previously reported here:
https://bugs.launchpad.net/juju/+bug/1969470

Was marked a dupe of this bug:
https://bugs.launchpad.net/juju/+bug/1968849

Fixed by this commit:
https://github.com/juju/juju/pull/13957

Released in 2.9.31

Hence solved by the controller upgrade to 2.9.37