2.9 controller and 2.8 model - cannot provision lxd containers

[Ian Booth]

bootstrap a 2.8 controller on aws or maas or openstack (or ...) and add a model
upgrade the controller to 2.9
it's not possible to then provision a container on a machine

$ juju add-machine lxd
$ juju status
...
0/lxd/0 down pending bionic unknown OS for series: "ubuntu"
...

The root cause is a 2.8 bug in the lxd borker which provisions containers. The series to provision is passed in via the instance config args. But we ignore that and instead get the series from the tools metadata. A 2.9 controller populates tools metadata with os type, not series. It wouldn't be an issue if the 2.8 code did not have the bug, and this has already been fixed in 2.9

However, we need to support older, buggy juju releases so a server side tweak to FindTools() is needed. There's already 2.8 compatibility added for upgrade-model - that needs to be extended for this case also.

[Ian Booth (wallyworld)]

https://github.com/juju/juju/pull/12943

[John A Meinel (jameinel)]

I just ran into this with just an LXD cloud and a 2.8 controller and model
then upgraded the controller to 2.9 (but not the model). At this point, I
did "juju deploy app" and it is now stuck for all units:
2021-05-04 13:20:29 ERROR juju.worker.dependency engine.go:671
"unit-agent-deployer" manifold worker returned unexpected error: cannot
read agent metadata in directory /var/lib/juju/tools/2.8.10-bionic-amd64:
open /var/lib/juju/tools/2.8.10-bionic-amd64/downloaded-tools.txt: no such
file or directory.

There does exist:
/var/lib/juju/tools/2.8.10-ubuntu-amd64/downloaded-tools.txt

But obviously 'ubuntu' vs 'bionic'

We should check that this is also fixed by whatever patch we do here.

On Mon, May 3, 2021 at 9:20 AM Ian Booth <email address hidden> wrote:

> ** Changed in: juju
> Status: In Progress => Fix Committed
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1926826
>
> Title:
> 2.9 controller and 2.8 model - cannot provision lxd containers
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1926826/+subscriptions
>

[John A Meinel (jameinel)]

Note that this is just doing a new machine (not logically a container) with
a 2.8 agent from a 2.9 controller.

On Tue, May 4, 2021 at 9:36 AM John Meinel <email address hidden> wrote:

> I just ran into this with just an LXD cloud and a 2.8 controller and model
> then upgraded the controller to 2.9 (but not the model). At this point, I
> did "juju deploy app" and it is now stuck for all units:
> 2021-05-04 13:20:29 ERROR juju.worker.dependency engine.go:671
> "unit-agent-deployer" manifold worker returned unexpected error: cannot
> read agent metadata in directory /var/lib/juju/tools/2.8.10-bionic-amd64:
> open /var/lib/juju/tools/2.8.10-bionic-amd64/downloaded-tools.txt: no such
> file or directory.
>
> There does exist:
> /var/lib/juju/tools/2.8.10-ubuntu-amd64/downloaded-tools.txt
>
> But obviously 'ubuntu' vs 'bionic'
>
> We should check that this is also fixed by whatever patch we do here.
>
>
> On Mon, May 3, 2021 at 9:20 AM Ian Booth <email address hidden>
> wrote:
>
>> ** Changed in: juju
>> Status: In Progress => Fix Committed
>>
>> --
>> You received this bug notification because you are subscribed to juju.
>> Matching subscriptions: juju bugs
>> https://bugs.launchpad.net/bugs/1926826
>>
>> Title:
>> 2.9 controller and 2.8 model - cannot provision lxd containers
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/juju/+bug/1926826/+subscriptions
>>
>

[Ian Booth (wallyworld)]

Yes, the scenario above was tested and verified to work as part of the fix.

[Tiago Pasqualini da Silva (tiago.pasqualini)]

Hi folks,

I tested this on 2.9.15 controller with a 2.8.11 model, on both LXD and Openstack and I'm still getting it to fail.

I'm seeing this error whenever I try to deploy anything on the 2.8.11 model:

2021-09-29 14:19:08 ERROR juju.worker.dependency engine.go:671 "unit-agent-deployer" manifold worker returned unexpected error: cannot read agent metadata in directory /var/lib/juju/tools/2.8.11-bionic-amd64: open /var/lib/juju/tools/2.8.11-bionic-amd64/downloaded-tools.txt: no such file or directory

Obviously there is no 2.8.11-bionic-amd64 folder, just the 2.8.11-ubuntu-amd64 one.

Is this a different issue than this bug or am I missing something?

[Ian Booth (wallyworld)]

It looks like you've found a corner case that we didn't know about previously. Container provisioning works but deploying charms in a certain way doesn't.

Steps to reproduce:

start with a 2.9.15 controller and a 2.8.11 model

juju add-machine --series=xenial
(the agent binary will be "2.8.11-ubuntu-amd64" since the 2.9 controller sets that up)

juju deploy cs:ubuntu --series=xenial --to 1
(the machine agent is still 2.8.11 so it looks for 2.8.11-xenial-amd64)

The quickest way to unblock is to upgrade the model to 2.9 until a fix can be figured out.

A new bug has been raised as it's a subtly different issue.
https://bugs.launchpad.net/juju/+bug/1945536