failed to fetch agent binaries from "": Get "": unsupported protocol scheme ""

[Haw Loeung]

Hi,

Working on upgrading Juju 2.x controllers to the latest 2.9.42. Two of the units is stuck upgrading with this:

| 2023-03-20 21:25:11 INFO juju.worker.upgrader upgrader.go:191 desired agent binary version: 2.9.42
| 2023-03-20 21:25:11 INFO juju.worker.upgrader upgrader.go:220 upgrade requested from 2.8.9 to 2.9.42
| 2023-03-20 21:25:11 INFO juju.worker.upgrader upgrader.go:280 fetching agent binaries from ""
| 2023-03-20 21:25:11 ERROR juju.worker.upgrader upgrader.go:250 failed to fetch agent binaries from "": Get "": unsupported protocol scheme ""

Any ideas how to fix this?

[Ian Booth (wallyworld)]

This looks like database corruption. There appears to be a record in the units collection with an empty "tools" attribute. A dump of the units collection should show the affected unit records. These should match the units in error.

mongo would need to be updated to repair those records to match other units.

[Haw Loeung (hloeung)]

This is empty across the board / all units?

| https://pastebin.canonical.com/p/2mttMMwKs7/

I tried checking this on another model and seeing the same thing:

| https://pastebin.canonical.com/p/f7WWjxZQBr/

So:

| tools:
| sha256: ""
| size: 0
| url: ""
| version: 2.9.42-ubuntu-amd64
| --
| tools:
| sha256: ""
| size: 0
| url: ""
| version: 2.8.9-xenial-amd64

[Ian Booth (wallyworld)]

I'll have to dig some more. I checked a new model as well. Seems we've changed what we store on the unit record.

[Ian Booth (wallyworld)]

I bootstrapped a fresh 2.8.9 controller and added a model.
I first upgraded the controller to 2.9.42 without issue.

The log messages show that the agent URL does get filled out correctly

machine-0: 13:41:36 INFO juju.worker.upgrader desired agent binary version: 2.9.42
machine-0: 13:41:36 INFO juju.worker.upgrader upgrade requested from 2.8.9 to 2.9.42
machine-0: 13:41:40 INFO juju.worker.upgrader fetching agent binaries from "https://172.31.10.75:17070/model/16d86a06-931d-421f-89dc-d0096f922e38/tools/2.9.42-bionic-amd64"
machine-0: 13:41:40 INFO juju.apiserver 2.9.42-bionic-amd64 agent binaries not found locally, fetching
machine-0: 13:41:42 INFO juju.apiserver fetching 2.9.42-bionic-amd64 agent binaries from https://juju-dist-aws.s3.amazonaws.com/agents/agent/2.9.42/juju-2.9.42-linux-amd64.tgz

Then I upgraded the model without issue.

I am not sure what's wrong with your controller. Is there anything of interest in any of the controller or model logs?

[Ian Booth (wallyworld)]

This might be a compatibility issue between the 2.8 controller and a later juju cli client (>= 2.9).
If you are using a juju cli > 2.8, can you retry with the 2.8 cli and see if that makes a difference.

[Haw Loeung (hloeung)]

Only seems to be these two units, I'm not sure how to retry with a 2.8 cli when other units in this model are already upgraded and on 2.9.42.

Here's `juju status --format=yaml` output:

| https://pastebin.canonical.com/p/VSzd7B7yCY/

What's odd is that the other unit having this issue, container-log-archive-launchpad-frontend/, juju status reports 2.9.42 when it is not:

          container-log-archive-launchpad-frontend/6:
            workload-status:
              current: active
              message: Ready (source version/commit bdb3f4a)
              since: 22 Sep 2022 14:05:43Z
            juju-status:
              current: idle
              since: 20 Mar 2023 18:25:30Z
              version: 2.9.42
            leader: true
            public-address: 10.131.66.248

ubuntu@juju-4112d9-prod-launchpad-manual-servers-0:~$ tail -f /var/log/juju/unit-container-log-archive-launchpad-frontend-6.log -n 5
2023-03-21 04:30:13 ERROR juju.worker.upgrader upgrader.go:250 failed to fetch agent binaries from "": Get "": unsupported protocol scheme ""
2023-03-21 04:30:18 INFO juju.worker.upgrader upgrader.go:191 desired agent binary version: 2.9.42
2023-03-21 04:30:18 INFO juju.worker.upgrader upgrader.go:220 upgrade requested from 2.8.9 to 2.9.42
2023-03-21 04:30:18 INFO juju.worker.upgrader upgrader.go:280 fetching agent binaries from ""
2023-03-21 04:30:18 ERROR juju.worker.upgrader upgrader.go:250 failed to fetch agent binaries from "": Get "": unsupported protocol scheme ""
2023-03-21 04:30:23 INFO juju.worker.upgrader upgrader.go:191 desired agent binary version: 2.9.42
2023-03-21 04:30:23 INFO juju.worker.upgrader upgrader.go:220 upgrade requested from 2.8.9 to 2.9.42
2023-03-21 04:30:23 INFO juju.worker.upgrader upgrader.go:280 fetching agent binaries from ""
2023-03-21 04:30:23 ERROR juju.worker.upgrader upgrader.go:250 failed to fetch agent binaries from "": Get "": unsupported protocol scheme ""
^C

[Haw Loeung (hloeung)]

Also:

ubuntu@juju-4112d9-prod-launchpad-manual-servers-0:/var/lib/juju/tools$ ls -la
total 20
drwxr-xr-x 5 root root 4096 Mar 20 21:24 .
drwxr-xr-x 6 root root 4096 Feb 7 2022 ..
drwxr-xr-x 2 root root 4096 Feb 4 2022 2.8.9-xenial-amd64
drwxr-xr-x 2 root root 4096 Aug 16 2022 2.9.32-ubuntu-amd64
drwxr-xr-x 2 root root 4096 Mar 20 02:26 2.9.42-ubuntu-amd64
lrwxrwxrwx 1 root root 39 Mar 20 02:26 machine-0 -> /var/lib/juju/tools/2.9.42-ubuntu-amd64
lrwxrwxrwx 1 root root 39 Mar 20 21:24 unit-autocert-launchpad-frontend-1 -> /var/lib/juju/tools/2.9.42-ubuntu-amd64
lrwxrwxrwx 1 root root 39 Mar 20 21:24 unit-container-log-archive-launchpad-frontend-6 -> /var/lib/juju/tools/2.9.42-ubuntu-amd64
lrwxrwxrwx 1 root root 39 Mar 20 21:24 unit-landscape-client-0 -> /var/lib/juju/tools/2.9.42-ubuntu-amd64
lrwxrwxrwx 1 root root 39 Mar 20 21:24 unit-launchpad-frontend-0 -> /var/lib/juju/tools/2.9.42-ubuntu-amd64
lrwxrwxrwx 1 root root 39 Mar 20 21:24 unit-nrpe-launchpad-frontend-1 -> /var/lib/juju/tools/2.9.42-ubuntu-amd64
lrwxrwxrwx 1 root root 39 Mar 20 21:24 unit-telegraf-launchpad-frontend-1 -> /var/lib/juju/tools/2.9.42-ubuntu-amd64
lrwxrwxrwx 1 root root 39 Mar 20 21:24 unit-turku-agent-launchpad-frontend-1 -> /var/lib/juju/tools/2.9.42-ubuntu-amd64
lrwxrwxrwx 1 root root 39 Mar 20 21:24 unit-ubuntu-advantage-1 -> /var/lib/juju/tools/2.9.42-ubuntu-amd64

[Haw Loeung (hloeung)]

Maybe Juju can be more clever here if it sees "" or an unsupported protocol scheme to fallback to the provider default?

[Harry Pidcock (hpidcock)]

Just to confirm, the machine agent for the machine this unit is on has already upgraded?

All agents use a common Facade for getting the tools, except units. So I guess there is a problem there, as the facade seems to just get the tools stored against the machine, but not resolve the urls for those tool versions. This wouldn't normally be a problem because the machine should already have the tools downloaded..

What is the contents of /var/lib/juju/tools/2.9.42-ubuntu-amd64/downloaded-tools.txt?

[Haw Loeung (hloeung)]

Yes, the machine for these two VMs are already upgraded. Most of the units on these two VMs were already upgraded, it's just these exceptions.

On turku-agent-launchpad-bazaar-codehosting/1 (machine 21):

| ubuntu@juju-4112d9-prod-launchpad-manual-servers-21:~$ cat /var/lib/juju/tools/2.9.42-ubuntu-amd64/downloaded-tools.txt; echo
| {"version":"2.9.42-ubuntu-amd64","url":"https://10.131.1.8:17070/model/5e6635fe-ec57-45d8-87d6-c713064112d9/tools/2.9.42-ubuntu-amd64","sha256":"10617ce334ef9e00efbcfd06f072687bfac1071b54bc910b491c940ec598877f","size":42029399}
| ubuntu@juju-4112d9-prod-launchpad-manual-servers-21:~$

On container-log-archive-launchpad-frontend/6 (machine 0):

| ubuntu@juju-4112d9-prod-launchpad-manual-servers-0:~$ cat /var/lib/juju/tools/2.9.42-ubuntu-amd64/downloaded-tools.txt; echo
| {"version":"2.9.42-ubuntu-amd64","url":"https://10.131.1.8:17070/model/5e6635fe-ec57-45d8-87d6-c713064112d9/tools/2.9.42-ubuntu-amd64","sha256":"10617ce334ef9e00efbcfd06f072687bfac1071b54bc910b491c940ec598877f","size":42029399}
| ubuntu@juju-4112d9-prod-launchpad-manual-servers-0:~$

[Haw Loeung (hloeung)]

`juju status --format=yaml` output for machine 0 - https://pastebin.canonical.com/p/k5Bfnx6PQx/

The same for machine 21 - https://pastebin.canonical.com/p/fH9RWRFG7S/

[Haw Loeung (hloeung)]

These two machines/units are still stuck on 2.8.9 so we'll want a fix to this.

[Nishant Dash (dash3)]

Hello,

Is there a fix or workaround for this?

[Nishant Dash (dash3)]

$ cat /var/lib/juju/tools/2.9.42-ubuntu-amd64/downloaded-tools.txt
{"version":"2.9.42-ubuntu-amd64","url":"https://172.16.1.99:17070/model/6a346a06-9197-4f52-8ee7-7a62c58ea30b/tools/2.9.42-ubuntu-amd64","sha256":"10617ce334ef9e00efbcfd06f072687bfac1071b54bc910b491c940ec598877f"
,"size":42029399}

In my case it was a landscape-rmq unit.

I was able to get it to work with a
juju_stop_unit landscape-rabbitmq/4
juju_start_unit landscape-rabbitmq/4

(and also a restart of the rmq server)

and then,
/var/lib/juju/agents/unit-landscape-rabbitmq-4/agent.conf:upgradedToVersion: 2.9.42

and logs did not complain about,
```
failed to fetch agent binaries from "": Get "": unsupported protocol scheme ""
```