juju upgrade-model on k8s tries to reach streams.canonical.com

[Anna Savchenko]

When I try to upgrade the model on k8s cloud (from 3.1.6 to 3.3.3), juju tries to reach streams.canonical.com instead of docker.

$ juju upgrade-model --debug --agent-version 3.3.3 --dry-run
21:53:15 INFO juju.cmd supercommand.go:56 running juju [3.3.3 3e20d5947e407dcb4ce9c6fc29ba04b24978468e gc go1.20.14]
21:53:15 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/26652/bin/juju", "upgrade-model", "--debug", "--agent-version", "3.3.3", "--dry-run"}
21:53:15 INFO juju.juju api.go:86 connecting to API addresses: [10.131.24.72:17070]
21:53:15 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://10.131.24.72:17070/api"
21:53:15 INFO juju.api apiclient.go:707 connection established to "wss://10.131.24.72:17070/api"
21:53:15 INFO juju.juju api.go:86 connecting to API addresses: [10.131.24.72:17070]
21:53:15 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://10.131.24.72:17070/api"
21:53:15 INFO juju.api apiclient.go:707 connection established to "wss://10.131.24.72:17070/api"
21:53:15 INFO juju.juju api.go:86 connecting to API addresses: [10.131.24.72:17070]
21:53:15 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://10.131.24.72:17070/model/1d2137f6-7b39-4976-8cd9-133544844323/api"
21:53:15 INFO juju.api apiclient.go:707 connection established to "wss://10.131.24.72:17070/model/1d2137f6-7b39-4976-8cd9-133544844323/api"
21:53:51 DEBUG juju.api monitor.go:35 RPC connection died
21:53:51 DEBUG juju.api monitor.go:35 RPC connection died
21:53:51 DEBUG juju.api monitor.go:35 RPC connection died
21:53:51 DEBUG juju.cmd.juju.commands upgrademodel.go:356 upgradeModel failed cannot read index data, attempt count exceeded: cannot access URL "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": Get "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": dial tcp: lookup streams.canonical.com on 10.152.183.10:53: read udp 10.1.151.120:46388->10.152.183.10:53: i/o timeout
ERROR cannot read index data, attempt count exceeded: cannot access URL "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": Get "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": dial tcp: lookup streams.canonical.com on 10.152.183.10:53: read udp 10.1.151.120:46388->10.152.183.10:53: i/o timeout
21:53:51 DEBUG cmd supercommand.go:549 error stack:
cannot read index data, attempt count exceeded: cannot access URL "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": Get "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": dial tcp: lookup streams.canonical.com on 10.152.183.10:53: read udp 10.1.151.120:46388->10.152.183.10:53: i/o timeout
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api/client/modelupgrader.(*Client).UpgradeModel:66:

It cannot reach streams.canonical.com because the deployment is air-gapped, but this should not be the issue because we don't need streams.canonical.com.

juju controller version: 3.3.3
controller model version: 3.3.3
target model version: 3.1.6

juju client: 3.3.3

cloud: microk8s

To reproduce: migrate a 3.1.6 model to 3.3.3 controller and try juju upgrade-model

The configs of controller and target (cos) models: https://pastebin.canonical.com/p/vdKkCvc6vN/

[Ian Booth (wallyworld)]

This is by design. The metadata recorded in streams is the definitive source of what agent versions are available. So Juju checks streams and then goes to look for a corresponding oci image. This ensures that there's no accidental divergence between agents available on k8s and machine models.

It's not strictly correct to say "instead of docker". Once juju verifies / finds the agent version on streams, it then goes to the oci image on the oci repo.

[Ian Booth (wallyworld)]

Marking as Invalid - in an air gaped environment, you can do this

$ juju model-config image-metadata-defaults-disabled=True
$ juju model-config container-image-metadata-defaults-disabled=True

on juju 3.3.3 or later. But you'll still need to have streams metadata available for juju to check. There's doc for how to do this but I haven't got a link handy right at the moment.

[Anna Savchenko (annsavchenko)]

Hi Ian, thanks for your feedback.

I have indeed set the above two parameters to True.
I also followed your recommendation and configured the streams the same way I did for the maas-controller in the same environment. I've also added the TLS certs to the model-operator containers (both controller and target models) and confirmed that I can reach the mirror.

But when I try to upgrade the model, I get the same error:

$ juju upgrade-model --debug --agent-version 3.3.3 --dry-run
12:07:39 INFO juju.cmd supercommand.go:56 running juju [3.3.3 3e20d5947e407dcb4ce9c6fc29ba04b24978468e gc go1.20.14]
12:07:39 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/26652/bin/juju", "upgrade-model", "--debug", "--agent-version", "3.3.3", "--dry-run"}
12:07:39 INFO juju.juju api.go:86 connecting to API addresses: [10.131.24.72:17070]
12:07:39 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://10.131.24.72:17070/api"
12:07:39 INFO juju.api apiclient.go:707 connection established to "wss://10.131.24.72:17070/api"
12:07:39 INFO juju.juju api.go:86 connecting to API addresses: [10.131.24.72:17070]
12:07:39 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://10.131.24.72:17070/api"
12:07:39 INFO juju.api apiclient.go:707 connection established to "wss://10.131.24.72:17070/api"
12:07:39 INFO juju.juju api.go:86 connecting to API addresses: [10.131.24.72:17070]
12:07:39 DEBUG juju.api apiclient.go:1172 successfully dialed "wss://10.131.24.72:17070/model/1d2137f6-7b39-4976-8cd9-133544844323/api"
12:07:39 INFO juju.api apiclient.go:707 connection established to "wss://10.131.24.72:17070/model/1d2137f6-7b39-4976-8cd9-133544844323/api"
12:08:16 DEBUG juju.api monitor.go:35 RPC connection died
12:08:16 DEBUG juju.api monitor.go:35 RPC connection died
12:08:16 DEBUG juju.api monitor.go:35 RPC connection died
12:08:16 DEBUG juju.cmd.juju.commands upgrademodel.go:356 upgradeModel failed cannot read index data, attempt count exceeded: cannot access URL "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": Get "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": dial tcp: lookup streams.canonical.com on 10.152.183.10:53: read udp 10.1.151.120:36222->10.152.183.10:53: i/o timeout
ERROR cannot read index data, attempt count exceeded: cannot access URL "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": Get "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": dial tcp: lookup streams.canonical.com on 10.152.183.10:53: read udp 10.1.151.120:36222->10.152.183.10:53: i/o timeout
12:08:16 DEBUG cmd supercommand.go:549 error stack:
cannot read index data, attempt count exceeded: cannot access URL "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": Get "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": dial tcp: lookup streams.canonical.com on 10.152.183.10:53: read udp 10.1.151.120:36222->10.152.183.10:53: i/o timeout
github.com/juju/juju/rpc.(*Conn).Call:178:
github.com/juju/juju/api/client/modelupgrader.(*Client).UpgradeModel:66:

[Anna Savchenko (annsavchenko)]

Adding controller config

[Anna Savchenko (annsavchenko)]

Adding target model config

[Anna Savchenko (annsavchenko)]

subscribed ~field-critical

[Ian Booth (wallyworld)]

Quick comment - those new settings to disable checking streams.canonical.com only work on Juju 3.3.3. I think the controller you are upgrading is 3.1.6 right? So this won't support the new settings.

[Ian Booth (wallyworld)]

Your only viable option might be to stand up a new 3.3.3 controller and migrate the model to it. This will ensure the "ignore default streams" settings are used. But I suspect (without testing) you'll still need to ensure that the controller has access to streams metadata mirror in the air gapped environment.

[Ian Booth (wallyworld)]

I just wanted to check I understand the scenario here.

The controller is already 3.3.3?
And you have successfully migrated a 3.1.6 model to that controller?
And now you just want to run "juju upgrade-model" to bring that migrated 3.1.6 model up to 3.3.3 to match the controller?

BTW, you don't need agent-version=3.3.3 in the upgrade-model command; it will automatically pick the controller version.

The error has this

access URL "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": Get "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson": dial tcp: lookup streams.canonical.com on 10.152.183.10:53: read udp 10.1.151.120:36222->10.152.183.10:53: i/o timeout

Here it's looking for agent binaries which matches my comment #1. Sadly it looks like when the new config for ignoring default simplestreams sources was added, it was only done for image metadata, not agent binaries. So this won't work air gapped until another patch is landed and released.

[Anna Savchenko (annsavchenko)]

Hey Ian, yes, indeed, the controller is on 3.3.3 and I migrated the model successfully. What doesn't work is upgrade-model.

I tried to point juju to the private mirror streams but I couldn't find a way to add the TLS cert to juju controller.

[Anna Savchenko (annsavchenko)]

Also, even though from the debug output I can see that it tries to reach https://streams.canonical.com/juju/tools/streams/v1/index2.sjson, I see from the api-server container log that it tries to reach the private mirror:

2024-03-18T14:01:39.497Z [jujud] 2024-03-18 14:01:39 DEBUG juju.environs.simplestreams simplestreams.go:421 falling back to search for unsigned metadata in datasource "agent-metadata-url"
2024-03-18T14:01:39.498Z [jujud] 2024-03-18 14:01:39 DEBUG juju.environs.simplestreams simplestreams.go:451 looking for data index using path streams/v1/index2.json
2024-03-18T14:01:39.498Z [jujud] 65d11e3c-0b73-446e-80dd-ba1e7a8e1c40: controller-0 2024-03-18 14:01:39 WARNING juju.environs.simplestreams datasource.go:212 Got error requesting "https://repo1.nfvi.redacted.prv/simple-streams-juju/streams/v1/index2.sjson": Get "https://repo1.nfvi.redacted.prv/simple-streams-juju/streams/v1/index2.sjson": tls: failed to verify certificate: x509: certificate signed by unknown authority
2024-03-18T14:01:39.498Z [jujud] 65d11e3c-0b73-446e-80dd-ba1e7a8e1c40: controller-0 2024-03-18 14:01:39 DEBUG juju.environs.simplestreams simplestreams.go:463 looking for data index using URL https://repo1.nfvi.redacted.prv/simple-streams-juju/streams/v1/index2.sjson

So will it help if I manage to add the TLS certificate to juju controller?
I already added it on model operators but looks like it's not enough.