Juju Controller keeps picking wrong meta data url during app deployment when used the right url on the model it skips without debug logs.

[Mohaseenkhan]

Hi all,

I have hit a dead end. whenever i try to add machines or deploy applications using juju. The controller machines fails to get published images meta data. It keeps on pointing to link1. this url is not accessible anymore. The installation use to work well but after Jan 1 2024 things have changes it does not works they way it was working.

Note: I am not using MAAS here for now.

I am trying install juju applications of LXD cluster

link 1 http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson

link 2 https://cloud-images.ubuntu.com/releases/

LXD version Client version: 5.19 Server version: 5.19

JUJU version’ 3.3.0-genericlinux-amd64 (tried on the classic 3.1.7 it dod not work either)

OS Jammy 22.04

Solns tried:

Hosted my own meta data internal site with pache2 it skips and defaults to link1

Changed juju version—same error

Tried with Ubuntu 20.04—same error

Boostraped the controller with link2 —same error

error
4-01-05 23:44:14 WARNING juju.worker.proxyupdater proxyupdater.go:297 unable to update apt mirrors: exit status 1, output: “” 2024-01-05 23:44:48 INFO juju.worker.logforwarder logforwarder.go:105 config change - log forwarding not enabled 2024-01-05 23:44:48 INFO juju.worker.upgrader upgrader.go:166 desired agent binary version: 3.3.0 2024-01-05 23:49:29 INFO juju.state addmachine.go:506 new machine “1” has preferred addresses: private “”, public “” 2024-01-05 23:50:30 WARNING juju.environs.simplestreams datasource.go:212 Got error requesting “link1”: Get “link1”: dial tcp 185.125.190.40:80: i/o timeout 2024-01-05 23:51:01 WARNING juju.environs.simplestreams datasource.go:212 Got error requesting “link1”: Get “link1”: dial tcp 185.125.190.37:80: i/o timeout 2024-01-05 23:51:33 WARNING juju.environs.simplestreams datasource.go:212 Got error requesting “link1”: Get “link1”: dial tcp 185.125.190.40:80: i/o timeout

[Joseph Phillips (manadart)]

Based on your Discourse post, you have a connectivity issue to the resolved domain.

From https://discourse.charmhub.io/t/juju-controller-tries-for-non-existent-meta-data-url-to-pull-the-simplestream-json-file-failing-to-build-juju-machines-and-applications/12882/4:

 ERROR juju.worker.provisioner 29 cannot get available image metadata: failed getting published images metadata from default ubuntu cloud images: cannot read index data, attempt count exceeded: cannot access URL “http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson”: Get “http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson”: dial tcp 185.125.190.40:80: i/o timeout controller-0: 17:32:31 ERROR juju.worker.provisioner 30 cannot get available image metadata: failed getting published images metadata from default ubuntu cloud images: cannot read index data, attempt count exceeded: cannot access URL “http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson”: Get “http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson”: dial tcp 185.125.190.37:80: i/o timeout controller-0: 17:35:35

This shows the dial attempt failing and retrying, as opposed to getting a not found error, which would cause a fallback to reading index.sjson.

As far as I know, we've had no other reports of this issue.

[Mohaseenkhan (mohsin1990)]

we have web proxy in between so I added that web proxy on the controller. so that http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson timeout should not occur and it should fall back to http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson but no luck. With help of this proxy we connect to port 80 online.

I can curl http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson with 404 on controller but the switch does not happen from index2 to index. The strange thing is it use to work before without any issues or such configuration. juju.environs.simplestreams Debug do not give any info related to this timeout.

Moreover when I use image-metadata-url to the model the source is not added either and does not gives any useful logs it skips it (image-metadata-url=https://cloud-images.ubuntu.com/releases/) and again tries index2.sjson same thing happens when add an internal hosted image server.