Juju ignores private image mirror settings and tries to fetch the images from Internet

[Anna Savchenko]

I have this setup
```bash
# lxc remote ls
+----------------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| NAME | URL | PROTOCOL | AUTH TYPE | PUBLIC | STATIC | GLOBAL |
+----------------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| images | https://lxdkvm.mirror.maas | simplestreams | none | YES | NO | NO |
+----------------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| local (current) | unix:// | lxd | file access | NO | YES | NO |
+----------------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu | https://cloud-images.ubuntu.com/releases | simplestreams | none | YES | YES | NO |
+----------------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu-daily | https://cloud-images.ubuntu.com/daily | simplestreams | none | YES | YES | NO |
+----------------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu-minimal | https://cloud-images.ubuntu.com/minimal/releases/ | simplestreams | none | YES | YES | NO |
+----------------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
| ubuntu-minimal-daily | https://cloud-images.ubuntu.com/minimal/daily/ | simplestreams | none | YES | YES | NO |
+----------------------+---------------------------------------------------+---------------+-------------+--------+--------+--------+
```

I can list the images on my mirror
```bash
# lxc image list images: | tail
| | ffc039ca79d7 | yes | ubuntu 20.04 LTS armhf (release) (20221213) | armv7l | VIRTUAL-MACHINE | 896.31MiB | Dec 13, 2022 at 12:00am (UTC) |
+--------------------+--------------+--------+-------------------------------------------------+--------------+-----------------+------------+-------------------------------+
| | ffcc26d46aa8 | yes | ubuntu 14.04 LTS ppc64el (release) (20160824.1) | ppc64le | VIRTUAL-MACHINE | 251.44MiB | Aug 24, 2016 at 12:00am (UTC) |
+--------------------+--------------+--------+-------------------------------------------------+--------------+-----------------+------------+-------------------------------+
| | ffd811035fde | yes | ubuntu 16.04 LTS ppc64el (release) (20180306) | ppc64le | VIRTUAL-MACHINE | 284.31MiB | Mar 6, 2018 at 12:00am (UTC) |
+--------------------+--------------+--------+-------------------------------------------------+--------------+-----------------+------------+-------------------------------+
| | fff8772c1526 | yes | ubuntu 16.04 LTS amd64 (release) (20181204) | x86_64 | VIRTUAL-MACHINE | 290.88MiB | Dec 4, 2018 at 12:00am (UTC) |
+--------------------+--------------+--------+-------------------------------------------------+--------------+-----------------+------------+-------------------------------+
| | fffaa6f204d8 | yes | ubuntu 16.04 LTS amd64 (release) (20170610) | x86_64 | CONTAINER | 121.25MiB | Jun 10, 2017 at 12:00am (UTC) |
+--------------------+--------------+--------+-------------------------------------------------+--------------+-----------------+------------+-------------------------------+
```

Now the problem is that on Juju it seems trying to pull the images from `ubuntu:` instead than from `images:`

```bash
# juju status
[..]
Machine State Address Inst id Base AZ Message
0 down pending ubuntu@22.04 cannot get available image metadata: failed getting published images metadata from default ubuntu cloud images: canno...
```

```bash
controller # less /var/log/juju/models/admin-maas-db-1faea0.log
[..]
2024-02-08 23:23:33 INFO juju.worker.provisioner provisioner_task.go:504 found machine pending provisioning id:0, details:0
2024-02-08 23:23:36 ERROR juju.worker.provisioner provisioner_task.go:1304 0 cannot get available image metadata: failed getting published images metadata from default ubuntu cloud images: cannot read index data, attempt count exceeded: cannot access URL "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": Get "http://cloud-images.ubuntu.com/releases/streams/v1/index2.sjson": dial tcp [2620:2d:4000:1::17]:80: connect: network is unreachable
```

This is my juju config
```bash
juju model-config
Attribute From Value
agent-metadata-url controller https://juju.mirror.maas
agent-stream default released
agent-version model 3.1.7
apt-ftp-proxy default ""
apt-http-proxy default ""
apt-https-proxy default ""
apt-mirror controller http://ubuntu-archive.mirror.maas:8000/ubuntu
apt-no-proxy default ""
automatically-retry-hooks default true
backup-dir default ""
charmhub-url default https://api.charmhub.io
cloudinit-userdata controller <value set, see juju model-config cloudinit-userdata>
container-image-metadata-url controller https://lxdkvm.mirror.maas/
container-image-stream controller released
container-inherit-properties default ""
container-networking-method model local
controller-charm-path controller ./juju-controller__3-1stable__rev79.charm
default-base default ""
default-series default ""
default-space default ""
development default false
disable-network-management default false
disable-telemetry default false
egress-subnets default ""
enable-os-refresh-update default true
enable-os-upgrade default true
fan-config default ""
firewall-mode default instance
ftp-proxy default ""
http-proxy default ""
https-proxy default ""
ignore-machine-addresses default false
image-metadata-url controller https://lxdkvm.mirror.maas
image-stream default released
juju-ftp-proxy default ""
juju-http-proxy default ""
juju-https-proxy default ""
juju-no-proxy default 127.0.0.1,localhost,::1
logforward-enabled default false
logging-config model <root>=INFO
logging-output default ""
lxd-snap-channel default 5.0/stable
max-action-results-age default 336h
max-action-results-size default 5G
max-status-history-age default 336h
max-status-history-size default 5G
mode default ""
net-bond-reconfigure-delay default 17
no-proxy default 127.0.0.1,localhost,::1
num-container-provision-workers default 4
num-provision-workers default 16
project default default
provisioner-harvest-mode default destroyed
proxy-ssh default false
resource-tags model {}
secret-backend default auto
snap-http-proxy default ""
snap-https-proxy default ""
snap-store-assertions controller |-
[..]
snap-store-proxy controller <redacted>
snap-store-proxy-url controller https://snap.mirror.maas
ssl-hostname-verification default true
storage-default-filesystem-source model lxd
test-mode default false
transmit-vendor-metrics default true
update-status-hook-interval default 5m
```

Reproduction steps: create an air-gapped environment and point Juju to the local mirror. Then, attempt to deploy Juju in HA or deploy any workload with Juju.

[Anna Savchenko (annsavchenko)]

I have seen the same issue with Juju 3.3.1 and Juju controller charm revisions 14 and 79.

[Gustavo Sanchez (gustavosr98)]

Some more context into the issue
- It is present on both Juju 3.1.7 as well as Juju 3.3.1
- It seem to be independent of the cloud as it shows using LXD as well as MAAS as substrate
- It is on an air gapped environment using a private image mirror

Bootstrapping of the first controller node works using the proper private image mirror
However, adding any extra machine, being a workload like Postgresql charm, a plain machine with juju add-machine or adding extra controllers will fail with the same error trying to use Internet image repository rather than private mirror set through Juju model default configurations

[Gustavo Sanchez (gustavosr98)]

Subscribing to field-critical since it is blocking two different Field Engineering customer project deployments and we do not have any workaround for it

[Joseph Phillips (manadart)]

Introduced by:
https://github.com/juju/juju/pull/16451

It does use the configured source, but we always attempt to accumulate metadata from the default official sources, which is now returning an error.

[John A Meinel (jameinel)]

Reading through the PR that Joe referenced, it seems that we are trying to check all possible sources, and ignore particular errors. It seems plausible that the way the official image sources are being blocked is causing a different error that isn't in our 'allowed set' of errors that we ignore.

The particular patch lists:
  IsFatalError: func(err error) bool {
   return errors.Is(err, errors.NotFound) || errors.Is(err, errors.Unauthorized)

But the error we are seeing is "network is unreachable" which is probably a different error type.

[Harry Pidcock (hpidcock)]

https://github.com/juju/juju/pull/16932

[Bayani Carbone (bcarbone)]

fyi, you propably need to add these config options when working in an airgapped environment:

--config image-metadata-defaults-disabled=true and --config container-image-metadata-defaults-disabled=true