Juju caas-image-repo not working with private repo that requires authentication to reach API

[Gustavo Sanchez]

# =================================== Context
Juju CLI 2.9.45
Microk8s 1.28
Container registry Jfrog

# =================================== Description
I am trying to bootstrap a Juju Controller on k8s using a private container registry

My assumption is:
It seems Juju CLI is trying to curl the container registry before using it to valide the URL exists, however, it doesn't seem to be using the provided credentials during this "ping" thus resulting in error

# =================================== Reproduce
# ================
$ docker login 10.46.90.139/artifactory/api/docker/jujusolutions/v2
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /root/snap/docker/2904/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

# ================ Using same credentials and URL that were used from Docker CLI to push images
$ juju bootstrap microk8s mk8s --debug --config caas-image-repo='{"repository":"10.46.90.139/jujusolutions", "serveraddress":"10.46.90.139/artifactory/api/docker/jujusolutions/v2", "username":"admin", "password": "cmVmdGtuOjAxOjE3MjkxODkwNjU6dHRBWTJFdjNwN01hMUZVQnBJZk04VFVNb3ZM" }'

13:20:10 INFO juju.cmd supercommand.go:56 running juju [2.9.45 afb8ee760af71d0bca8c3e4e0dc28af2dabc9b1d gc go1.20.8]
13:20:10 DEBUG juju.cmd supercommand.go:57 args: []string{"/snap/juju/24550/bin/juju", "bootstrap", "microk8s", "mk8s", "--debug", "--config", "caas-image-repo={\"repository\":\"10.46.90.139/jujusolutions\", \"serveraddress\":\"10.46.90.139/artifactory/api/docker/jujusolutions/v2\", \"username\":\"admin\", \"password\": \"cmVmdGtuOjAxOjE3MjkxODkwNjU6dHRBWTJFdjNwN01hMUZVQnBJZk04VFVNb3ZM\" }"}
13:20:10 DEBUG juju.kubernetes.provider provider.go:140 opening model "add-cloud".
13:20:10 INFO juju.kubernetes.klog klog.go:59 Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
13:20:10 INFO juju.kubernetes.klog klog.go:59 Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
13:20:10 INFO juju.kubernetes.klog klog.go:59 Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
13:20:10 INFO juju.kubernetes.klog klog.go:59 Use tokens from the TokenRequest API or manually created secret-based tokens instead of auto-generated secret-based tokens.
13:20:10 DEBUG juju.kubernetes.provider metadata.go:169 use the default Storage class "microk8s-hostpath" for operator storage class because it also matches Juju preferred config {hostpath microk8s.io/hostpath map[] false WaitForFirstConsumer}
13:20:10 DEBUG juju.kubernetes.provider metadata.go:190 use the default Storage class "microk8s-hostpath" for nominated storage class
13:20:10 INFO cmd cloudcredential.go:47 updating credential store
13:20:10 DEBUG juju.cmd.juju.commands bootstrap.go:1313 authenticating with region "" and credential "microk8s" ()
13:20:10 DEBUG juju.cmd.juju.commands bootstrap.go:1461 provider attrs: map[operator-storage: workload-storage:]
13:20:11 DEBUG juju.docker.registry.internal base_client.go:193 baseClient ping "https://10.46.90.139/artifactory/api/docker/jujusolutions/v2"
ERROR constructing controller config: Get "https://10.46.90.139/artifactory/api/docker/jujusolutions/v2": refreshing OAuth token: failed to refresh bearer token
13:20:11 DEBUG cmd supercommand.go:537 error stack:
failed to refresh bearer token
github.com/juju/juju/docker/registry/internal.(*tokenTransport).retry:214: refreshing OAuth token
github.com/juju/juju/docker/registry/internal.errorTransport.RoundTrip:244:
github.com/juju/juju/docker/registry/internal.unwrapNetError:283: Get "https://10.46.90.139/artifactory/api/docker/jujusolutions/v2"
github.com/juju/juju/docker/registry/internal.baseClient.Ping:198:
github.com/juju/juju/controller.validateCAASImageRepo:961:
github.com/juju/juju/controller.Validate:1170:
github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).bootstrapConfigs:1506: constructing controller config
github.com/juju/juju/cmd/juju/commands.(*bootstrapCommand).Run:685:

# =================================== Additional Contex
# ================ Public repo does not requires Auth to access API
curl https://index.docker.io/v2
<a href="/v2/">Moved Permanently</a>.

# ================ Private repo requiring Auth to access API
# It fails curling the API when not using token
curl -v https://10.46.90.139/artifactory/api/docker/jujusolutions/v2

* Trying 10.46.90.139:443...
* TCP_NODELAY set
* Connected to 10.46.90.139 (10.46.90.139) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=XX; ST=StateName; L=CityName; O=CompanyName; OU=CompanySectionName; CN=CommonNameOrHostname
* start date: Oct 13 23:51:25 2023 GMT
* expire date: Oct 10 23:51:25 2033 GMT
* subjectAltName: host "10.46.90.139" matched cert's IP address!
* issuer: C=XX; ST=StateName; L=CityName; O=CompanyName; OU=CompanySectionName; CN=CommonNameOrHostname
* SSL certificate verify ok.
> GET /artifactory/api/docker/jujusolutions/v2 HTTP/1.1
> Host: 10.46.90.139
> User-Agent: curl/7.68.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 401
< Date: Wed, 18 Oct 2023 22:49:58 GMT
< Content-Type: application/json;charset=ISO-8859-1
< Content-Length: 91
< Connection: keep-alive
< WWW-Authenticate: Basic realm="Artifactory Realm"
< X-Content-Type-Options: nosniff
<
{
  "errors" : [ {
    "status" : 401,
    "message" : "Authentication is required"
  } ]
* Connection #0 to host 10.46.90.139 left intact
}%

# It reponds 200 OK when using token
$ curl -H "Authorization: Bearer $token" https://10.46.90.139/artifactory/api/docker/jujusolutions/v2 -v
* Trying 10.46.90.139:443...
* TCP_NODELAY set
* Connected to 10.46.90.139 (10.46.90.139) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=XX; ST=StateName; L=CityName; O=CompanyName; OU=CompanySectionName; CN=CommonNameOrHostname
* start date: Oct 13 23:51:25 2023 GMT
* expire date: Oct 10 23:51:25 2033 GMT
* subjectAltName: host "10.46.90.139" matched cert's IP address!
* issuer: C=XX; ST=StateName; L=CityName; O=CompanyName; OU=CompanySectionName; CN=CommonNameOrHostname
* SSL certificate verify ok.
> GET /artifactory/api/docker/jujusolutions/v2 HTTP/1.1
> Host: 10.46.90.139
> User-Agent: curl/7.68.0
> Accept: */*
> Authorization: Bearer cmVmdGtuOjAxOjE3MjkxODkwNjU6dHRBWTJFdjNwN01hMUZVQnBJZk04VFVNb3ZM
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< Date: Wed, 18 Oct 2023 18:23:44 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: keep-alive
< X-JFrog-Version: Artifactory/7.68.14 76814900
< X-Artifactory-Id: 0b7e7d6412f08a06:1b0f85f9:18b43eb3113:-8000
< X-Artifactory-Node-Id: jfrog
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: always
<
* Connection #0 to host 10.46.90.139 left intact
{}%

[Gustavo Sanchez (gustavosr98)]

Subscribing to "field-critical" since it is blocking a current customer deployment

[Harry Pidcock (hpidcock)]

I believe this is due to WWW-Authenticate challenge type of Basic not being handled, we are expecting a Bearer type with a realm etc to then use Basic auth on that (like most other OCI v2 registries). This is indeed a OCI v2/Docker v2 compliant response, we are just missing this specific handling.

[Harry Pidcock (hpidcock)]

https://github.com/juju/juju/pull/16465

[John A Meinel (jameinel)]

Merged into 3.1 w/ https://github.com/juju/juju/pull/16619