fail to pull metallb-speaker-image and metallb-container-imag

[Andy Wu]

metallb-controller deployment and metallb-speaker daemonset stuck in pulling charm images

- microk8s version: 1.26
- juju: 2.9.38
- metallb-controller: 1.26/stable, rev 37
- metallb-speaker: 1.26/stable, rev 32

juju status
metallb-controller/8 error idle 10.1.35.154 7472/TCP unknown container reason "ImagePullBackOff": Back-off pulling image "registry.jujucharms.com/charm/87uf862ea0eoos74bw...
metallb-speaker/6* error idle 10.140.11.25 7472/TCP unknown container reason "ImagePullBackOff": Back-off pulling image "registry.jujucharms.com/charm/3u4dploqfqe5c89joq...
metallb-speaker/7 error idle 10.140.12.148 7472/TCP unknown container reason "ImagePullBackOff": Back-off pulling image "registry.jujucharms.com/charm/3u4dploqfqe5c89joq...
metallb-speaker/8 error idle 10.140.13.118 7472/TCP unknown container reason "ImagePullBackOff": Back-off pulling image "registry.jujucharms.com/charm/3u4dploqfqe5c89joq...

+ kubectl get pods -n metallb-system
NAME READY STATUS RESTARTS AGE
modeloperator-5cc6f985dd-nd5dc 1/1 Running 0 83m
metallb-controller-operator-0 1/1 Running 0 83m
metallb-speaker-operator-0 1/1 Running 0 83m
metallb-controller-794c99b8-hbkvp 1/1 Running 0 22m
metallb-speaker-dv74x 0/1 ImagePullBackOff 0 82m
metallb-speaker-6dszk 0/1 ImagePullBackOff 0 82m
metallb-controller-699b75bfd8-7xhm9 0/1 ImagePullBackOff 0 19m
metallb-speaker-p69x5 0/1 ErrImagePull 0 82m

kubectl describe pods -n metallb-system metallb-speaker-6dszk

Failed to pull image "registry.jujucharms.com/charm/3u4dploqfqe5c89joq8oaubnbio8a634xj64t/metallb-speaker-image@sha256:9018c0c2d85c8fdd47a58bcff0d07740ea5de8849652bf3531a7ab3a2904bb0d": rpc error: code = Unknown desc = failed to pull and unpack image "registry.jujucharms.com/charm/3u4dploqfqe5c89joq8oaubnbio8a634xj64t/metallb-speaker-image@sha256:9018c0c2d85c8fdd47a58bcff0d07740ea5de8849652bf3531a7ab3a2904bb0d": failed to resolve reference "registry.jujucharms.com/charm/3u4dploqfqe5c89joq8oaubnbio8a634xj64t/metallb-speaker-image@sha256:9018c0c2d85c8fdd47a58bcff0d07740ea5de8849652bf3531a7ab3a2904bb0d": failed to authorize: failed to fetch oauth token: Post "https://api.jujucharms.com/charmstore/docker-registry/token": Forbidden

The password in set in secret but appeared not used by juju during image pull , this can be proved by
  - Forbidden error in pod's log
  - pulling the same image with password manually in microk8s nodes is successul

+ kubectl get secret -n metallb-system
NAME TYPE DATA AGE
model-exec kubernetes.io/service-account-token 3 88m
metallb-controller-controller-secret kubernetes.io/dockerconfigjson 1 88m
metallb-speaker-speaker-secret kubernetes.io/dockerconfigjson 1 88m
memberlist Opaque 1 88m

+ kubectl get -o=yaml secret metallb-speaker-speaker-secret
apiVersion: v1
data:
  .dockerconfigjson: eyJhdXRocyI6eyJyZWdpc3RyeS5qdWp1Y2hhcm1zLmNvbSI6eyJVc2VybmFtZSI6ImRvY2tlci1yZWdpc3RyeSIsIlBhc3N3b3JkIjoiTURBeE9HeHZZMkYwYVc5dUlHTm9ZWEp0YzNSdmNtVUtNREF6TUdsa1pXNTBhV1pwWlhJZ09UQmxPRFV3WVdVMk16bGxOVFptTUdVMk9UYzVNbVl5T1RRelpXWTRaRFlLTURBMU9XTnBaQ0JwY3kxa2IyTnJaWEl0Y21Wd2J5QmphR0Z5YlM4emRUUmtjR3h2Y1daeFpUVmpPRGxxYjNFNGIyRjFZbTVpYVc4NFlUWXpOSGhxTmpSMEwyMWxkR0ZzYkdJdGMzQmxZV3RsY2kxcGJXRm5aUW93TURFelkybGtJR0ZzYkc5M0lIQjFiR3dLTURBeVpuTnBaMjVoZEhWeVpTRHkyb3hBSEc0djh4Y1dsQ2lVaWFqZWtuNGpMSEROMnVNR21FeU1GOTVZY0FvIiwiRW1haWwiOiIifX19

echo "^^base64-strings^^>" | base64 -d | jq -r .
{
  "auths": {
    "registry.jujucharms.com": {
      "Username": "docker-registry",
      "Password": "MDAxOGxvY2F0aW9uIGNoYXJtc3RvcmUKMDAzMGlkZW50aWZpZXIgOTBlODUwYWU2MzllNTZmMGU2OTc5MmYyOTQzZWY4ZDYKMDA1OWNpZCBpcy1kb2NrZXItcmVwbyBjaGFybS8zdTRkcGxvcWZxZTVjODlqb3E4b2F1Ym5iaW84YTYzNHhqNjR0L21ldGFsbGItc3BlYWtlci1pbWFnZQowMDEzY2lkIGFsbG93IHB1bGwKMDAyZnNpZ25hdHVyZSDy2oxAHG4v8xcWlCiUiajekn4jLHDN2uMGmEyMF95YcAo",
      "Email": ""
    }
  }
}

# this works
microk8s.ctr image pull -u docker-registry:<passwd> registry.jujucharms.com/charm/3u4dploqfqe5c89joq8oaubnbio8a634xj64t/metallb-speaker-image@sha256:9018c0c2d85c8fdd47a58bcff0d07740ea5de8849652bf3531a7ab3a2904bb0d

also pod spec has imagePullPolicy set to Always which prevent it from using the local cached images

Subscribe Field high since it impact COS deployment for PS6

[Andy Wu (qch2012)]

turns out to be the proxy issues

change containerd_env to following (epscially add api.jujucharms.com to NO_PROXY) solve the issue

HTTPS_PROXY=http://squid.internal:3128
NO_PROXY=10.1.0.0/16,10.152.183.0/24,api.jujucharms.com
ulimit -n 65536 || true
ulimit -l 16384 || true

make issue as invalid