Failed to pull image from charmhub on pod reschedule

[Arturo Enrique Seijas Fernández]

arturo-seijas
12:05 PM
Hi! We ran through an issue recently with a kubernetes charm. The pod got rescheduled into another node and Juju failed to pull the OCI image from charmcraft due to an authentication failure. I was wondering what's the mechanism behind and if this bug has already been reported/fixed. We are currently running version 2.9.38.

I opened a thread in Mm https://chat.charmhub.io/charmhub/pl/tmpxk9y84tns98oon1mzxis9zw

[Tom Haddon (mthaddon)]

Fwiw, the "events" related to the pod for this are:

Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Pulling 49m (x687 over 2d10h) kubelet Pulling image "registry.jujucharms.com/charm/kotcfrohea62xreenq1q75n1lyspke0qkurhk/postgresql-image@sha256:8a72e1152d4a01cd9f4697093d3815af4a48bb68731b2b292e3cb18163d8feff"
  Normal BackOff 4m46s (x15628 over 2d10h) kubelet Back-off pulling image "registry.jujucharms.com/charm/kotcfrohea62xreenq1q75n1lyspke0qkurhk/postgresql-image@sha256:8a72e1152d4a01cd9f4697093d3815af4a48bb68731b2b292e3cb18163d8feff"

[Tom Haddon (mthaddon)]

Some other information here is that the pod had been running for over 100 days before being rescheduled.

[Ian Booth (wallyworld)]

Juju uses the credential handed to it when the charm is deployed. The path to the image and a macaroon is generated by charmhub with the oci image is attached as a resource to the charm and is published. Thereafter, juju simply fetches the resource content and uses the info from that to set up the image pull secret in k8s. If now the registry is no longer accepting the credential, then a resource update would be needed to fix that. I'm not sure this is a juju issue per se.

[Tom Haddon (mthaddon)]

Looking at the kubelet logs, the first failure was https://paste.ubuntu.com/p/hrmHY7TSfN/. What's odd about this is that it says "requested access to the resource is denied\nunauthorized: authentication required". Any ideas why it might be trying to pull the image without the auth credentials?

[Tom Haddon (mthaddon)]

Also, the line after the paste above is https://paste.ubuntu.com/p/2fysnKZ3XX/ - should there be a mention of `postgresql-postgresql-secret` there?

[Tom Haddon (mthaddon)]

If I deploy postgresql-k8s locally and then edit the pod configuration (`kubectl edit pod -n pg-test postgresql-k8s-0`), I see this as part of the configuration:

```
  imagePullSecrets:
  - name: postgresql-k8s-postgresql-k8s-secret
```

But there's no such entry for the pod that's failing that this bug was filed about.

[Tom Haddon (mthaddon)]

And to confirm, the relevant secret (`postgresql-postgresql-secret` as the application name here is `postgresql`) does exist in the right namespace and have the correct authentication credentials.

[Arturo Enrique Seijas Fernández (arturo-seijas)]

Triggering the pod recreation didn't fix the issue. After deleting the pod, the kubelet is still unable to pull the image. The secret is still missing in the pod definition.

[Tom Haddon (mthaddon)]

We've worked around this by manually pulling the image onto the k8s worker where this pod was being scheduled but it would be good to know why the pod definition doesn't include a reference to the secret it needs to pull the image.

[Ian Booth (wallyworld)]

One thing to check is the actual content of the juju resource which corresponds to the image. The resource content is a json snippet which has the image path and optionally pull credential. If the image pull credential is missing, juju won't include anything in the pod definition.

[Tom Haddon (mthaddon)]

I've checked that resource and it includes a .dockerconfigjson key which if base64 decoded shows:

{"auths":{"registry.jujucharms.com":{"Username":"docker-registry","Password":"REDACTED","Email":""}}}

I've confirmed the password (redacted here) is correct.