Containerd fails to pull image from private registry due to "tls: bad certificate"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Containerd Subordinate Charm |
Fix Released
|
Undecided
|
Kevin W Monroe |
Bug Description
I deployed charmed kubernetes from edge, with a local build of containerd including https:/
Pods are failing to come up because containerd cannot pull images from the registry:
Failed create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "172.31.
The docker-registry logs give a clearer message:
http: TLS handshake error from 172.31.43.76:52726: tls: client didn't provide a certificate
The registry is configured with mutual authentication enabled, meaning that containerd needs to provide a client certificate signed by the easyrsa charm. However, our containerd does not currently do this, and it's not even possible to do it with the version of containerd we ship (1.2.6).
Looks like we will have to update to containerd 1.3.0 and fix up the config.
Changed in charm-containerd: | |
status: | New → In Progress |
assignee: | nobody → George Kraft (cynerva) |
Changed in charm-containerd: | |
status: | Confirmed → In Progress |
assignee: | George Kraft (cynerva) → Kevin W Monroe (kwmonroe) |
Changed in charm-containerd: | |
milestone: | none → 1.19 |
Changed in charm-containerd: | |
status: | In Progress → Fix Committed |
tags: | removed: review-needed |
Changed in charm-containerd: | |
milestone: | 1.19 → 1.18+ck1 |
Changed in charm-containerd: | |
status: | Fix Committed → Fix Released |
-- Workaround 1: Disable docker-registry mutual authentication
juju config docker-registry tls-ca-path=""
Please note that this comes with security implications. Normally, the docker-registry charm will refuse to serve images to any client that does not present a client certificate signed by the easyrsa it is related to. This means that only members of the Charmed Kubernetes cluster can pull from it. If you disable mutual authentication, then docker-registry will serve images to any client who can reach its endpoint.
-- Workaround 2: Use docker instead of containerd
The docker charm is not affected by this bug. Using docker instead of containerd is briefly covered here: https:/ /ubuntu. com/kubernetes/ docs/container- runtime