Containerd fails to pull image from private registry due to "tls: bad certificate"

[George Kraft]

I deployed charmed kubernetes from edge, with a local build of containerd including https://github.com/charmed-kubernetes/charm-containerd/pull/28. I deployed and related the docker-registry charm from edge as documented in https://ubuntu.com/kubernetes/docs/docker-registry. I configured kubernetes-master with image-registry=<registry-ip>:5000 and addons-registry=<registry-ip>:5000.

Pods are failing to come up because containerd cannot pull images from the registry:

Failed create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "172.31.33.200:5000/pause-amd64:3.1": failed to pull image "172.31.33.200:5000/pause-amd64:3.1": failed to resolve image "172.31.33.200:5000/pause-amd64:3.1": no available registry endpoint: failed to do request: Head https://172.31.33.200:5000/v2/pause-amd64/manifests/3.1: remote error: tls: bad certificate

The docker-registry logs give a clearer message:

http: TLS handshake error from 172.31.43.76:52726: tls: client didn't provide a certificate

The registry is configured with mutual authentication enabled, meaning that containerd needs to provide a client certificate signed by the easyrsa charm. However, our containerd does not currently do this, and it's not even possible to do it with the version of containerd we ship (1.2.6).

Looks like we will have to update to containerd 1.3.0 and fix up the config.

[George Kraft (cynerva)]

-- Workaround 1: Disable docker-registry mutual authentication

juju config docker-registry tls-ca-path=""

Please note that this comes with security implications. Normally, the docker-registry charm will refuse to serve images to any client that does not present a client certificate signed by the easyrsa it is related to. This means that only members of the Charmed Kubernetes cluster can pull from it. If you disable mutual authentication, then docker-registry will serve images to any client who can reach its endpoint.

-- Workaround 2: Use docker instead of containerd

The docker charm is not affected by this bug. Using docker instead of containerd is briefly covered here: https://ubuntu.com/kubernetes/docs/container-runtime

[George Kraft (cynerva)]

I did a test deployment where I started with containerd 1.2.6 and upgraded to 1.3.0. It looks like the impact is pretty small. The only impact I observed was that I got kicked out of a `kubectl exec` session, but I was able to restart it immediately after. Pod containers were not restarted. Integration tests showed no issues following the upgrade.

The containerd apt package maintainer indicated that there should be no major issues with updating the package to 1.3.0, and we can request it by opening an issue here: https://bugs.launchpad.net/ubuntu/+source/containerd

I'm going to sync with the team before moving forward. Expect an update on this issue sometime next week.

[George Kraft (cynerva)]

I've requested an update to containerd 1.3.1 here: https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1854841

[George Kraft (cynerva)]

This is currently blocked awaiting the update to containerd 1.3.1.

[Kevin W Monroe (kwmonroe)]

Now that bionic has containerd 1.3.3, I was able to verify pulling images with crictl works against a docker-registry with TLS/mutual auth enabled. However, the config.toml written by the containerd charm needs tweaking to make this happen. Pseudo patch that worked for me:

--- config.toml.orig
+++ config.toml.works
- [plugins.cri.registry.auths."172.31.20.67:5000"]
- username = "admin"
- password = "password"

+ [plugins.cri.registry.configs."172.31.20.67:5000".auth]
+ username = "admin"
+ password = "password"

- [plugins.cri.registry.tls_configs."172.31.20.67:5000"]
- ca_file = "/root/cdk/ca.crt"
- cert_file = "/root/cdk/server.crt"
- key_file = "/root/cdk/server.key"

+ [plugins.cri.registry.configs."172.31.20.67:5000".tls]
+ ca_file = "/root/cdk/ca.crt"
+ cert_file = "/root/cdk/server.crt"
+ key_file = "/root/cdk/server.key"

I'll work this into a formal PR for the containerd charm, but wanted to share what's working for me. Here are a couple runs:

## config.toml.orig
# ./crictl pull 172.31.20.67:5000/defaultbackend-amd64:1.5
FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to pull and unpack image "172.31.20.67:5000/defaultbackend-amd64:1.5": failed to resolve reference "172.31.20.67:5000/defaultbackend-amd64:1.5": failed to do request: Head https://172.31.20.67:5000/v2/defaultbackend-amd64/manifests/1.5: remote error: tls: bad certificate

## config.toml.works
# ./crictl pull 172.31.20.67:5000/defaultbackend-amd64:1.5
Image is up to date for sha256:b5af743e598496e8ebd7a6eb3fea76a6464041581520d1c2315c95f993287303

I also confirmed that 'Workaround 1' from comment #1 works without any containerd changes.

[Kevin W Monroe (kwmonroe)]

PR ready for review:

https://github.com/charmed-kubernetes/charm-containerd/pull/34

[George Kraft (cynerva)]

Cherry-pick to stable: https://github.com/charmed-kubernetes/charm-containerd/commit/29ebcf97afb050f901c395b84f2d6db664871146