Docker Registry: Easy-rsa renders bad certificate

[Pedro Guimarães]

Running latest stable charms with k8s 1.16.
Following: https://ubuntu.com/kubernetes/docs/docker-registry

When I deploy Containerd + Docker Registry + Easyrsa, if I run "ctr pull" command, it will return "bad certificate", as below:

ctr: failed to resolve reference "image-registry.maas:5000/cdk/calico/node:v3.10.1": failed to do request: Head https://image-registry.maas:5000/v2/cdk/calico/node/manifests/v3.10.1: remote error: tls: bad certificate

Certificate in itself seems to be valid. However, if I try to wget the link above, wget will also throw openssl errors with bad certificate:

$ wget --no-check-certificate https://image-registry.maas:5000/v2/_catalog
--2020-03-10 20:51:16-- https://image-registry.maas:5000/v2/_catalog
Resolving image-registry.maas (image-registry.maas)... 172.27.16.197
Connecting to image-registry.maas (image-registry.maas)|172.27.16.197|:5000... connected.
OpenSSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
Unable to establish SSL connection.

That issue also happens if I manually define certificates on docker-registry options.

[Stamatis Katsaounis (skatsaounis)]

Just to add my experience here.

I set basic auth username and password. Then, inside the host of docker-registry, I am running docker login and by giving the auth credentials the .docker/config.json is produced. Furthermore, I can push and pull images to docker-registry.

On the other hand, if I try to do the same from kubernetes workers, meaning create a secret dockerconfig and use this secret inside a pod spec with an image from that registry, I am receiving bad certificate. I also receive bad certificates if I do the same actions as Pedro.

[Kevin W Monroe (kwmonroe)]

This is looking like bug 1853653. Couple notes:

When easyrsa is related to docker-registry, TLS is enabled. By default, this means mutual authentication is enabled and clients will need to send a cert to access the registry. To illustrate this, attempt a bare wget from a k8s worker and watch the docker-registry logs:

----------
worker:~$ wget https://172.31.20.67:5000/v2
Connecting to 172.31.20.67:5000... connected.
OpenSSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

registry:~$ docker logs -f registry
2020/04/03 20:38:19 http: TLS handshake error from 172.31.5.244:54498: tls: client didn't provide a certificate
----------

Now do it again by using a cert/key with the wget client (and user auth if configured):

----------
worker:~$ wget --user=admin --password=password --certificate=/root/cdk/server.crt --private-key=/root/cdk/server.key https://172.31.20.67:5000/v2
Connecting to 172.31.20.67:5000... connected.
HTTP request sent, awaiting response... 200 OK
Saving to: ‘v2’

registry:~$ docker logs -f registry
172.31.5.244 - - [03/Apr/2020:20:45:54 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Wget/1.19.4 (linux-gnu)"
----------

So in this case, the issue is not that easyrsa certs are bad nor in the wrong place. The client needs to *send* a cert before the registry will respond, and wget doesn't do that by default. I could not find a way to specify certs when using 'ctr', so I'm not sure that client will ever work with an https registry (at least not one with a self-signed cert and mutual auth).

I *was* able to pull images from a k8s worker using crictl [0], but only after working on a fix for the aforementioned bug 1853653:

----------
worker:~$ sudo ./crictl pull 172.31.20.67:5000/defaultbackend-amd64:1.5
Image is up to date for sha256:b5af743e598496e8ebd7a6eb3fea76a6464041581520d1c2315c95f993287303

registry:~$ docker logs -f registry
172.31.5.244 - - [03/Apr/2020:20:56:29 +0000] "HEAD /v2/defaultbackend-amd64/manifests/1.5 HTTP/1.1" 200 528 "" "containerd/1.3.3-0ubuntu1~18.04.1"
----------

To sum up, clients that interact with a TLS-enabled docker-registry will need to send cert data. I'm going to mark this as a dupe of 1853653. Watch that bug for a fix and current workarounds.

[0]: https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#installing-crictl