Comment 2 for bug 1866957

Revision history for this message
Kevin W Monroe (kwmonroe) wrote :

This is looking like bug 1853653. Couple notes:

When easyrsa is related to docker-registry, TLS is enabled. By default, this means mutual authentication is enabled and clients will need to send a cert to access the registry. To illustrate this, attempt a bare wget from a k8s worker and watch the docker-registry logs:

----------
worker:~$ wget https://172.31.20.67:5000/v2
Connecting to 172.31.20.67:5000... connected.
OpenSSL: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate

registry:~$ docker logs -f registry
2020/04/03 20:38:19 http: TLS handshake error from 172.31.5.244:54498: tls: client didn't provide a certificate
----------

Now do it again by using a cert/key with the wget client (and user auth if configured):

----------
worker:~$ wget --user=admin --password=password --certificate=/root/cdk/server.crt --private-key=/root/cdk/server.key https://172.31.20.67:5000/v2
Connecting to 172.31.20.67:5000... connected.
HTTP request sent, awaiting response... 200 OK
Saving to: ‘v2’

registry:~$ docker logs -f registry
172.31.5.244 - - [03/Apr/2020:20:45:54 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Wget/1.19.4 (linux-gnu)"
----------

So in this case, the issue is not that easyrsa certs are bad nor in the wrong place. The client needs to *send* a cert before the registry will respond, and wget doesn't do that by default. I could not find a way to specify certs when using 'ctr', so I'm not sure that client will ever work with an https registry (at least not one with a self-signed cert and mutual auth).

I *was* able to pull images from a k8s worker using crictl [0], but only after working on a fix for the aforementioned bug 1853653:

----------
worker:~$ sudo ./crictl pull 172.31.20.67:5000/defaultbackend-amd64:1.5
Image is up to date for sha256:b5af743e598496e8ebd7a6eb3fea76a6464041581520d1c2315c95f993287303

registry:~$ docker logs -f registry
172.31.5.244 - - [03/Apr/2020:20:56:29 +0000] "HEAD /v2/defaultbackend-amd64/manifests/1.5 HTTP/1.1" 200 528 "" "containerd/1.3.3-0ubuntu1~18.04.1"
----------

To sum up, clients that interact with a TLS-enabled docker-registry will need to send cert data. I'm going to mark this as a dupe of 1853653. Watch that bug for a fix and current workarounds.

[0]: https://kubernetes.io/docs/tasks/debug-application-cluster/crictl/#installing-crictl