External Docker Registry with Self-Signed CA TLS endpoints not supported by charm

[Drew Freiberger]

When trying to deploy workloads within a kubernetes cluster that point to an external https://myregistry.myco.com/ URL for an image, with the TLS of that registry being configured under a private CA, there is no way to inject that CA into the /etc/docker/certs.d directory in the kubernetes-workers, hence you cannot deploy from non-public registries into your CDK cluster if TLS is enabled.

This is a feature request to allow for adding a mechanism to import a docker-registry-ca-bundle to the charm separate from tls-certificates and docker-registry relation interfaces as either a config option or an attachable resource.

[Drew Freiberger (afreiberger)]

Manual workaround is to land the CA file(s) into /etc/docker/certs.d directory on each kubernetes-worker node.

[Paul Goins (vultaire)]

Tagged as field-medium.

[Kevin W Monroe (kwmonroe)]

Seems reasonable to offer a way to configure external registry TLS blobs. Let's do this in the container runtime layer to handle both dockerd and containerd at the same time. I'm invalidating k8s-worker work, as that charm won't need anything special once the runtime supports this.

[Kevin W Monroe (kwmonroe)]

@drew, I'd like to confirm a few things to get this fixed:

- k8s version
- k8s worker charm revision; if local, do you recall when the local charm was pulled?
- container runtime (docker or containerd) charm revs; same ^ request if they're local.
- xenial or bionic workers?

In the description, you asked for a config on k8s-worker, but in comment 3, I alluded to new config in the container runtime. I still think the runtime layer is the right place for the fix, but want to be sure it covers your case.

[Drew Freiberger (afreiberger)]

The particular cloud this is of interest for is running Kubernetes 1.13 with an 18.09.2 docker on bionic. I see that there's a docker rev 19.03.6 available in the bionic-updates repository that's on-deck for next patch cycle.

Your point that this could be put in a common layer that can be integrated into all k8s charms that generically injects into the container runtime makes great sense. If the hook isn't available back to 1.13 we may be able to continue using our workaround until the environment can be upgraded to 1.17+.

[Kevin W Monroe (kwmonroe)]

PR is available for review here:

https://github.com/charmed-kubernetes/layer-container-runtime-common/pull/7

Unfortunately, this won't help the 1.13 deployment. I've been looking into ways to provide similar functionality, but we completely changed the way container runtimes were handled in 1.14+ -- we now utilize runtime subordinate charms with all config being handled on those. This means we don't have a good way to deliver docker-ish features (like custom CA config) to something like a k8s-worker charm, as that wouldn't be applicable to deployments >= 1.14.

My best workaround is the one you already discovered back in comment #1: manually get the ca.crt onto the k8s-workers (either in /etc/docker/certs.d, or /usr/local/share/ca-certificates && update-ca-certificates), and 'systemctl restart docker'.

[George Kraft (cynerva)]

Cherry-picked to stable: https://github.com/charmed-kubernetes/layer-container-runtime-common/commit/0986a61192dc276043d1b6f91d4d18557a975223