default image-registry results in invalid image paths of cdk addons

[LP]

With the latest kubernetes-master charm, the 'image-registry' is set to image-registry.canonical.com:5000/cdk

This results in cdk addon deployment container image paths which are not existing. E.g.
image-registry.canonical.com:5000/cdk/addon-resizer:1.8.3

I also think this default registry on port 5000 is not very firewall-friendly, though that is not the problem here.

[root@cdk-master ~]# kubectl describe deployment heapster-v1.6.0-beta.1 -n kube-system |grep Image
    Image: k8s.gcr.io/heapster-amd64:v1.6.0-beta.1
    Image: k8s.gcr.io/heapster-amd64:v1.6.0-beta.1
    Image: k8s.gcr.io/addon-resizer:1.8.3
    Image: k8s.gcr.io/addon-resizer:1.8.2
[root@cdk-master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-policy-controller-6fffbb48f6-d8cn6 1/1 Running 2 59d
heapster-v1.6.0-beta.1-6db4b87d-rz5hg 4/4 Running 8 24h
kube-dns-596fbb8fbd-6nrrw 3/3 Running 6 24h
kubernetes-dashboard-6bdb474f78-hxlgb 1/1 Running 1 24h
metrics-server-v0.3.1-67bb5c8d7-knb64 2/2 Running 4 24h
monitoring-influxdb-grafana-v4-65cc9bb8c8-njmzj 2/2 Running 4 24h

==========
juju config kubernetes-master image-registry="image-registry.canonical.com:5000/cdk"
==========

[root@cdk-master ~]# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
calico-policy-controller-6fffbb48f6-d8cn6 1/1 Running 2 59d
heapster-v1.6.0-beta.1-6db4b87d-rz5hg 0/4 Terminating 8 24h
heapster-v1.6.0-beta.1-745bbc7858-nxhrs 2/4 ErrImagePull 0 3s
kube-dns-596fbb8fbd-6nrrw 3/3 Running 6 24h
kube-dns-84846d8894-v6fqm 0/3 ContainerCreating 0 4s
kubernetes-dashboard-6bdb474f78-hxlgb 1/1 Running 1 24h
kubernetes-dashboard-b5f76f75b-gw4wz 0/1 ContainerCreating 0 3s
metrics-server-v0.3.1-67bb5c8d7-knb64 2/2 Terminating 4 24h
metrics-server-v0.3.1-95485c99f-l4kmk 0/2 ContainerCreating 0 2s
monitoring-influxdb-grafana-v4-65cc9bb8c8-njmzj 2/2 Terminating 4 24h
monitoring-influxdb-grafana-v4-94b567944-xwvn7 0/2 ContainerCreating 0 3s
[root@cdk-master ~]# kubectl describe deployment heapster-v1.6.0-beta.1 -n kube-system |grep Image
    Image: image-registry.canonical.com:5000/cdk/heapster-amd64:v1.6.0-beta.1
    Image: image-registry.canonical.com:5000/cdk/heapster-amd64:v1.6.0-beta.1
    Image: image-registry.canonical.com:5000/cdk/addon-resizer:1.8.3
    Image: image-registry.canonical.com:5000/cdk/addon-resizer:1.8.2
[root@cdk-master ~]# docker pull image-registry.canonical.com:5000/cdk/addon-resizer:1.8.3
Error response from daemon: manifest for image-registry.canonical.com:5000/cdk/addon-resizer:1.8.3 not found

[George Kraft (cynerva)]

What version of Kubernetes are you running?

[LP (lopf)]

1.13 with the latest kubernetes-master charm version 700

[George Kraft (cynerva)]

Thanks. I deployed the latest stable charms configured with channel=1.13/stable, but did not encounter the issue.

In my deployment, the addon-resizer image in use is image-registry.canonical.com:5000/cdk/addon-resizer:1.8.4

$ juju status
Model Controller Cloud/Region Version SLA Timestamp
test aws-dev aws/us-east-2 2.6.5 unsupported 11:16:06-05:00

App Version Status Scale Charm Store Rev OS Notes
containerd active 5 containerd jujucharms 2 ubuntu
easyrsa 3.0.1 active 1 easyrsa jujucharms 254 ubuntu
etcd 3.2.10 active 3 etcd jujucharms 434 ubuntu
flannel 0.10.0 active 5 flannel jujucharms 425 ubuntu
kubeapi-load-balancer 1.14.0 active 1 kubeapi-load-balancer jujucharms 649 ubuntu exposed
kubernetes-master 1.13.7 active 2 kubernetes-master jujucharms 700 ubuntu
kubernetes-worker 1.13.7 active 3 kubernetes-worker jujucharms 552 ubuntu exposed

...

$ kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
heapster-v1.6.0-beta.1-5cc96db88b-l898s 4/4 Running 0 11m
kube-dns-5c4bb99ddf-j7lf4 3/3 Running 0 13m
kubernetes-dashboard-86445dc6f9-24zvj 1/1 Running 0 13m
metrics-server-v0.3.1-758586fddb-47sxh 2/2 Running 0 12m
monitoring-influxdb-grafana-v4-5447fdf9bc-jrfmd 2/2 Running 0 13m

$ kubectl describe deployment -n kube-system heapster-v1.6.0-beta.1 | grep Image
    Image: image-registry.canonical.com:5000/cdk/heapster-amd64:v1.6.0-beta.1
    Image: image-registry.canonical.com:5000/cdk/heapster-amd64:v1.6.0-beta.1
    Image: image-registry.canonical.com:5000/cdk/addon-resizer:1.8.4
    Image: image-registry.canonical.com:5000/cdk/addon-resizer:1.8.4

[George Kraft (cynerva)]

It seems like the unusual part in your deployment is the use of addon-resizer:1.8.3 and addon-resize:1.8.2. That's not expected.

The heapster addon definition is provided by cdk-addons. Can you check what version of cdk-addons is running on your kubernetes-master units?

$ juju run --application kubernetes-master snap list
- Stdout: |
    Name Version Rev Tracking Publisher Notes
    amazon-ssm-agent 2.3.612.0 1335 stable/… aws* classic
    cdk-addons 1.13.7 913 1.13 canonical* -
    core 16-2.39.2 7169 stable canonical* core
    kube-apiserver 1.13.7 1019 1.13 canonical* -
    kube-controller-manager 1.13.7 1014 1.13 canonical* -
    kube-proxy 1.13.7 1028 1.13 canonical* classic
    kube-scheduler 1.13.7 1013 1.13 canonical* -
    kubectl 1.13.7 1019 1.13 canonical* classic
  UnitId: kubernetes-master/0
- Stdout: |
    Name Version Rev Tracking Publisher Notes
    amazon-ssm-agent 2.3.612.0 1335 stable/… aws* classic
    cdk-addons 1.13.7 913 1.13 canonical* -
    core 16-2.39.2 7169 stable canonical* core
    kube-apiserver 1.13.7 1019 1.13 canonical* -
    kube-controller-manager 1.13.7 1014 1.13 canonical* -
    kube-proxy 1.13.7 1028 1.13 canonical* classic
    kube-scheduler 1.13.7 1013 1.13 canonical* -
    kubectl 1.13.7 1019 1.13 canonical* classic
  UnitId: kubernetes-master/1

[LP (lopf)]

I'm so sorry, I was wrong about the Kubernetes version. I was testing it on a cluster with version 1.12. I can't verify on 1.13 because we're stuck in the upgrade path. Ticket is open with support. I'll report back once we get to 1.13.

[Kevin W Monroe (kwmonroe)]

Officially, we support the latest 3 k8s releases (that is 1.13/14/15 today). I do see that addons for 1.12 uses addon-resizer 1.8.2 and 1.8.3:

https://github.com/charmed-kubernetes/bundle/blob/master/container-images.txt#L2

Yet those images are somehow not present in the registry:

https://image-registry.canonical.com:5000/v2/cdk/addon-resizer/tags/list

We can push them, but it won't be necessary if you're going to 1.13. I'm marking this bug incomplete for now; let us know if you'd like to close this as invalid (fixed in 1.13) or if you're stuck on 1.12 for some reason and need these images pushed.