Not able to create TLS_DISABLED k8s cluster

[yatin]

Setup magnum with devstack:-
My env:-
[centos@devstack-1 magnum]$ magnum-api --version
5.0.1.dev15
[centos@devstack-1 magnum]$ magnum --version
2.7.0

Create TLS_DISABLED k8s cluster.

Login to k8s master node:-
kube api server failed to start:-

[fedora@k8s-cluster-ecked4vw4tnl-master-0 ~]$ sudo atomic containers list
   CONTAINER ID IMAGE COMMAND CREATED STATE BACKEND RUNTIME
   kubelet docker.io/openstackm /usr/bin/kubelet-doc 2017-09-04 05:11 running ostree runc
   kube-proxy docker.io/openstackm /usr/bin/kube-proxy- 2017-09-04 05:11 running ostree runc
   kube-apiserv docker.io/openstackm /usr/bin/kube-apiser 2017-09-04 05:12 failed ostree runc
   kube-control docker.io/openstackm /usr/bin/kube-contro 2017-09-04 05:12 running ostree runc
   kube-schedul docker.io/openstackm /usr/bin/kube-schedu 2017-09-04 05:12 running ostree runc

Following Error is seen in apiserver logs: Sep 04 06:09:54 k8s-cluster-ecked4vw4tnl-master-0.novalocal runc[7368]: I0904 06:09:54.619508 1 server.go:112] Version: v1.7.4
Sep 04 06:09:54 k8s-cluster-ecked4vw4tnl-master-0.novalocal runc[7368]: error creating self-signed certificates: open /var/run/kubernetes/apiserver.crt: read-only file system

/var/run/kubernetes is the default cirt-dir for apiserver certs.

It looks like atomic container(k8s apiserver 1.7.4) is trying to create certs even for TLS_DISABLED cluster.

[Spyros Trigazis (strigazi)]

If we mount /var/run/kubernetes does it work? I mean if it can write in /var/run/kubernetes

[yatin (yatinkarel)]

Haven't tried this, will try and update.

[Spyros Trigazis (strigazi)]

We can try to pass this parameter to the apiserver --cert-dir
when tls-disbled.

from the docs [1]:

      --cert-dir string The directory where the TLS certs
                               are located. If --tls-cert-file and
                               --tls-private-key-file are provided,
                               this flag will be ignored. (default
                               "/var/run/kubernetes")

with /etc/kubernetes/certs/ (make sure that the dir exists in the non-tls case)

[1] https://kubernetes.io/docs/admin/kube-apiserver/

[Spyros Trigazis (strigazi)]

It doesn't work, we need to have a writable directory for the kubernetes-apiserver system container OR create certs for the apiserver. I'll open an issue in atomic-system-containers

[Spyros Trigazis (strigazi)]

https://github.com/projectatomic/atomic-system-containers/issues/133

[Lingxian Kong (kong)]

Hi, Spyros, how we fix this in magnum ourselves?

[Spyros Trigazis (strigazi)]

In the v1.7.7 container images the problem is fixed (in pike). I'm not sure how useful is a non tls cluster though. With RBAC enabled it can not work. AFAIK there is no option for RBAC enabled clusters to work without tls. If there is a requirement to have non-tls cluster we can do the following separation.

Generate certs and enable rbac and expose the api at 6443 with tls, AND expose the api at 8080 as well.

[Spyros Trigazis (strigazi)]

In any case you need the fixed images with 1.7.7 or higher to have the proper mounts and permissions.

[Rick Cano (ledsole)]

I am still experiencing this issue with the latest Fedora Atomic (Fedora-Atomic-27-20180326.1.x86_64.qcow2 as of this writing) on Queens. Do I need to build my own atomic image with disk image-builder for this to work?