Brief Description
-----------------
Install DC system-controller controller-0, run ansible-playbook bootstrap failed. The error msg shows,
It seems like the kubelet isn't running or healthy.", "[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.",
Severity
--------
Critical
Steps to Reproduce
------------------
as description
TC-name: DC installation
Expected Behavior
------------------
Actual Behavior
----------------
Reproducibility
---------------
intermittent (failed 2 of 3)
System Configuration
--------------------
DC system
IPv6
Lab-name: WCP_90
Branch/Pull Time/Commit
-----------------------
19.10 master as of 2019-10-06_20-00-00
Last Pass
---------
2019-09-30_20-00-00
Timestamp/Logs
--------------
TASK [bootstrap/bringup-essential-services : Initializing Kubernetes master] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": ["kubeadm", "init", "--config=/etc/kubernetes/kubeadm.yaml"], "delta": "0:01:57.697315", "end": "2019-10-07 19:48:22.425851", "msg": "non-zero return code", "rc": 1, "start": "2019-10-07 19:46:24.728536", "stderr": "\t[WARNING IsDockerSystemdCheck]: detected \"cgroupfs\" as the Docker cgroup driver. The recommended driver is \"systemd\". Please follow the guide at https://kubernetes.io/docs/setup/cri/\nerror execution phase wait-control-plane: couldn't initialize a Kubernetes cluster\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["\t[WARNING IsDockerSystemdCheck]: detected \"cgroupfs\" as the Docker cgroup driver. The recommended driver is \"systemd\". Please follow the guide at https://kubernetes.io/docs/setup/cri/", "error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "[init] Using Kubernetes version: v1.16.0\n[preflight] Running pre-flight checks\n[preflight] Pulling images required for setting up a Kubernetes cluster\n[preflight] This might take a minute or two, depending on the speed of your internet connection\n[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'\n[kubelet-start] Writing kubelet environment file with flags to file \"/var/lib/kubelet/kubeadm-flags.env\"\n[kubelet-start] Writing kubelet configuration to file \"/var/lib/kubelet/config.yaml\"\n[kubelet-start] Activating the kubelet service\n[certs] Using certificateDir folder \"/etc/kubernetes/pki\"\n[certs] Generating \"ca\" certificate and key\n[certs] Generating \"apiserver\" certificate and key\n[certs] apiserver serving cert is signed for DNS names [controller-0 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [fd04::1 aefd::2 aefd::2 ::1]\n[certs] Generating \"apiserver-kubelet-client\" certificate and key\n[certs] Generating \"front-proxy-ca\" certificate and key\n[certs] Generating \"front-proxy-client\" certificate and key\n[certs] External etcd mode: Skipping etcd/ca certificate authority generation\n[certs] External etcd mode: Skipping etcd/server certificate generation\n[certs] External etcd mode: Skipping etcd/peer certificate generation\n[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation\n[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation\n[certs] Generating \"sa\" key and public key\n[kubeconfig] Using kubeconfig folder \"/etc/kubernetes\"\n[kubeconfig] Writing \"admin.conf\" kubeconfig file\n[kubeconfig] Writing \"kubelet.conf\" kubeconfig file\n[kubeconfig] Writing \"controller-manager.conf\" kubeconfig file\n[kubeconfig] Writing \"scheduler.conf\" kubeconfig file\n[control-plane] Using manifest folder \"/etc/kubernetes/manifests\"\n[control-plane] Creating static Pod manifest for \"kube-apiserver\"\n[control-plane] Creating static Pod manifest for \"kube-controller-manager\"\n[control-plane] Creating static Pod manifest for \"kube-scheduler\"\n[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory \"/etc/kubernetes/manifests\". This can take up to 4m0s\n[kubelet-check] Initial timeout of 40s passed.\n[kubelet-check] It seems like the kubelet isn't running or healthy.\n[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
It seems kubelet keep restart
yow-cgcs-wildcat-90:~$ systemctl status kubelet
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─kubeadm.conf
Active: active (running) since Mon 2019-10-07 20:20:08 UTC; 3min 23s ago
Docs: https://kubernetes.io/docs/concepts/overview/components/#kubelet
https://kubernetes.io/docs/reference/generated/kubelet/
Process: 193679 ExecStartPost=/bin/bash -c echo $MAINPID > /var/run/kubelet.pid; (code=exited, status=0/SUCCESS)
Process: 193670 ExecStartPre=/usr/bin/kubelet-cgroup-setup.sh (code=exited, status=0/SUCCESS)
Main PID: 193678 (kubelet)
Tasks: 17
Memory: 33.0M
CGroup: /system.slice/kubelet.service
└─193678 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupf...
yow-cgcs-wildcat-90:~$ systemctl status kubelet
● kubelet.service - Kubernetes Kubelet Server
Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─kubeadm.conf
Active: active (running) since Mon 2019-10-07 20:24:01 UTC; 11s ago
Docs: https://kubernetes.io/docs/concepts/overview/components/#kubelet
https://kubernetes.io/docs/reference/generated/kubelet/
Process: 196594 ExecStopPost=/bin/rm -f /var/run/kubelet.pid (code=exited, status=0/SUCCESS)
Process: 197931 ExecStartPost=/bin/bash -c echo $MAINPID > /var/run/kubelet.pid; (code=exited, status=0/SUCCESS)
Process: 197901 ExecStartPre=/usr/bin/kubelet-cgroup-setup.sh (code=exited, status=0/SUCCESS)
Main PID: 197930 (kubelet)
Tasks: 15
Memory: 27.5M
CGroup: /system.slice/kubelet.service
└─197930 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupf...
yow-cgcs-wildcat-90:~$
Test Activity
-------------
DC installation
Reinstalled the same load and run ansible bootstrap again. This time it is passed. It is an intermittent issue.