k8s_5.0: kube_apiserver doesn't start after the Master reboot

[Venkatesh Velpula]

Setup

master :nodei23
nodes : nodei24, nodei25

orchestrator :K8s
image :latest
HostOS:

[root@nodei23 ~]# cat /etc/*release
CentOS Linux release 7.4.1708 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.4.1708 (Core)
CentOS Linux release 7.4.1708 (Core)

root@nodei23 ~]# netstat -nlp | grep 6443
tcp6 0 0 :::6443 :::* LISTEN 4584/kube-apiserver
[root@nodei23 ~]# ps -aef | grep kube-api
root 4584 4566 4 Mar04 ? 01:24:30 kube-apiserver --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --advertise-address=10.204.217.135 --service-account-key-file=/etc/kubernetes/pki/sa.pub --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt --insecure-port=0 --allow-privileged=true --service-cluster-ip-range=10.96.0.0/12 --tls-private-key-file=/etc/kubernetes/pki/apiserver.key --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota --requestheader-allowed-names=front-proxy-client --client-ca-file=/etc/kubernetes/pki/ca.crt --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt --secure-port=6443 --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key --requestheader-extra-headers-prefix=X-Remote-Extra- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cert-file=/etc/kubernetes/pki/apiserver.crt --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt --enable-bootstrap-token-auth=true --authorization-mode=Node,RBAC --etcd-servers=http://127.0.0.1:2379

after reboot
=============
[root@nodei23 ~]# uptime
 13:11:59 up 3 min, 1 user, load average: 3.01, 1.89, 0.77
[root@nodei23 ~]#
[root@nodei23 ~]# netstat -nlp | grep 6443
[root@nodei23 ~]#
[root@nodei23 ~]# systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since Mon 2018-03-05 13:12:08 IST; 10s ago
     Docs: http://kubernetes.io/docs/
  Process: 10769 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 10769 (code=exited, status=1/FAILURE)

Mar 05 13:12:08 nodei23 systemd[1]: Unit kubelet.service entered failed state.
Mar 05 13:12:08 nodei23 systemd[1]: kubelet.service failed.
Mar 05 13:12:18 nodei23 systemd[1]: kubelet.service holdoff time over, scheduling restart

[Sachchidanand Vaidya (vaidyasd)]

Can you pls include kubernetes version information.

Thanks
Sachin

On 3/5/18, 1:41 AM, "<email address hidden> on behalf of Sudheendra Rao" <<email address hidden> on behalf of <email address hidden>> wrote:

      k8s_5.0: kube_apiserver doesn't start after the aster reboot

[Venkatesh Velpula (vvelpula)]

# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.3", GitCommit:"d2835416544f298c919e2ead3be3d0864b52323b", GitTreeState:"clean", BuildDate:"2018-02-07T12:22:21Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

docker version
Client:
 Version: 17.03.1-ce
 API version: 1.27
 Go version: go1.7.5
 Git commit: c6d412e
 Built: Fri Mar 24 00:36:45 2017
 OS/Arch: linux/amd64

Server:
 Version: 17.03.1-ce
 API version: 1.27 (minimum version 1.12)
 Go version: go1.7.5
 Git commit: c6d412e
 Built: Fri Mar 24 00:36:45 2017
 OS/Arch: linux/amd64
 Experimental: false

cat /etc/*release
CentOS Linux release 7.4.1708 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

CentOS Linux release 7.4.1708 (Core)
CentOS Linux release 7.4.1708 (Core)

[Prasanna Mucharikar (mprasanna)]

Fixed by
https://review.opencontrail.org/#/c/40334/1