kubernetes pods not getting IPs at times

[Vedamurthy Joshi]

This was seen when pods where deleting and re-added quickly. Any new pods from then on dont get their IPs

workaround to restart the contrail-kube-manager

root@vedu-vm1-1:/# supervisorctl -s unix:///var/run/supervisord_kubernetes.sock
contrail-kube-manager RUNNING pid 560, uptime 21:53:36
supervisor> restart all
contrail-kube-manager: stopped
contrail-kube-manager: started
supervisor> exit
root@vedu-vm1-1:/#

[root@vedu-vm1-1 centos]# cat nginx.yaml
apiVersion: v1
kind: ReplicationController
metadata:
 name: nginx-controller
spec:
 replicas: 2
 selector:
   name: nginx
 template:
   metadata:
     labels:
       name: nginx
   spec:
     containers:
       - name: nginx
         image: nginx
         ports:
           - containerPort: 80

[root@vedu-vm1-1 centos]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-controller-6cz1r 0/1 Terminating 0 1m <none> vedu-vm1-2
nginx-controller-752br 0/1 Terminating 0 1m <none> vedu-vm1-2
nginx-controller-mnbb1 0/1 ContainerCreating 0 45s <none> vedu-vm1-2
nginx-controller-xpg7g 0/1 ContainerCreating 0 45s <none> vedu-vm1-2
[root@vedu-vm1-1 centos]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-controller-6cz1r 0/1 Terminating 0 1m <none> vedu-vm1-2
nginx-controller-752br 0/1 Terminating 0 1m <none> vedu-vm1-2
nginx-controller-mnbb1 0/1 ContainerCreating 0 46s <none> vedu-vm1-2
nginx-controller-xpg7g 0/1 ContainerCreating 0 46s <none> vedu-vm1-2
[root@vedu-vm1-1 centos]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-controller-6cz1r 0/1 Terminating 0 1m <none> vedu-vm1-2
nginx-controller-752br 0/1 Terminating 0 1m <none> vedu-vm1-2
nginx-controller-mnbb1 0/1 ContainerCreating 0 48s <none> vedu-vm1-2
nginx-controller-xpg7g 0/1 ContainerCreating 0 48s <none> vedu-vm1-2
[root@vedu-vm1-1 centos]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-controller-mnbb1 0/1 ContainerCreating 0 1m <none> vedu-vm1-2
nginx-controller-xpg7g 0/1 ContainerCreating 0 1m <none> vedu-vm1-2
[root@vedu-vm1-1 centos]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-controller-mnbb1 0/1 ContainerCreating 0 1m <none> vedu-vm1-2
nginx-controller-xpg7g 0/1 ContainerCreating 0 1m <none> vedu-vm1-2
[root@vedu-vm1-1 centos]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
nginx-controller-mnbb1 0/1 ContainerCreating 0 1m <none> vedu-vm1-2
nginx-controller-xpg7g 0/1 ContainerCreating 0 1m <none> vedu-vm1-2
[root@vedu-vm1-1 centos]#

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28582
Submitter: Yuvaraja Mariappan

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28582
Committed: http://github.org/Juniper/contrail-controller/commit/d7812d326c5add4cbf9ed65dd5a465815768a44a
Submitter: Zuul (<email address hidden>)
Branch: master

commit d7812d326c5add4cbf9ed65dd5a465815768a44a
Author: Yuvaraja Mariappan <email address hidden>
Date: Wed Feb 8 20:28:14 2017 -0800

Fixed connection issue with kube_api_server

kube_api_server closes the connection which is registered
for watch event on timeout which caused the issue.
By increasing the timeout in the kube_api_server,
the connection close can be postponed. But it
wont solve the issue completly. fix added to reinitiate
the connection, if it is lost.

Change-Id: Ic3318588d6176e6e7a977800c625ed18187e2d2e
Closes-bug: 1657918