kube-dns service did not come up for almost an hour

[Vedamurthy Joshi]

R4.0.1.0 Continuous build 19 Ubuntu 16.04.2

After provisioning k8s,contrail cluster, kube-dns pod took a very long time(about an hour) to come up. It was seen in agent that 9091:vm/<vm-uuid> request was failing even though interface was Active in agent.

Gcore of agent is taken. We will have to see if this is reproducible later.

All logs will be in http://10.204.216.50/Docs/bugs/#

root@nodec1:~# kubectl get pods --namespace=kube-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE
etcd-nodec1 1/1 Running 1 2h 10.204.216.58 nodec1
kube-apiserver-nodec1 1/1 Running 1 2h 10.204.216.58 nodec1
kube-controller-manager-nodec1 1/1 Running 2 2h 10.204.216.58 nodec1
kube-dns-1327093014-1kt43 0/3 ContainerCreating 0 24s <none> nodek3
kube-proxy-0fjt0 1/1 Running 1 2h 10.204.216.223 nodek3
kube-proxy-5lwkt 1/1 Running 1 2h 10.204.216.221 nodek1
kube-proxy-766kc 1/1 Running 1 2h 10.204.216.222 nodek2
kube-proxy-fmtxq 1/1 Running 1 2h 10.204.216.58 nodec1
kube-scheduler-nodec1 1/1 Running 2 2h 10.204.216.58 nodec1
root@nodec1:~#

CNI log :

I : 16139 : 2017/08/14 15:34:00 vrouter.go:79: VRouter request. Operation : GET Url : http://127.0.0.1:9091/vm/bfdcd977-80cf-11e7-9b94-002590c30af2
E : 16139 : 2017/08/14 15:34:00 vrouter.go:147: Failed HTTP Get operation. Return code 404
I : 16139 : 2017/08/14 15:34:00 vrouter.go:181: Iteration 13 : Get vrouter failed
I : 16139 : 2017/08/14 15:34:05 vrouter.go:79: VRouter request. Operation : GET Url : http://127.0.0.1:9091/vm/bfdcd977-80cf-11e7-9b94-002590c30af2
E : 16139 : 2017/08/14 15:34:05 vrouter.go:147: Failed HTTP Get operation. Return code 404
I : 16139 : 2017/08/14 15:34:05 vrouter.go:181: Iteration 14 : Get vrouter failed
E : 16139 : 2017/08/14 15:34:10 vrouter.go:285: Error in polling VRouter
I : 16139 : 2017/08/14 15:34:10 cni.go:171: Error in Add to VRouter

[Vedamurthy Joshi (vedujoshi)]

This problem was not seen in recent days. Marking it as incomplete for now