Multicloud :: Azure OnPrem :: k8s pods stuck in state ContainerCreating
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R5.0 |
Incomplete
|
High
|
Ritam Gangopadhyay | |||
Trunk |
Incomplete
|
High
|
Ritam Gangopadhyay |
Bug Description
Setup:-
contrail-controller and k8s_master - 10.87.74.129 - 192.168.1.1
contrail-compute and k8s_node - 10.87.74.130 - 192.168.1.2
root@5c3s1-node1:~# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
ubuntuapp-custom1 1/1 Running 0 2d 10.1.1.4 5c3s1-node2
ubuntuapp-custom2 0/1 Pending 0 2d <none> <none>
ubuntuapp-custom4 0/1 ContainerCreating 0 4h <none> rg-compute-1
ubuntuapp-local1 0/1 ContainerCreating 0 4h <none> 5c3s1-node2
root@5c3s1-node1:~#
cni logs
*******
*******
*******
I : 14779 : 2018/10/26 08:00:14 contrail-
I : 14779 : 2018/10/26 08:00:14 contrail-
I : 14779 : 2018/10/26 08:00:14 cni.go:88: ContainerID : a74e4674c95bf8d
I : 14779 : 2018/10/26 08:00:14 cni.go:89: NetNS : /proc/14713/ns/net
I : 14779 : 2018/10/26 08:00:14 cni.go:90: Container Ifname : eth0
I : 14779 : 2018/10/26 08:00:14 cni.go:91: Args : IgnoreUnknown=
I : 14779 : 2018/10/26 08:00:14 cni.go:92: CNI VERSION : 0.2.0
I : 14779 : 2018/10/26 08:00:14 cni.go:93: MTU : 1500
I : 14779 : 2018/10/26 08:00:14 cni.go:94: Config File : {"cniVersion"
I : 14779 : 2018/10/26 08:00:14 cni.go:95: &{cniArgs:
I : 14779 : 2018/10/26 08:00:14 vrouter.go:446: {Server:127.0.0.1 Port:9091 Dir:/var/
I : 14779 : 2018/10/26 08:00:14 vrouter.go:79: VRouter request. Operation : GET Url : http://
E : 14779 : 2018/10/26 08:00:14 vrouter.go:147: Failed HTTP Get operation. Return code 404
I : 14779 : 2018/10/26 08:00:14 vrouter.go:181: Iteration 0 : Get vrouter failed
I : 14779 : 2018/10/26 08:00:19 vrouter.go:79: VRouter request. Operation : GET Url : http://
E : 14779 : 2018/10/26 08:00:19 vrouter.go:147: Failed HTTP Get operation. Return code 404
I : 14779 : 2018/10/26 08:00:19 vrouter.go:181: Iteration 1 : Get vrouter failed
*******
*******
*******
TRACEBACK SEEN IN KUBE-MANAGER.LOG
*******
*******
*******
The above is a description of an error in a Python program. Here is
the original traceback:
Traceback (most recent call last):
File "/usr/lib/
self.
File "/usr/lib/
specified_
File "/usr/lib/
service_ip, ports)
File "/usr/lib/
service_
File "/usr/lib/
vn_obj, service_ip, service_
File "/usr/lib/
tags=tags)
File "/usr/lib/
self.
File "/usr/lib/
return func(self, *args, **kwargs)
File "/usr/lib/
OP_PUT, uri, data=json_body)
File "/usr/lib/
retry_
File "/usr/lib/
% (op, url, data, content))
NoIdError: Unknown id: Error: oper 3 url /instance-
10/26/2018 03:15:01 PM [contrail-
10/26/2018 03:15:01 PM [contrail-
10/26/2018 03:15:01 PM [contrail-
10/26/2018 03:15:01 PM [contrail-
*******
*******
*******
the uuid for the instance IP does not exist in config api but the vmi ref does
root@5c3s1-node1:~# curl -s http://
root@5c3s1-node1:~# curl -s http://
"href": "http://
"uuid": "e3b2eeb7-
root@5c3s1-node1:~#
"uuid": "e3b2eeb7-
]
},
Not a blocker, k8s cleanup with drain and delete of all nodes and re-join fixes the issue.