k8s:Multi interface pod creation fails unless kubemanager is restarted

[Venkatesh Velpula]

pod creation fails while associating multiple interface and also repeated Error messages are popped up in kube-manager logs .

work around is to restart the kube manager

deployer :contrail-command
ansible :2.5.2
docker :18.03.1-ce
HostOs :CentOS
build :Queens-Master-422

Topology
=========
[root@nodec21 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
nodec21 NotReady master 60m v1.12.1
nodei18 Ready <none> 59m v1.12.1
[root@nodec21 ~]#

[root@nodec21 ~]# kubectl describe multinetworkpodtest3
error: the server doesn't have a resource type "multinetworkpodtest3"
[root@nodec21 ~]# kubectl describe pods multinetworkpodtest3
Name: multinetworkpodtest3
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: nodei18/10.204.217.130
Start Time: Mon, 10 Dec 2018 00:07:32 +0530
Labels: <none>
Annotations: k8s.v1.cni.cncf.io/networks: [ { "name": "net100" } ]
Status: Pending
IP:
Containers:
  busybox10:
    Container ID:
    Image: busybox
    Image ID:
    Port: <none>
    Host Port: <none>
    Command:
      sleep
      3600
    State: Waiting
      Reason: ContainerCreating
    Ready: False
    Restart Count: 0
    Environment: <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dn992 (ro)
Conditions:
  Type Status
  Initialized True
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  default-token-dn992:
    Type: Secret (a volume populated by a Secret)
    SecretName: default-token-dn992
    Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 10m default-scheduler Successfully assigned default/multinetworkpodtest3 to nodei18
  Warning FailedCreatePodSandBox 9m15s kubelet, nodei18 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "55067dcb8e88bc667d48c5dd9c57e332c147921476c7caac44cc2d315ec22fbe" network for pod "multinetworkpodtest3": NetworkPlugin cni failed to set up pod "multinetworkpodtest3_default" network: Failed in Poll VM-CFG. Error : Iteration 14 : Get VRouter Incomplete - Interfaces Expected: 2, Actual: 1

kube-manager logs
=================
12/10/2018 12:16:06 AM [contrail-kube-manager] [ERROR]: __default__ [SYS_ERR]: KubeManagerErrorLog: NetworkMonitor - Event handler not found. Cannot process its events.
12/10/2018 12:16:06 AM [contrail-kube-manager] [ERROR]: NetworkMonitor - Event handler not found. Cannot process its events.

yaml file used to create the pod and network attachment definitions

[root@nodec21 ~]# cat pod.yaml
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: net100
  annotations:
    opencontrail.org/cidr: "192.168.10.0/24"
    opencontrail.org/ip_fabric_snat: "false"
    opencontrail.org/ip_fabric_forwarding: "true"
spec:
  config: '{
    cniVersion: "0.3.0",
    type: "contrail-k8s-cni"
}'
---
apiVersion: v1
kind: Pod
metadata:
  name: multinetworkpodtest3
  annotations:
    k8s.v1.cni.cncf.io/networks: '[
      { "name": "net100" }
    ]'
spec:
  containers:
  - image: busybox
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
    name: busybox10
    stdin: true
    tty: true
  restartPolicy: Always

[Venkatraman Venkatapathy (vvenkatapath)]

Does this happen only with contrail command or otherwise too? I tried locally and I don't see this happening.

[Venkatesh Velpula (vvelpula)]

Yes..venkat ..its the same behaviour i am observing with ansible deployer too

[Venkatraman Venkatapathy (vvenkatapath)]

Update: Venky tried with latest queens-master-437 and the issue is not seen. To be reopened if seen.