no flannel binaries after recreating relation

[Jorge Niedbalski]

[Description]

When the juju relation between flannel and kubernetes-worker gets removed,
the new pods won't start.

[Reproducer]

1) Deploy the following bundle (http://paste.ubuntu.com/p/CsBMHvy2XW/)
2) Create a pod as follows

$ cat busybox.yaml
apiVersion: v1
kind: Pod
metadata:
  name: busybox-default
  namespace: default
  labels:
    app: busybox
spec:
  containers:
  - image: busybox
    imagePullPolicy: IfNotPresent
    name: busybox-default

    command:
      - sleep
      - "36000"

$ kubectl create -f busybox.yaml

2.1) Check kubectl pod status as running

$ kubectl get po

3) Remove the flannel -> kubernetes-worker relation

$ juju remove-relation kubernetes-worker flannel

4) Re-add the flannel kubernetes-worker relation

$ juju add-relation kuberentes-worker flannel

5) Check that pod status remains as creating

$ kubectl get po

busybox-default 0/1 ContainerCreating 0 3m14s

6) Check current pod description

$ kubectl describe po

  Normal Scheduled 13m default-scheduler Successfully assigned default/busybox-default to juju-e9281f-k8s-break-flannel-5
  Warning FailedCreatePodSandBox 13m kubelet, juju-e9281f-k8s-break-flannel-5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4b3094a2112015ea4a7eb95d1545b503a8c981fce4f446cf446fbf376506eb5b": failed to find plugin "flannel" in path [/opt/cni/bin]
  Warning FailedCreatePodSandBox 13m kubelet, juju-e9281f-k8s-break-flannel-5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6ac9b49a5bf50eba410290b0c2c93a2b20e8f65411d2e9dd780d13f1beca7db1": failed to find plugin "flannel" in path [/opt/cni/bin]
  Warning FailedCreatePodSandBox 13m kubelet, juju-e9281f-k8s-break-flannel-5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "87a8585acd5054e14a06a17d3ecc412e42300ba3aab228ca4d76acbc3a36b18e": failed to find plugin "flannel" in path [/opt/cni/bin]
  Warning FailedCreatePodSandBox 12m kubelet, juju-e9281f-k8s-break-flannel-5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "e9a93cb4df85af07e9e226700c494faf8a8706d298e1ce92be264010b1026589": failed to find plugin "flannel" in path [/opt/cni/bin]
  Warning FailedCreatePodSandBox 12m kubelet, juju-e9281f-k8s-break-flannel-5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "6da96ad52cfb3aefa393423a47bd1f25e223c547a19750dcc61d8bc00388dbc4": failed to find plugin "flannel" in path [/opt/cni/bin]
  Warning FailedCreatePodSandBox 12m kubelet, juju-e9281f-k8s-break-flannel-5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "b832bc2c9edf77e77b869c969cbb31ecdab690ba521f3f534f6098fffb1be163": failed to find plugin "flannel" in path [/opt/cni/bin]
  Warning FailedCreatePodSandBox 12m kubelet, juju-e9281f-k8s-break-flannel-5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "5b72ccca2c48fd7e9351fb81446baea0934db2e7773c98da8d0b2b73a89cf680": failed to find plugin "flannel" in path [/opt/cni/bin]
  Warning FailedCreatePodSandBox 11m kubelet, juju-e9281f-k8s-break-flannel-5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4c1c970dffacb484d1eb24eeb25429249ba3c5c74acfea0bd2842097e4ddbb26": failed to find plugin "flannel" in path [/opt/cni/bin]
  Warning FailedCreatePodSandBox 11m kubelet, juju-e9281f-k8s-break-flannel-5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "f7435ac5627687f51022783094744e697edde507d7f45ec4fc62d0362deb5270": failed to find plugin "flannel" in path [/opt/cni/bin]
  Warning FailedCreatePodSandBox 3m32s (x38 over 11m) kubelet, juju-e9281f-k8s-break-flannel-5 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "ff523cc7ced491769d7bd9e0e88f1e0c71999401b32a1df3c0e6d7591cf164e4": failed to find plugin "flannel" in path [/opt/cni/bin]

[Expected behavior]

New pods are in running state after re-adding the relation.

[Suggested solution]

The 'stop' cleanup_deployment method should clear the installed
state (https://github.com/charmed-kubernetes/charm-flannel/blob/master/reactive/flannel.py#L322)

[George Kraft (cynerva)]

Looks like the flannel charm is removing files from /opt/cni/bin/ during its stop hook: https://github.com/charmed-kubernetes/charm-flannel/blob/f79f180a4f8c3f19652f77596537cd9ca359b402/reactive/flannel.py#L312-L314

It shouldn't do that. Those files are installed by kubernetes-worker, not by flannel. And kubernetes-worker doesn't know it's happening, so those files never get re-installed when the new flannel subordinate is deployed.

I think if we remove those three lines, that should fix the issue.

[George Kraft (cynerva)]

PR: https://github.com/charmed-kubernetes/charm-flannel/pull/65