flannel juju charm erroring due to FileNotFounderror

[Ed Stewart]

Our K8S deployments using Juju are erroring on the install step of the flannel charm. This is using flannel charm revision 351 on bionic:

Model Controller Cloud/Region Version SLA Timestamp
mgmt-k8s juju_main cloud/RegionOne 2.5.0 unsupported 15:00:52Z

App Version Status Scale Charm Store Rev OS Notes
easyrsa 3.0.1 active 1 easyrsa jujucharms 195 ubuntu
etcd 3.2.10 active 1 etcd jujucharms 378 ubuntu
flannel error 3 flannel jujucharms 351 ubuntu
kubernetes-master 1.13.2 waiting 1 kubernetes-master jujucharms 542 ubuntu exposed
kubernetes-worker 1.13.2 waiting 2 kubernetes-worker jujucharms 398 ubuntu exposed
openstack-integrator active 1 openstack-integrator jujucharms 8 ubuntu

Unit Workload Agent Machine Public address Ports Message
easyrsa/0* active idle 0/lxd/0 253.12.75.208 Certificate Authority connected.
etcd/0* active idle 0 172.16.20.52 2379/tcp Healthy with 1 known peer
kubernetes-master/0* waiting idle 0 172.16.20.52 Waiting for CNI plugins to become available
  flannel/1 error idle 172.16.20.52 hook failed: "install"
kubernetes-worker/0* waiting idle 1 172.16.20.65 Waiting for cluster DNS.
  flannel/0* error idle 172.16.20.65 hook failed: "install"
kubernetes-worker/1 waiting idle 2 172.16.20.51 Waiting for cluster DNS.
  flannel/2 error idle 172.16.20.51 hook failed: "install"
openstack-integrator/0* active idle 0/lxd/1 253.12.68.47 ready

Machine State DNS Inst id Series AZ Message
0 started 172.16.20.52 b13a1ac2-2f7c-4c6f-ad4d-54e7bc94e6b9 bionic nova ACTIVE
0/lxd/0 started 253.12.75.208 juju-4fec8a-0-lxd-0 bionic nova Container started
0/lxd/1 started 253.12.68.47 juju-4fec8a-0-lxd-1 bionic nova Container started
1 started 172.16.20.65 adcfc59e-767b-4780-addd-729d7a67025a bionic nova ACTIVE
2 started 172.16.20.51 842b0c6a-4786-47ae-aee6-5f93fe9e37ab bionic nova ACTIVE

Note all flannel units are in error.

Looking at the flannel logs, we get this:

2019-02-04 14:17:52 ERROR juju-log Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-flannel-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 73, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-flannel-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-flannel-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-flannel-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-flannel-0/charm/reactive/flannel.py", line 224, in set_flannel_version
    version = check_output(split(cmd), stderr=STDOUT).decode('utf-8')
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 403, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.6/subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.6/subprocess.py", line 1344, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'flanneld': 'flanneld'

Oddly, if I ssh into the unit, I can find flanneld here:

root@juju-4fec8a-mgmt-k8s-0:/# find . -name flanneld
./usr/local/bin/flanneld
./var/lib/juju/agents/unit-flannel-1/charm/files/flannel/flanneld