kubernetes-master is blocked due to Stopped services: kube-controller-manager

[Konstantinos Kaskavelis]

We have two test runs where kubernetes-master is in a blocked state, due to kube-controller-manager service being stopped.

Some related errors:

./8/baremetal/var/log/syslog:May 23 16:16:06 ip-172-31-43-80 kube-controller-manager.daemon[20494]: W0523 16:16:06.312004 20494 client_config.go:622] error creating inClusterConfig, falling back to default config: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
./8/baremetal/var/log/syslog:May 23 16:36:01 ip-172-31-43-80 kube-controller-manager.daemon[136149]: E0523 16:36:01.882232 136149 plugins.go:613] "Error initializing dynamic plugin prober" err="error (re-)creating driver directory: mkdir /usr/libexec: read-only file system"

Indicative run:

https://solutions.qa.canonical.com/testruns/testRun/11ca7dd7-599b-4242-ae5e-e93cf3992ec4

Artifacts:

https://oil-jenkins.canonical.com/artifacts/11ca7dd7-599b-4242-ae5e-e93cf3992ec4/index.html

[Chris Johnston (cjohnston)]

I think this is very similar to LP#1943986. I've seen both happen throughout the releases/charm versions.