hook failed: "config-changed"

[yen]

Kubenetes master entering error stage.

```
kubernetes-master/0* error idle 0 172.29.100.185 6443/tcp hook failed: "config-changed"
```

I tried followed on this tutorial, but it couldn't really get the hook to run.
https://docs.jujucharms.com/1.25/en/authors-hook-debug

Can anyone explain to me where the 'charms' module is and how to load it for python3?
```
root@kubernetesBM:~# /var/lib/juju/agents/unit-kubernetes-master-0/charm/hooks/config-changed
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kubernetes-master-0/charm/hooks/config-changed", line 7, in <module>
    from charms.layer import basic # noqa
ModuleNotFoundError: No module named 'charms'

```

[Mike Wilson (knobby)]

The first place I would start is the logs. `juju debug-log --replay` is useful here to try and get a callstack for the failure. Another option is to use the cdk field agent, which will grab lots of logs and debug information and tar it up.

[yen (antigenius0910)]

below is the result from "juju debug-log --replay --include kubernetes-master/0"

```
unit-kubernetes-master-0: 14:38:15 DEBUG worker.uniter.jujuc running hook tool "add-metric"
unit-kubernetes-master-0: 14:40:16 DEBUG juju.worker.uniter.remotestate update status timer triggered
unit-kubernetes-master-0: 14:42:26 DEBUG juju.worker.meterstatus got meter status change signal from watcher
unit-kubernetes-master-0: 14:43:03 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 14:43:03 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 14:43:03 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 14:43:03 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 14:43:03 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 14:43:03 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 14:43:03 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 14:43:03 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 14:43:04 DEBUG worker.uniter.jujuc running hook tool "add-metric"
unit-kubernetes-master-0: 14:44:04 INFO juju.worker.upgrader desired agent binary version: 2.4.7
unit-kubernetes-master-0: 14:44:53 DEBUG juju.worker.uniter.remotestate update status timer triggered

```

[yen (antigenius0910)]

Sorry here is a more detail logs.

```
unit-kubernetes-master-0: 09:30:03 INFO juju.worker.uniter unit "kubernetes-master/0" started
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter juju-run listener running
unit-kubernetes-master-0: 09:30:03 INFO juju.worker.uniter hooks are retried true
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter.remotestate got config change: ok=true
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter.remotestate got action change: [] ok=true
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter.remotestate got relations change: ok=true
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter.remotestate got update status interval change: ok=true
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter.remotestate got address change: ok=true
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter.remotestate got unit change
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter.remotestate got storage change: [] ok=true
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter.remotestate got config change: ok=true
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter.remotestate got application change
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter.remotestate got leader settings change: ok=true
unit-kubernetes-master-0: 09:30:03 INFO juju.worker.uniter awaiting error resolution for "config-changed" hook
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.uniter [AGENT-STATUS] error: hook failed: "config-changed"
unit-kubernetes-master-0: 09:30:03 DEBUG juju.worker.dependency "metric-collect" manifold worker started
unit-kubernetes-master-0: 09:30:04 DEBUG juju.worker.uniter.remotestate got unit change
unit-kubernetes-master-0: 09:30:04 DEBUG juju.worker.uniter.operation running operation run config-changed hook
unit-kubernetes-master-0: 09:30:04 DEBUG juju.machinelock acquire machine lock for uniter (run config-changed hook)
unit-kubernetes-master-0: 09:30:24 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 09:30:24 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 09:30:24 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 09:30:24 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 09:30:24 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 09:30:24 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was refused - did you specify the right host or port?
unit-kubernetes-master-0: 09:30:24 DEBUG unit.unit-kubernetes-master-0.collect-metrics The connection to the server localhost:8080 was...

[Cory Johns (johnsca)]

That file is included in the charm, so it's definitely available, so it seems like the issue is with the sys path or import process, but I can't for the life of me think of anything that would make it to that point but fail there. (For reference, the code being executed can be found here: https://github.com/juju-solutions/layer-basic/blob/master/hooks/hook.template#L7)

The docs link referenced is for a very old version of Juju. The current version of that page would be https://docs.jujucharms.com/stable/developer-debugging but I would also like to confirm what version of Juju you are using? I don't believe that we support Juju 1.x for CDK.

Other than that, you might try using `juju debug-hooks` and add a pdb breakpoint before that line and stepping through to see why the import is failing.

[yen (antigenius0910)]

Thanks @cory

However I have tear down the cluster. Let me see if I could reproduce it once again. thank you!
```
root@MAAS:~# juju --version
2.5.1-xenial-amd64
```