I'm able to deploy other charms just fine, only kafka seems to be getting stuck. It's not even getting to the install hook, so I don't think it's anything to do with the charm. I forced the log level on one of the stuck units to DEBUG (by editing agent.conf) and it looks like the resolver is stuck waiting for storage, see log snippet below.

My deployment is using mojo, which uses juju-deployer. In my deployer bundle, I specify the storage volume up front. I imagine that (though I am not familiar with the internals of juju-deployer) the storage is being added immediately after the deploy API call or command is executed.

I commented out the storage declaration in the bundle for now, and instead, I'm running a script to attach storage after the charm is successfully deployed. That seems to workaround this issue.

2019-01-07 18:25:51 DEBUG juju.worker.logger logger.go:70 reconfiguring logging from "<root>=WARNING;juju=DEBUG" to "juju=DEBUG"
2019-01-07 18:25:51 DEBUG juju.worker.leadership tracker.go:130 kafka/6 making initial claim for kafka leadership
2019-01-07 18:25:51 DEBUG juju.worker.dependency engine.go:545 "metric-spool" manifold worker started
2019-01-07 18:25:51 DEBUG juju.worker.dependency engine.go:545 "logging-config-updater" manifold worker started
2019-01-07 18:25:51 DEBUG juju.worker.dependency engine.go:545 "metric-sender" manifold worker started
2019-01-07 18:25:51 DEBUG juju.worker.dependency engine.go:545 "hook-retry-strategy" manifold worker started
2019-01-07 18:25:51 DEBUG juju.worker.logger logger.go:58 overriding logging config with override from agent.conf "juju=DEBUG"
2019-01-07 18:25:51 DEBUG juju.worker.dependency engine.go:545 "uniter" manifold worker started
2019-01-07 18:25:51 DEBUG juju.worker.proxyupdater proxyupdater.go:168 applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.25.2.109,10.25.2.110,10.25.2.111", AutoNoProxy
:""}
2019-01-07 18:25:51 DEBUG juju.worker.proxyupdater proxyupdater.go:188 saving new legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.25.2.109,10.25.2.110,10.25.2.111", AutoNoProxy:""}
2019-01-07 18:25:51 DEBUG juju.worker.proxyupdater proxyupdater.go:252 new apt proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"", AutoNoProxy:""}
2019-01-07 18:25:51 DEBUG juju.worker.meterstatus connected.go:88 got meter status change signal from watcher
2019-01-07 18:25:51 DEBUG juju.network network.go:507 no lxc bridge addresses to filter for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:543 cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
2019-01-07 18:25:51 DEBUG juju.network network.go:543 cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
2019-01-07 18:25:51 DEBUG juju.network network.go:492 including address local-cloud:10.25.2.111 for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:492 including address local-machine:127.0.0.1 for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:492 including address local-machine:::1 for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:561 addresses after filtering: [local-cloud:10.25.2.111 local-machine:127.0.0.1 local-machine:::1]
2019-01-07 18:25:51 DEBUG juju.network network.go:507 no lxc bridge addresses to filter for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:543 cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
2019-01-07 18:25:51 DEBUG juju.network network.go:543 cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
2019-01-07 18:25:51 DEBUG juju.network network.go:492 including address local-cloud:10.25.2.110 for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:492 including address local-machine:127.0.0.1 for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:492 including address local-machine:::1 for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:561 addresses after filtering: [local-cloud:10.25.2.110 local-machine:127.0.0.1 local-machine:::1]
2019-01-07 18:25:51 DEBUG juju.network network.go:507 no lxc bridge addresses to filter for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:543 cannot get "lxdbr0" addresses: route ip+net: no such network interface (ignoring)
2019-01-07 18:25:51 DEBUG juju.network network.go:543 cannot get "virbr0" addresses: route ip+net: no such network interface (ignoring)
2019-01-07 18:25:51 DEBUG juju.network network.go:492 including address local-cloud:10.25.2.109 for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:492 including address local-machine:127.0.0.1 for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:492 including address local-machine:::1 for machine
2019-01-07 18:25:51 DEBUG juju.network network.go:561 addresses after filtering: [local-cloud:10.25.2.109 local-machine:127.0.0.1 local-machine:::1]
2019-01-07 18:25:51 DEBUG juju.worker.apiaddressupdater apiaddressupdater.go:73 updating API hostPorts to [[10.25.2.111:17070 127.0.0.1:17070 [::1]:17070] [10.25.2.110:17070 127.0.0.1:17070 [::1]:17070] [10.25.2.109:17070 127.0.0.1:17070 [::1]:17070]]
2019-01-07 18:25:51 DEBUG juju.agent agent.go:581 API server address details [["10.25.2.111:17070" "127.0.0.1:17070" "[::1]:17070"] ["10.25.2.110:17070" "127.0.0.1:17070" "[::1]:17070"] ["10.25.2.109:17070" "127.0.0.1:17070" "[::1]:17070"]] written to agent config as ["10.25.2.111:17070" "10.25.2.110:17070" "10.25.2.109:17070"]
2019-01-07 18:25:51 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-kafka-6
2019-01-07 18:25:51 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.4.7-xenial-amd64
2019-01-07 18:25:51 DEBUG juju.agent.tools symlinks.go:44 jujud path /var/lib/juju/tools/2.4.7-xenial-amd64/jujud
2019-01-07 18:25:51 INFO juju.worker.leadership tracker.go:199 kafka/6 promoted to leadership of kafka
2019-01-07 18:25:51 DEBUG juju.worker.proxyupdater proxyupdater.go:168 applying in-process legacy proxy settings proxy.Settings{Http:"", Https:"", Ftp:"", NoProxy:"10.25.2.109,10.25.2.110,10.25.2.111", AutoNoProxy:""}
2019-01-07 18:25:51 DEBUG juju.worker.uniter uniter.go:580 starting juju-run listener on unix:/var/lib/juju/agents/unit-kafka-6/run.socket2019-01-07 18:25:51 INFO juju.worker.uniter uniter.go:197 unit "kafka/6" started
2019-01-07 18:25:51 DEBUG juju.worker.uniter runlistener.go:107 juju-run listener running
2019-01-07 18:25:51 INFO juju.worker.uniter uniter.go:236 hooks are retried true
2019-01-07 18:25:51 DEBUG juju.worker.uniter.remotestate watcher.go:431 got action change: [] ok=true
2019-01-07 18:25:51 DEBUG juju.worker.uniter.remotestate watcher.go:441 got relations change: ok=true
2019-01-07 18:25:51 DEBUG juju.worker.uniter.remotestate watcher.go:451 got storage change: [logs/12] ok=true
2019-01-07 18:25:51 DEBUG juju.worker.uniter.remotestate watcher.go:381 got unit change
2019-01-07 18:25:51 DEBUG juju.worker.uniter.remotestate watcher.go:364 got config change: ok=true
2019-01-07 18:25:51 DEBUG juju.worker.uniter.remotestate watcher.go:391 got application change
2019-01-07 18:25:51 DEBUG juju.worker.uniter.remotestate watcher.go:364 got config change: ok=true
2019-01-07 18:25:51 DEBUG juju.worker.uniter.remotestate watcher.go:461 got update status interval change: ok=true
2019-01-07 18:25:51 DEBUG juju.worker.uniter.remotestate watcher.go:411 got address change: ok=true
2019-01-07 18:25:51 DEBUG juju.worker.uniter.remotestate watcher.go:421 got leader settings change: ok=true
2019-01-07 18:25:51 DEBUG juju.worker.uniter.storage resolver.go:125 next hook op for storage-logs-12: {Kind:0 Life:alive Attached:false Location:}
2019-01-07 18:25:51 DEBUG juju.worker.uniter.storage resolver.go:91 still pending [storage-logs-12]