So I set up an lxd container with limited disk space with: $ lxc storage create small zfs size=2GB $ lxc launch juju/bionic/amd64 biotest --storage=small $ lxc exec bash biotest $$ su -l ubuntu $$ ssh-import-id jameinel $$ exit $ lxc list $ juju add-machine ssh:ubuntu@${IP} $ cd $GOPATH/github.com/juju/juju/acceptance_tests/repository/trusty $ juju deploy ./dummy-source --to 0 $ lxc exec bash biotest $$ su -l ubuntu $$ dd if=/dev/urandom of=$HOME/fillmeup bs=1M Note that I had to use urandom instead of /dev/zero because I was using ZFS which seemed to happily allow me to create a 14GB file full of zeros on a 2GB partition. Once the disk was full, I could see on the controller these failures: 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:47 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-dummy-source-0 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:47 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.5.5.1-bionic-amd64 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:47 ERROR juju.worker.dependency engine.go:636 "uniter" manifold worker returned unexpected error: failed to initialize uniter for "unit-dummy-source-0": creating juju run listener: mkdir /var/lib/juju/agents/unit-dummy-source-0/477599416: no space left on device 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:50 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-dummy-source-0 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:50 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.5.5.1-bionic-amd64 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:50 ERROR juju.worker.dependency engine.go:636 "uniter" manifold worker returned unexpected error: failed to initialize uniter for "unit-dummy-source-0": creating juju run listener: mkdir /var/lib/juju/agents/unit-dummy-source-0/879169719: no space left on device 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:52 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-dummy-source-0 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:52 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.5.5.1-bionic-amd64 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:52 ERROR juju.worker.dependency engine.go:636 "uniter" manifold worker returned unexpected error: failed to initialize uniter for "unit-dummy-source-0": creating juju run listener: mkdir /var/lib/juju/agents/unit-dummy-source-0/761994922: no space left on device 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:56 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-dummy-source-0 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:56 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.5.5.1-bionic-amd64 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:56 ERROR juju.worker.dependency engine.go:636 "uniter" manifold worker returned unexpected error: failed to initialize uniter for "unit-dummy-source-0": creating juju run listener: mkdir /var/lib/juju/agents/unit-dummy-source-0/142897153: no space left on device 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:59 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-dummy-source-0 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:59 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.5.5.1-bionic-amd64 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:10:59 ERROR juju.worker.dependency engine.go:636 "uniter" manifold worker returned unexpected error: failed to initialize uniter for "unit-dummy-source-0": creating juju run listener: mkdir /var/lib/juju/agents/unit-dummy-source-0/982682988: no space left on device 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:11:02 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-dummy-source-0 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:11:02 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.5.5.1-bionic-amd64 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:11:02 ERROR juju.worker.dependency engine.go:636 "uniter" manifold worker returned unexpected error: failed to initialize uniter for "unit-dummy-source-0": creating juju run listener: mkdir /var/lib/juju/agents/unit-dummy-source-0/774528987: no space left on device 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:11:05 INFO juju.agent.tools symlinks.go:20 ensure jujuc symlinks in /var/lib/juju/tools/unit-dummy-source-0 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:11:05 INFO juju.agent.tools symlinks.go:40 was a symlink, now looking at /var/lib/juju/tools/2.5.5.1-bionic-amd64 45ba77ba-73bc-45c2-827a-a3fffde9d2b1: unit-dummy-source-0 2019-05-09 12:11:05 ERROR juju.worker.dependency engine.go:636 "uniter" manifold worker returned unexpected error: failed to initialize uniter for "unit-dummy-source-0": creating juju run listener: mkdir /var/lib/juju/agents/unit-dummy-source-0/082283390: no space left on device At this point it definitely is triggering every 3s which means it a) Is waiting between attempts b) Isn't triggering the exponential backoff c) Is clearly using 2.5.5 for everything. I can try with juju-2.4.7, but according to the code, exponential backoff was added in 2.4.5, so it should have been there anyway. It might be something special with metric-sender, as that may be using different code paths. (I don't know that it is, but I know metrics gathering doesn't use the hook execution lock so that a long-running hook doesn't prevent metrics from being updated.)