From the Juju side, we can see that we have code that calls AttachFilesystems, that had the contract if source and target were stable, that it should be a 'noop'. As far as we can tell, the rootfs code has always been wrong with its check for 'is it already mounted'. The assumption is that if you do: mount --bind $SOURCE $TARGET that you would have df --output=source $TARGET return (in the second line) $SOURCE eg: $ mount --bind /var/lib/juju/data /srv/data would then have: $ df --output=source /srv/data Filesystem /var/lib/juju/data In practice, what you actually get is: Filesystem /dev/sda1 That is a bad assumption on our part, but has been a bad assumption for all of the life of juju. The other issue that caused us to notice this is that there was a change between juju 2.9.30 and 2.9.44 (I have to dig to find the exact patch, but it was by Simon Richardson). That patch caused us to be more persistent with mounts, because of some other issues (specifically the order that volumes/blocks show up was not guaranteed, so we needed to try again). That bug fix is causing us to wake up periodically (current period unknown) and essentially call 'mount --bind /source/ /target' over and over again. To get a workaround, we just need to interrupt something. Either 1) Put a `df` into the $PATH of `jujud` such that `df --output source $TARGET` for a bind mount can return $SOURCE rather than SOURCE_DEVICE 2) Put a `mount` into $PATH of `jujud` so that `mount --bind $SOURCE $TARGET` is a no-op when source is already bind mounted at $TARGET. For (1) we still haven't found what command could be run for bind mounts. Because reading /proc/mounts and 'df' always give the device, and we need to figure out the relative mount location. https://unix.stackexchange.com/questions/18048/list-only-bind-mounts/346444#346444 Seems to say that the information might be available from '/proc/self/mountinfo' From what I can tell, the issue is that you get 'mount the filesystem from this block device, taking into account this relative path' but you never have a way to unfold it unambiguously. We did a test where '/home' was a mounted filesystem '/dev/nvme0n1p3' we then did 'mkdir a; mkdir b; mount --bind a b' and we could get: 198 29 259:3 / /home rw,relatime shared:117 - ext4 /dev/nvme0n1p3 rw ... 6314 198 259:3 /ian/a /home/ian/b rw,relatime shared:117 - ext4 /dev/nvme0n1p3 rw 6443 6314 259:3 /ian/a /home/ian/b rw,relatime shared:117 - ext4 /dev/nvme0n1p3 rw 6581 198 259:3 /ian/a /home/ian/a rw,relatime shared:117 - ext4 /dev/nvme0n1p3 rw Which says that '/dev/nvme0n1p3' is mounted at '/home' from the '/' of its filesystem, and then mounted again at '/home/ian/b' with '/ian/a' as its relative filesystem. There are a few possibilities to give some answers, in increasing complexity but also increasing correctness. 1) 'mount --bind /home/ian/a /home/ian/b' looks at /proc/self/mountinfo and sees that there exists an exact entry for /home/ian/b already, and just refuses to do anything 2) 'mount --bind ...' looks at the same line, and sees that there is a relative directory mount (/ian/a) and a non-relative mount ('/'). It then tacks on '/ian/a' to '/home' to realize that '/home/ian/a' is already mounted to '/home/ian/b' and exits. 3) Note though, that you might need to recurse, and there are ways that /home/ian/a/ian/a could be the desired source, and that would actually result in the same content at '/home/ian/b'. I don't think we care about this case enough to worry about it. Ultimately, the juju code needs a) To do the recursion based on '/proc/self/mountinfo' so that it can determine that '/var/lib/juju/...' is already mounted at '/srv/data' b) Fix the bug that is waking us up over and over to try and mount it again. If we only did (b) without (a) then whenever an agent got restarted, it would cause more mounts. This would still be far better than every X seconds, but still result in multiple mounts of the same thing. If we only did (a) we would end up with a correct filesystem each time, but we would be inefficient and trying over and over again.