Comment 27 for bug 1551854

So the cause of all this was /sys/kernel/debug/tracing which is a weird auto-mounted kernel path. That is, the sole action of listing that directory will cause it to get mounted for you by the kernel.

That means that any number of thing could accidentally cause it to mount.

Once it's mounted, the kernel considers /sys/kernel/debug to have a directory that's hidden through overmounting and so will not allow unprivileged users to bind-mount the underlying directory, which means /sys/kernel/debug isn't mounted in the container and causes mountall to fail.

There are quite a few ways to fix this.
The best would be to not have the kernel do this weird auto-mount thing, sadly fixing that would be a userspace regression so as weird and inconsistent (trying to remain polite) as the current design is, reverting it is unlieky.

As mentioned before, we could also fix mountall not to be so picky and not die when mounts that it knows as "optional" fail to mount. Unfortunately there are a lot of images out there using mountall, so we can't really rely on being able to push a fix to all of them.

A third option and the one we'll be using for now is to have LXD recursively bind-mount paths, therefore not exposing the container to any more information than would be visible on the host and so avoiding the kernel security feature entirely.

The fix in LXD is a one character change (bind to rbind) and I've sent a pull request upstream to do just that.

I'd just like to stress that I think the kernel behavior here is absolutely ridiculous, we have a security feature which triggers when it shouldn't (the path doesn't exist so can't be "hidden") combined with a crazy feature that's been added to be "user friendly" and causes automatic mounting of a filesystem by simply accessing a path inside another filesystem. The combination of both results in this bug... But the fact is, it's way easier and faster for us to workaround this in LXD than to try and fix the source of the problem...