Comment 11 for bug 1871189

This really shows what's wrong, from the part when we are explaining:

unmount (none /usr/libexec none x-snapd.kind=symlink,x-snapd.symlink=/var/snap/microk8s/common/usr/libexec,x-snapd.origin=layout 0 0)

This really means: remove the symlink at /usr/libexec

unmount (/snap/microk8s/x1/var/lib/snapd/lib/gl/libEGL_nvidia.so.0 /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.0 none bind,rw,x-snapd.kind=file,x-snapd.origin

This means: umount the bind-mount at /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.0 and unlink the placeholder file we created.

What really happens:

remove "/usr/libexec" (error: <nil>)

This is as I explained above.

umount "/usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.0" UMOUNT_NOFOLLOW (error: <nil>)

This also is as I explained above:

remove "/usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.0" (error: remove /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.0: device or resource busy)

This fails, we also have a bigger writable mimic for *all of* /usr! In other words /usr is still a mount point.

This is very surprising because we have robust-mount-namespace-updates enabled and they were implemented to handle this situation *exactly* so what gives?

Well, this is a *file* bind mount, and that case is not accounted for in the code. https://github.com/snapcore/snapd/blob/master/cmd/snap-update-ns/change.go#L450 lacks a check for kind=="file".