Can't purge snapd in LXD: rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/...': Function not implemented

[Paride Legovini]

In LXD containers snapd can't be purge-removed. One tool affected by this bug is autopkgtest-build-lxd, which attempts to purge several packages (including snapd) to create a base image to run autopkgtests on.

Steps to reproduce:

  lxc launch ubuntu:jammy paride-j
  lxc exec paride-j -- apt-get -y remove --purge snapd

Kinetic is also affected, I didn't test older releases. To see how this affects autopkgtest-build-lxd just install autopkgtest and run:

  autopkgtest-build-lxd ubuntu:jammy

This reminds me of LP: #1903967, but it's likely a different issue.

--- relevant log excerpt ---

[...]
Purging configuration files for snapd (2.56.2+22.04ubuntu1) ...
Stopping snap.lxd.activate.service
Stopping unit snap.lxd.activate.service
Waiting until unit snap.lxd.activate.service is stopped [attempt 1]
snap.lxd.activate.service is stopped.
Removing snap.lxd.activate.service
Stopping snap.lxd.daemon.service
Stopping unit snap.lxd.daemon.service
Waiting until unit snap.lxd.daemon.service is stopped [attempt 1]
snap.lxd.daemon.service is stopped.
Removing snap.lxd.daemon.service
Stopping snap.lxd.user-daemon.service
Stopping unit snap.lxd.user-daemon.service
Waiting until unit snap.lxd.user-daemon.service is stopped [attempt 1]
snap.lxd.user-daemon.service is stopped.
Removing snap.lxd.user-daemon.service
Stopping snap-core20-1611.mount
Stopping unit snap-core20-1611.mount
Waiting until unit snap-core20-1611.mount is stopped [attempt 1]
snap-core20-1611.mount is stopped.
Removing snap core20 and revision 1611
Removing snap-core20-1611.mount
Stopping snap-lxd-23541.mount
Stopping unit snap-lxd-23541.mount
Waiting until unit snap-lxd-23541.mount is stopped [attempt 1]
snap-lxd-23541.mount is stopped.
Removing snap lxd and revision 23541
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/cpuinfo': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/meminfo': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/stat': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/uptime': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/diskstats': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/swaps': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/loadavg': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/slabinfo': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/disable': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/above': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/time': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/rejected': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/power': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/residency': Function not implemented
[...]

[Alberto Mardegan (mardy)]

It looks like the LXD snap has mounted "proc" and "sys" over /var/snap/lxd/common/var/lib/lxcfs/{proc,sys}/ but it's not unmounting them.

In the "remove" hook, the LXD snap should unmount any mounted filesystems; not doing so can have disastrous effects if there are also user partitions mounted in there.

[Paride Legovini (paride)]

I tried purging lxd (snap remove --purge lxd) and it worked just fine, and after that an `apt purge snapd` also worked, so perhaps the umount of .../lxcfs/{proc,sys} is racy?

Uninstalling and reinstalling lxd also works:

 - [jump into *fresh* kinetic lxd container]
 - snap remove lxd
 - snap install lxd
 - apt purge snapd # works!

While refreshing lxd does not work:

 - [jump into *fresh* kinetic lxd container]
 - snap refresh lxd --edge
 - apt purge snapd # FAIL

[Paride Legovini (paride)]

This was retriaged by LXD tream as a snapd bug, see:

https://github.com/lxc/lxd-pkg-snap/issues/106

and in particular

https://github.com/lxc/lxd-pkg-snap/issues/106#issuecomment-1252549005

[Paride Legovini (paride)]

This is still reproducible at least on Focal and Jammy. The link to the (not really) Fix Released upstream bug is dead.