Can't purge snapd in LXD: rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/...': Function not implemented

[Paride Legovini]

In LXD containers snapd can't be purge-removed. One tool affected by this bug is autopkgtest-build-lxd, which attempts to purge several packages (including snapd) to create a base image to run autopkgtests on.

Steps to reproduce:

  lxc launch ubuntu:jammy paride-j
  lxc exec paride-j -- apt-get -y remove --purge snapd

Kinetic is also affected, I didn't test older releases. To see how this affects autopkgtest-build-lxd just install autopkgtest and run:

  autopkgtest-build-lxd ubuntu:jammy

This reminds me of LP: #1903967, but it's likely a different issue.

--- relevant log excerpt ---

[...]
Purging configuration files for snapd (2.56.2+22.04ubuntu1) ...
Stopping snap.lxd.activate.service
Stopping unit snap.lxd.activate.service
Waiting until unit snap.lxd.activate.service is stopped [attempt 1]
snap.lxd.activate.service is stopped.
Removing snap.lxd.activate.service
Stopping snap.lxd.daemon.service
Stopping unit snap.lxd.daemon.service
Waiting until unit snap.lxd.daemon.service is stopped [attempt 1]
snap.lxd.daemon.service is stopped.
Removing snap.lxd.daemon.service
Stopping snap.lxd.user-daemon.service
Stopping unit snap.lxd.user-daemon.service
Waiting until unit snap.lxd.user-daemon.service is stopped [attempt 1]
snap.lxd.user-daemon.service is stopped.
Removing snap.lxd.user-daemon.service
Stopping snap-core20-1611.mount
Stopping unit snap-core20-1611.mount
Waiting until unit snap-core20-1611.mount is stopped [attempt 1]
snap-core20-1611.mount is stopped.
Removing snap core20 and revision 1611
Removing snap-core20-1611.mount
Stopping snap-lxd-23541.mount
Stopping unit snap-lxd-23541.mount
Waiting until unit snap-lxd-23541.mount is stopped [attempt 1]
snap-lxd-23541.mount is stopped.
Removing snap lxd and revision 23541
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/cpuinfo': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/meminfo': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/stat': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/uptime': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/diskstats': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/swaps': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/loadavg': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/proc/slabinfo': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/disable': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/above': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/time': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/rejected': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/power': Function not implemented
rm: cannot remove '/var/snap/lxd/common/var/lib/lxcfs/sys/devices/system/cpu/cpu0/cpuidle/state5/residency': Function not implemented
[...]

[Alberto Mardegan (mardy)]

It looks like the LXD snap has mounted "proc" and "sys" over /var/snap/lxd/common/var/lib/lxcfs/{proc,sys}/ but it's not unmounting them.

In the "remove" hook, the LXD snap should unmount any mounted filesystems; not doing so can have disastrous effects if there are also user partitions mounted in there.

[Paride Legovini (paride)]

I tried purging lxd (snap remove --purge lxd) and it worked just fine, and after that an `apt purge snapd` also worked, so perhaps the umount of .../lxcfs/{proc,sys} is racy?

Uninstalling and reinstalling lxd also works:

 - [jump into *fresh* kinetic lxd container]
 - snap remove lxd
 - snap install lxd
 - apt purge snapd # works!

While refreshing lxd does not work:

 - [jump into *fresh* kinetic lxd container]
 - snap refresh lxd --edge
 - apt purge snapd # FAIL

[Paride Legovini (paride)]

This was retriaged by LXD tream as a snapd bug, see:

https://github.com/lxc/lxd-pkg-snap/issues/106

and in particular

https://github.com/lxc/lxd-pkg-snap/issues/106#issuecomment-1252549005

[Paride Legovini (paride)]

This is still reproducible at least on Focal and Jammy. The link to the (not really) Fix Released upstream bug is dead.

[Paride Legovini (paride)]

Stumbled on this again today, and I still believe it's a snapd bug and not a lxd bug: snapd should be uninstallable cleanly no matter of what individual snaps do.

[Ernest Lotter (ernestl)]

Hi, had a first look at this,

THE LINE OF CODE THAT CAUSES THE FAILURE:
-------------------------------------------------------------------------
$ ls /var/lib/dpkg/info/snapd.*
> snapd.prerm snapd.postrm ...

See: https://github.com/canonical/snapd/blob/master/packaging/ubuntu-16.04/snapd.postrm

As per log: "Removing snap lxd and revision 23541" =>
https://github.com/canonical/snapd/blob/master/packaging/ubuntu-16.04/snapd.postrm#L74

which is within the "purge" condition https://github.com/canonical/snapd/blob/master/packaging/ubuntu-16.04/snapd.postrm#L30

This is the line that fails:
rm -rf --one-file-system "/var/snap/$snap/common"

https://github.com/canonical/snapd/blob/master/packaging/ubuntu-16.04/snapd.postrm#L90

WITH SNAPD, LXD INSTALLED
----------------------------------------------

After installing lxd, and running lxd init, a link
/var/snap/lxd/common/var/lib/lxcfs -> /var/snap/lxd/common/shmounts/lxcfs
but /var/snap/lxd/common/shmounts/lxcfs does not exist.

See: https://paste.ubuntu.com/p/Jg4Z2dbfgd/

So its seems the issue is, that the postrm script tries to remove something that it did not add.

QUESTIONS
-------------------

1) @Paride: for your test, does the symlink actaully point to anything, otherwise is not representative of the issue.

2) @LXD team: What in lxd is responsible for populating "/var/snap/lxd/common/shmounts/lxcfs"?

3) @LXD team: Does lxd snap use a remove hook?
   https://snapcraft.io/docs/supported-snap-hooks#heading--remove

[Paride Legovini (paride)]

The answer is different depending on the system: /var/snap/lxd/common/var/lib/lxcfs is a broken symlink on Noble, but not on Focal. And: snapd purge+remove works on Noble, but not on Focal. See below for reproducers.

## NOBLE ##

$ lxc launch ubuntu:noble paride-n
$ lxc exec paride-n bash
root@paride-n:~# lxd init --auto
Installing LXD snap, please be patient.

root@paride-n:~# ls -l /var/snap/lxd/common/var/lib/lxcfs
lrwxrwxrwx 1 root root 35 Dec 3 13:43 /var/snap/lxd/common/var/lib/lxcfs -> /var/snap/lxd/common/shmounts/lxcfs

root@paride-n:~# ls -l /var/snap/lxd/common/shmounts/lxcfs
ls: cannot access '/var/snap/lxd/common/shmounts/lxcfs': No such file or directory

root@paride-n:~# apt-get --yes --purge remove snapd
[...]
root@paride-n:~# echo $?
0

^^ Success

## FOCAL ##

$ lxc launch ubuntu:focal paride-f
$lxc exec paride-f bash
root@paride-f:~# lxd init --auto

root@paride-f:~# ls -l /var/snap/lxd/common/var/lib/lxcfs
total 0
dr-xr-xr-x 2 nobody nogroup 0 Dec 3 13:53 proc
dr-xr-xr-x 2 nobody nogroup 0 Dec 3 13:53 sys

root@paride-f:~# apt-get --yes --purge remove snapd
[...]
Removing snap lxd and revision 29619
rm: skipping '/var/snap/lxd/common/var/lib/lxcfs', since it's on a different device
dpkg: error processing package snapd (--purge):
 installed snapd package post-removal script subprocess returned error exit status 1
dmesg: read kernel buffer failed: Operation not permitted
                                                         Errors were encountered while processing:
 snapd
E: Sub-process /usr/bin/dpkg returned an error code (1)
root@paride-f:~# echo $?
100

^^ Fail