/dev/loopX devices left around for removed snap revisions

[Alberto Donato]

On my 19.04 box, I noticed the Gnome file manager at times starts showing extra drivers, if I click on those they get mounted and they turn out to be snaps filesystems.

Looking at gnome VFS mounts I see:

$ gio mount -li
Drive(0): SanDisk SD8TB8U512G1001
  Type: GProxyDrive (GProxyVolumeMonitorUDisks2)
  ids:
   unix-device: '/dev/sda'
  themed icons: [drive-harddisk-solidstate] [drive-harddisk] [drive] [drive-harddisk-solidstate-symbolic] [drive-harddisk-symbolic] [drive-symbolic]
  symbolic themed icons: [drive-harddisk-solidstate-symbolic] [drive-harddisk-symbolic] [drive-symbolic] [drive-harddisk-solidstate] [drive-harddisk] [drive]
  is_removable=0
  is_media_removable=0
  has_media=1
  is_media_check_automatic=1
  can_poll_for_media=0
  can_eject=0
  can_start=0
  can_stop=0
  start_stop_type=shutdown
  sort_key=00coldplug/00fixed/sd____a
Volume(0): 57 MB Volume
  Type: GProxyVolume (GProxyVolumeMonitorUDisks2)
  ids:
   class: 'loop'
   unix-device: '/dev/loop47'
  themed icons: [drive-removable-media] [drive-removable] [drive] [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic]
  symbolic themed icons: [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic] [drive-removable-media] [drive-removable] [drive]
  can_mount=1
  can_eject=0
  should_automount=0
  sort_key=gvfs.time_detected_usec.1565684753456472
Volume(1): 57 MB Volume
  Type: GProxyVolume (GProxyVolumeMonitorUDisks2)
  ids:
   class: 'loop'
   unix-device: '/dev/loop8'
  themed icons: [drive-removable-media] [drive-removable] [drive] [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic]
  symbolic themed icons: [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic] [drive-removable-media] [drive-removable] [drive]
  can_mount=1
  can_eject=0
  should_automount=0
  sort_key=gvfs.time_detected_usec.1565864986223896
Volume(2): 57 MB Volume
  Type: GProxyVolume (GProxyVolumeMonitorUDisks2)
  ids:
   class: 'loop'
   unix-device: '/dev/loop9'
  themed icons: [drive-removable-media] [drive-removable] [drive] [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic]
  symbolic themed icons: [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic] [drive-removable-media] [drive-removable] [drive]
  can_mount=1
  can_eject=0
  should_automount=0
  sort_key=gvfs.time_detected_usec.1565956723163136
Volume(3): 57 MB Volume
  Type: GProxyVolume (GProxyVolumeMonitorUDisks2)
  ids:
   class: 'loop'
   unix-device: '/dev/loop51'
  themed icons: [drive-removable-media] [drive-removable] [drive] [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic]
  symbolic themed icons: [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic] [drive-removable-media] [drive-removable] [drive]
  can_mount=1
  can_eject=0
  should_automount=0
  sort_key=gvfs.time_detected_usec.1566122858757365
Volume(4): 57 MB Volume
  Type: GProxyVolume (GProxyVolumeMonitorUDisks2)
  ids:
   class: 'loop'
   unix-device: '/dev/loop3'
  themed icons: [drive-removable-media] [drive-removable] [drive] [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic]
  symbolic themed icons: [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic] [drive-removable-media] [drive-removable] [drive]
  can_mount=1
  can_eject=0
  should_automount=0
  sort_key=gvfs.time_detected_usec.1566224621352304
Volume(5): 99 MB Volume
  Type: GProxyVolume (GProxyVolumeMonitorUDisks2)
  ids:
   class: 'loop'
   unix-device: '/dev/loop14'
  themed icons: [drive-removable-media] [drive-removable] [drive] [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic]
  symbolic themed icons: [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic] [drive-removable-media] [drive-removable] [drive]
  can_mount=1
  can_eject=0
  should_automount=0
  sort_key=gvfs.time_detected_usec.1566324204041632
Volume(6): 57 MB Volume
  Type: GProxyVolume (GProxyVolumeMonitorUDisks2)
  ids:
   class: 'loop'
   unix-device: '/dev/loop60'
  themed icons: [drive-removable-media] [drive-removable] [drive] [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic]
  symbolic themed icons: [drive-removable-media-symbolic] [drive-removable-symbolic] [drive-symbolic] [drive-removable-media] [drive-removable] [drive]
  can_mount=1
  can_eject=0
  should_automount=0
  sort_key=gvfs.time_detected_usec.1566499678555259

Those loop devices correspond to old snap revisions that have been removed (by snapd itself), but it seems the loopback devices are left around:

$ losetup -l | grep deleted
/dev/loop47 0 0 1 1 /var/lib/snapd/snaps/lxd_11353.snap (deleted) 0 512
/dev/loop8 0 0 1 1 /var/lib/snapd/snaps/lxd_11437.snap (deleted) 0 512
/dev/loop51 0 0 1 1 /var/lib/snapd/snaps/lxd_11633.snap (deleted) 0 512
/dev/loop9 0 0 1 1 /var/lib/snapd/snaps/lxd_11595.snap (deleted) 0 512
/dev/loop14 0 0 1 1 /var/lib/snapd/snaps/telegram-desktop_836.snap (deleted) 0 512
/dev/loop60 0 0 1 1 /var/lib/snapd/snaps/lxd_11672.snap (deleted) 0 512
/dev/loop3 0 0 1 1 /var/lib/snapd/snaps/lxd_11643.snap (deleted) 0 512

Note that those snap files don't exist anymore under /var/lib/snapd/snaps.

A reboot fixes the issue, but it starts appearing again at the next udpate of a snap.

Snapd versions are the following:

$ snap version
snap 2.40+19.04
snapd 2.40+19.04
series 16
ubuntu 19.04
kernel 5.0.0-21-generic

ii snapd 2.40+19.04 amd64 Daemon and tooling that enable snap packages

[Alberto Donato (ack)]

FTR, "losetup -d" on those loop devices doesn't seem to work

[Alberto Donato (ack)]

As a further test, I tried removing the telegram-desktop snap entirely, now all past revisions are stuck as "deleted":

$ losetup | grep deleted
/dev/loop47 0 0 1 1 /var/lib/snapd/snaps/lxd_11353.snap (deleted) 0 512
/dev/loop8 0 0 1 1 /var/lib/snapd/snaps/lxd_11437.snap (deleted) 0 512
/dev/loop63 0 0 1 1 /var/lib/snapd/snaps/telegram-desktop_877.snap (deleted) 0 512
/dev/loop51 0 0 1 1 /var/lib/snapd/snaps/lxd_11633.snap (deleted) 0 512
/dev/loop38 0 0 1 1 /var/lib/snapd/snaps/telegram-desktop_891.snap (deleted) 0 512
/dev/loop9 0 0 1 1 /var/lib/snapd/snaps/lxd_11595.snap (deleted) 0 512
/dev/loop14 0 0 1 1 /var/lib/snapd/snaps/telegram-desktop_836.snap (deleted) 0 512
/dev/loop60 0 0 1 1 /var/lib/snapd/snaps/lxd_11672.snap (deleted) 0 512
/dev/loop3 0 0 1 1 /var/lib/snapd/snaps/lxd_11643.snap (deleted) 0 512

[Paweł Stołowski (stolowski)]

That's interesting; I had one such loop device (for an old revision of core that was no longer present in the system) on my 18.04 test box. I played a bit with this on 19.04, installed and removed a bunch of snaps, but couldn't reproduce the problem with snapd 2.40 on 19.04, apparently there are some factors that make it happen over longer periods of time.

[Zygmunt Krynicki (zyga)]

ack: are you seeing any leftover LXD processes or leftover telegram processes?

[Zygmunt Krynicki (zyga)]

I'm assigning this to myself. There are identical issues detected by my mount leak detector while debugging the snapd test suite. I just chatted with other snapd developers and some of them were affected as well. I will write a quick test and see what I find.

[Alberto Donato (ack)]

zyga: no, there are no leftover processes.

If it matters, I'm probably seeing this a lot on my laptop since I always suspend and rarely reboot.
Yesterday before rebooting I had about 10 leftover loop mounts.

[Zygmunt Krynicki (zyga)]

Yes, this is certainly a factor. Rebooting will automatically clear those out. We're working through theories that could explain how we leak those now. I'll update the bug as we learn more.

[Zygmunt Krynicki (zyga)]

We've reproduced this bug on a Ubuntu 16.04 machine and we have some ideas on why it might happen but we are not any closer towards having a solution.

[Alberto Donato (ack)]

So, I thought this bug had disappeared, as I haven't seen it for a long time.

After upgrading to focal, now I see this happening again.
I have two drives in Gnome file manager, which match these:

$ losetup | grep del
/dev/loop13 0 0 1 1 /var/lib/snapd/snaps/lxd_13717.snap (deleted) 0 512
/dev/loop31 0 0 1 1 /var/lib/snapd/snaps/lxd_13704.snap (deleted) 0 512

These files no longer exist.

As before, `losetup -d` is not effective.

[Alberto Donato (ack)]

FTR:

$ snap version
snap 2.44~pre1+20.04
snapd 2.44~pre1+20.04
series 16
ubuntu 20.04
kernel 5.4.0-14-generic

[Zygmunt Krynicki (zyga)]

Are either /dev/loop13 or /dev/loop31 still mounted anywhere?

[Alberto Donato (ack)]

They're not, but if I click on one of the drives in the file manager, then they do get mounted:

$ mount | grep del
/var/lib/snapd/snaps/lxd_13704.snap (deleted) on /media/ack/disk type squashfs (ro,nosuid,nodev,relatime,uhelper=udisks2)

[Zygmunt Krynicki (zyga)]

I'm demoting to low.

While we know those exist it is beyond our current range, specifically, it is something libmount is responsible for. I think we would have to go on a journey to understand why they are not released automatically like they should.

[Andre Ruiz (andre-ruiz)]

This is happening on an up-to-date (as of 2020 Apr/30) Eoan with
$ snap --version
snap 2.44.3
snapd 2.44.3
series 16
ubuntu 19.10
kernel 5.3.0-46-generic

>>> 15:55:37 andre@thinkpad ~
$ losetup -l
NAME SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE DIO LOG-SEC
/dev/loop1 0 0 1 1 /var/lib/snapd/snaps/core_8935.snap (deleted) 0 512
/dev/loop29 0 0 1 1 /var/lib/snapd/snaps/lxd_14729.snap (deleted) 0 512
/dev/loop19 0 0 1 1 /var/lib/snapd/snaps/gtk-common-themes_1506.snap 0 512
/dev/loop27 0 0 1 1 /var/lib/snapd/snaps/lxd_14709.snap (deleted) 0 512
/dev/loop8 0 0 1 1 /var/lib/snapd/snaps/core18_1705.snap 0 512
/dev/loop25 0 0 1 1 /var/lib/snapd/snaps/lxd_14663.snap (deleted) 0 512
/dev/loop15 0 0 1 1 /var/lib/snapd/snaps/lxd_14555.snap (deleted) 0 512
/dev/loop6 0 0 1 1 /var/lib/snapd/snaps/openstackclients_38.snap 0 512
/dev/loop33 0 0 1 1 /var/lib/snapd/snaps/gnome-3-34-1804_27.snap 0 512
/dev/loop23 0 0 1 1 /var/lib/snapd/snaps/lxd_14639.snap (deleted) 0 512
/dev/loop4 0 0 1 1 /var/lib/snapd/snaps/gnome-calculator_730.snap 0 512
/dev/loop31 0 0 1 1 /var/lib/snapd/snaps/juju_11454.snap 0 512
/dev/loop21 0 0 1 1 /var/lib/snapd/snaps/lxd_14583.snap (deleted) 0 512
/dev/loop11 0 0 1 1 /var/lib/snapd/snaps/charm_461.snap 0 512
/dev/loop0 0 0 1 1 /var/lib/snapd/snaps/lxd_14877.snap (deleted) 0 512
/dev/loop28 0 0 1 1 /var/lib/snapd/snaps/code_31.snap 0 512
/dev/loop18 0 0 1 1 /var/lib/snapd/snaps/lxd_14594.snap (deleted) 0 512
/dev/loop9 0 0 1 1 /var/lib/snapd/snaps/lxd_14804.snap (deleted) 0 512
/dev/loop16 0 0 1 1 /var/lib/snapd/snaps/lxd_14680.snap (deleted) 0 512
/dev/loop7 0 0 1 1 /var/lib/snapd/snaps/gnome-characters_495.snap 0 512
/dev/loop34 0 0 1 1 /var/lib/snapd/snaps/lxd_14890.snap 0 512
/dev/loop24 0 0 1 1 /var/lib/snapd/snaps/lxd_14649.snap (deleted) 0 512
/dev/loop14 0 0 1 1 /var/lib/snapd/snaps/gnome-logs_93.snap 0 512
/dev/loop32 0 0 1 1 /var/lib/snapd/snaps/core_9066.snap 0 512
/dev/loop22 0 0 1 1 /var/lib/snapd/snaps/lxd_14611.snap (deleted) 0 512
/dev/loop30 0 0 1 1 /var/lib/snapd/snaps/lxd_14759.snap (deleted) 0 512
/dev/loop10 0 0 1 1 /var/lib/snapd/snaps/gnome-3-28-1804_116.snap 0 512
>>> 15:56:00 andre@thinkpad ~
$ snap list
Name Version Rev Tracking ...

[Mike Rushton (leftyfb)]

This is still an issue in Ubuntu 22.04.1

$ mount|tail -1
/var/lib/snapd/snaps/lxd_24164.snap (deleted) on /media/leftyfb/disk type squashfs (ro,nosuid,nodev,relatime,errors=continue,uhelper=udisks2)
$ losetup|grep deleted
/dev/loop25 0 0 1 1 /var/lib/snapd/snaps/lxd_24164.snap (deleted) 0 512
$ ls -l /var/lib/snapd/snaps/lxd_24164.snap
ls: cannot access '/var/lib/snapd/snaps/lxd_24164.snap': No such file or directory

[yurenchen (yurenchen)]

This has bothered me for a long time
and lsof/fuser find nothing

```sh
$ sudo losetup -d /dev/loop38

$ losetup -l | grep del
/dev/loop38 0 0 1 1 /var/lib/snapd/snaps/snapd_19122.snap (deleted) 0 512
```

After tried several times of those cmds, the loop dev finally released

```sh
$ sudo systemctl stop *snap*
$ sudo systemctl start snapd.service
```

It's on
- ubuntu 22.04.2
- kernel 5.19.0-41-generic
- snapd 2.58+22.04

[Mike Rushton (leftyfb)]

@Zygmunt Krynicki (zyga)

It's been over 4 years, any progress on this? Is there at least a workaround we can do when this happens other than rebooting? With more applications becoming snaps, this is becoming a more common issue.