Unable to remove snap with content interface (with robust namespace update)

[Alberto Donato]

I've been testing the maas snap with robust-mount-namespace-update, to work around another issue when refreshing the snap.

I'm now getting the following error, when trying to remove the snap.

(this is from a fresh bionic container)

root@b:~# snap install core core18
2020-03-17T09:55:07Z INFO Waiting for restart...
core 16-2.43.3 from Canonical✓ installed
core18 20200124 from Canonical✓ installed

root@b:~# snap set core experimental.robust-mount-namespace-updates=true

root@b:~# snap get -d core
{
        "cloud": {
                "name": "unknown"
        },
        "experimental": {
                "robust-mount-namespace-updates": true
        },
        "refresh": {
                "hold": "2020-03-17T11:48:23.085562186Z"
        },
        "seed": {
                "loaded": true
        }
}

root@b:~# snap install maas --channel=2.7/edge
maas (2.7/edge) 2.7.0-8235-g.fea3a1678 from Canonical✓ installed

root@b:~# snap remove maas
error: cannot perform the following tasks:
- Disconnect maas:network-bind from core:network-bind (cannot update mount namespace of snap "maas": cannot update preserved namespace of snap "maas": cannot update snap namespace: read-only file system)

root@b:~# snap list
Name Version Rev Tracking Publisher Notes
core 16-2.43.3 8689 stable canonical✓ core
core18 20200124 1668 stable canonical✓ base
maas 2.7.0-8235-g.fea3a1678 5177 2.7/edge canonical✓ -
maas-cli 0.6.5 13 stable canonical✓ -

[Adam Collard (adam-collard)]

ubuntu@bionic-clean:~$ snap changes
ID Status Spawn Ready Summary
28 Done today at 10:28 UTC today at 10:28 UTC Change configuration of "core" snap
29 Done today at 10:28 UTC today at 10:29 UTC Install "maas" snap from "edge" channel
30 Done today at 10:28 UTC today at 10:28 UTC Running service command for snap "maas"
31 Error today at 10:29 UTC today at 10:29 UTC Remove "maas" snap

ubuntu@bionic-clean:~$ snap change 31
Status Spawn Ready Summary
Undone today at 10:29 UTC today at 10:29 UTC Stop snap "maas" services
Undone today at 10:29 UTC today at 10:29 UTC Run remove hook of "maas" snap if present
Done today at 10:29 UTC today at 10:29 UTC Disconnect interfaces of snap "maas"
Hold today at 10:29 UTC today at 10:29 UTC Save data of snap "maas" in automatic snapshot set #1
Hold today at 10:29 UTC today at 10:29 UTC Remove aliases for snap "maas"
Hold today at 10:29 UTC today at 10:29 UTC Make snap "maas" unavailable to the system
Hold today at 10:29 UTC today at 10:29 UTC Remove security profile for snap "maas" (5171)
Hold today at 10:29 UTC today at 10:29 UTC Remove data for snap "maas" (5171)
Hold today at 10:29 UTC today at 10:29 UTC Remove snap "maas" (5171) from the system
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:time-control from snapd:time-control
Error today at 10:29 UTC today at 10:29 UTC Disconnect maas:network-bind from snapd:network-bind
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:maas-cli from maas-cli:maas-cli
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:mount-observe from snapd:mount-observe
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:network from snapd:network
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:hardware-observe from snapd:hardware-observe
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:system-observe from snapd:system-observe
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:avahi-observe from snapd:avahi-observe
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:kernel-module-observe from snapd:kernel-module-observe
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:network-control from snapd:network-control
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:home from snapd:home
Hold today at 10:29 UTC today at 10:29 UTC Disconnect maas:network-observe from snapd:network-observe

......................................................................
Disconnect maas:network-bind from snapd:network-bind

2020-03-17T10:29:49Z ERROR cannot update mount namespace of snap "maas": cannot update preserved namespace of snap "maas": cannot update snap namespace: read-only file system

[Adam Collard (adam-collard)]

$ sudo snap remove maas
error: cannot perform the following tasks:
- Disconnect maas:maas-cli from maas-cli:maas-cli (cannot update mount namespace of snap "maas": cannot update preserved namespace of snap "maas":
-----
common.go:60: DEBUG: locking mount namespace of snap "maas"
common.go:81: DEBUG: freezing processes of snap "maas"
change.go:363: DEBUG: umount "/snap/maas/5171/maas-cli/lib" UMOUNT_NOFOLLOW (error: no such file or directory)
update.go:85: cannot change mount namespace according to change unmount (/snap/maas-cli/13/lib /snap/maas/5171/maas-cli/lib none bind,ro 0 0): no such file or directory
change.go:353: DEBUG: mount --make-rprivate "/snap/maas/5171" (error: <nil>)
change.go:363: DEBUG: umount "/snap/maas/5171" UMOUNT_NOFOLLOW|MNT_DETACH (error: <nil>)
change.go:408: DEBUG: remove "/snap/maas/5171" (error: remove /snap/maas/5171: device or resource busy)
change.go:433: DEBUG: cannot remove busy mount point "/snap/maas/5171"
change.go:353: DEBUG: mount --make-rprivate "/root" (error: <nil>)
change.go:363: DEBUG: umount "/root" UMOUNT_NOFOLLOW|MNT_DETACH (error: <nil>)
change.go:408: DEBUG: remove "/root" (error: remove /root: read-only file system)
common.go:89: DEBUG: unlocking mount namespace of snap "maas"
common.go:91: DEBUG: thawing processes of snap "maas"
cannot update snap namespace: read-only file system

[Zygmunt Krynicki (zyga)]

I've reproduced this and created a regression test.

[Zygmunt Krynicki (zyga)]

Some additional information:

google:ubuntu-18.04-64 .../tests/regression/lp-1867752# lxc exec bionic -- bash
root@bionic:~# SNAPD_DEBUG=1 /usr/lib/snapd/snap-update-ns maas
2020/03/17 13:32:31.283960 common.go:60: DEBUG: locking mount namespace of snap "maas"
2020/03/17 13:32:31.284697 common.go:81: DEBUG: freezing processes of snap "maas"
2020/03/17 13:32:31.286688 change.go:363: DEBUG: umount "/snap/maas/5177/maas-cli/lib" UMOUNT_NOFOLLOW (error: no such file or directory)
2020/03/17 13:32:31.287032 update.go:85: cannot change mount namespace according to change unmount (/snap/maas-cli/13/lib /snap/maas/5177/maas-cli/lib none bind,ro 0 0): no such file or directory
2020/03/17 13:32:31.287293 change.go:353: DEBUG: mount --make-rprivate "/snap/maas/5177" (error: <nil>)
2020/03/17 13:32:31.287544 change.go:363: DEBUG: umount "/snap/maas/5177" UMOUNT_NOFOLLOW|MNT_DETACH (error: <nil>)
2020/03/17 13:32:31.288782 change.go:408: DEBUG: remove "/snap/maas/5177" (error: remove /snap/maas/5177: device or resource busy)
2020/03/17 13:32:31.293743 change.go:433: DEBUG: cannot remove busy mount point "/snap/maas/5177"
2020/03/17 13:32:31.293784 change.go:353: DEBUG: mount --make-rprivate "/root" (error: <nil>)
2020/03/17 13:32:31.293817 change.go:363: DEBUG: umount "/root" UMOUNT_NOFOLLOW|MNT_DETACH (error: <nil>)
2020/03/17 13:32:31.293897 change.go:408: DEBUG: remove "/root" (error: remove /root: read-only file system)
2020/03/17 13:32:31.293919 common.go:89: DEBUG: unlocking mount namespace of snap "maas"
2020/03/17 13:32:31.293937 common.go:91: DEBUG: thawing processes of snap "maas"
cannot update snap namespace: read-only file system

root@bionic:~# cat /var/lib/snapd/mount/snap.maas.fstab
/snap/maas/5177/etc/freeipmi /etc/freeipmi none rbind,rw,x-snapd.origin=layout 0 0
/snap/maas/5177/etc/libvirt /etc/libvirt none rbind,rw,x-snapd.origin=layout 0 0
/var/snap/maas/5177/root /root none rbind,rw,x-snapd.origin=layout 0 0
/snap/maas-cli/13/lib /snap/maas/5177/maas-cli/lib none bind,ro 0 0

root@bionic:~# cat /run/snapd/ns/snap.maas.fstab
tmpfs /etc tmpfs x-snapd.synthetic,x-snapd.needed-by=/etc/freeipmi,mode=0755,uid=0,gid=0 0 0
/etc/.pwd.lock /etc/.pwd.lock none bind,x-snapd.kind=file,x-snapd.synthetic,x-snapd.needed-by=/etc/freeipmi 0 0
/etc/NetworkManager /etc/NetworkManager none rbind,x-snapd.synthetic,x-snapd.needed-by=/etc/freeipmi,x-snapd.detach 0 0
/etc/X11 /etc/X11 none rbind,x-snapd.synthetic,x-snapd.needed-by=/etc/freeipmi,x-snapd.detach 0 0
/etc/acpi /etc/acpi none rbind,x-snapd.synthetic,x-snapd.needed-by=/etc/freeipmi,x-snapd.detach 0 0
/etc/adduser.conf /etc/adduser.conf none bind,x-snapd.kind=file,x-snapd.synthetic,x-snapd.needed-by=/etc/freeipmi 0 0
/etc/alternatives /etc/alternatives none rbind,x-snapd.synthetic,x-snapd.needed-by=/etc/freeipmi,x-snapd.detach 0 0
/etc/apm /etc/apm none rbind,x-snapd.synthetic,x-snapd.needed-by=/etc/freeipmi,x-snapd.detach 0 0
/etc/apparmor /etc/apparmor none rbind,x-snapd.synthetic,x-snapd.needed-by=/etc/freeipmi,x-snapd.detach 0 0
/etc/apparmor.d /etc/apparmor.d none rbind,x-snapd.synthetic,x-snapd.needed-by=/etc/freeipmi,x-snapd.detach 0 0
/etc/apport /etc/apport none rbind,x-snapd.synthetic,x...

[Zygmunt Krynicki (zyga)]

I can say that this line is certainly unexpected:

/var/snap/maas/5177/root /root none rbind,rw,x-snapd.origin=layout 0 0

I think similarly to /home, layouts should not be allowed to put stuff over /root.

[Zygmunt Krynicki (zyga)]

Technically what is going on is that we try to remove /root but fail because it's a squashfs from the base snap. The reason this happens is because of the flaw in the following pseudo-code logic in snap-update-ns:

- unmount $path
- statfs $path
- if squashfs, then bail (it is read only)
- remove $path

What happens is that this is done inside a container so all the checks for squashfs fail, as we see FUSE instead. I need to adjust snap-update-ns to cope with that.

[Zygmunt Krynicki (zyga)]

I looked at this some more and I think there's a 2nd bug lurking here. I will post details soon but it seems we are removing a file from ext4 and we are really getting EROFS.

[Zygmunt Krynicki (zyga)]

So this is indeed interesting:

root@bionic:/# stat /root
  File: /root
  Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 801h/2049d Inode: 304158 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-03-17 15:45:50.385194035 +0000
Modify: 2020-03-17 15:44:38.592202223 +0000
Change: 2020-03-17 15:44:38.592202223 +0000
 Birth: -

root@bionic:/# stat -f /root
  File: "/root"
    ID: a11dca7cc1c64f21 Namelen: 255 Type: ext2/ext3
Block size: 4096 Fundamental block size: 4096
Blocks: Total: 2495808 Free: 1049984 Available: 1045888
Inodes: Total: 1290240 Free: 1077705

root@bionic:/# rmdir /root
rmdir: failed to remove '/root': Read-only file system

root@bionic:/# grep '/root ' < /proc/self/mountinfo
841 892 8:1 /var/snap/lxd/common/lxd/storage-pools/default/containers/bionic/rootfs/var/snap/maas/5177/root /root rw,relatime master:298 - ext4 /dev/sda1 rw,data=ordered

[Zygmunt Krynicki (zyga)]

This is now fixed by https://github.com/snapcore/snapd/pull/8285

[Zygmunt Krynicki (zyga)]

This was fixed and released with snapd 2.44.1, marking as such.