Refreshing a snap using core18 to one using core16 confuses the snap apps

[Alfonso Sanchez-Beato]

Refreshing a snap using core18 to one using core16 makes the apps in the snap fail due to problems loading libraries. For instance, installing network-manager snap from the 1.10 track, that has core18 as base:

$ snap install --channel=1.10 network-manager
$ nmcli
...

If we refresh to the default track, that has core16 as base:

$ snap refresh --stable network-manager
$ nmcli
/snap/network-manager/516/usr/bin/nmcli: error while loading shared libraries: libreadline.so.6: cannot open shared object file: No such file or directory
$ snap run --shell network-manager.nmcli
<user@dev>:/home/admin$ ls /snap/core/current/lib/arm-linux-gnueabihf/libreadline.so.6
/snap/core/current/lib/arm-linux-gnueabihf/libreadline.so.6
<user@dev>:/home/admin$ ls /lib/arm-linux-gnueabihf/libreadline.so.6
ls: cannot access '/lib/arm-linux-gnueabihf/libreadline.so.6': No such file or directory

It looks like /lib/arm-linux-gnueabihf/ does not contain the right libraries. Rebooting the system solves the problem.

This has been detected on a UC16 system.

[Zygmunt Krynicki (zyga)]

I've reproduced this issue.

It seems that there's something wrong with the base transition detector. Some correlated output:

Loopback devices:

/dev/loop0 0 0 0 1 /writable/system-data/var/lib/snapd/snaps/core_8594.snap
/dev/loop5 0 0 1 1 /var/lib/snapd/snaps/core_8594.snap
/dev/loop20 0 0 1 1 /var/lib/snapd/snaps/core18_1671.snap

Preserved (stale) mount namespace:

1974 1643 7:20 / / ro,nodev,relatime master:284 - squashfs /dev/loop20 ro
2067 2064 7:5 / /snap/core/8594 ro,nodev,relatime master:230 - squashfs /dev/loop5 ro

Diagnostic message during detection:

DEBUG: block device of snap core, revision 8594 is 7:5
DEBUG: preserved mount is not stale, reusing

So snap-confine was correctly looking for presence of 7:5 (corresponding to core) but failed to observe that in the mount namespace the root filesystem is 7:20 (corresponding to core18).

There is an integration test for this feature so it must be something more subtle.

[Zygmunt Krynicki (zyga)]

It seems this was never implemented for UC16 systems. The comment in snap-confine ns-support.c reads:

        // TODO: enable this for core distributions. This is complex because on
        // core the rootfs is mounted in initrd and is _not_ changed (no
        // pivot_root) and the base snap is again mounted (2nd time) by
        // systemd. This makes us end up in a situation where the outer base
        // snap will never match the rootfs inside the mount namespace.

The code is guarded with a check for "normal" mode (which is used everywhere except for UC16)

        if (inv->is_normal_mode
            && should_discard_current_ns(base_snap_dev)) {

I'll look at correcting this.

[Zygmunt Krynicki (zyga)]

This is now fixed by the following pull request https://github.com/snapcore/snapd/pull/8093

[Zygmunt Krynicki (zyga)]

This fix will be backported to 2.43.3

[Zygmunt Krynicki (zyga)]

The snapd 2.43.3 release has been in the stable channel for a while. Marking as released.