Hardlinking snaps wastes 400 MB tmpfs RAM in live CDs

[Alkis Georgopoulos]

Boot with e.g. the current ubuntu-20.04-desktop-amd64.iso. Open a terminal and run `df -h`. You'll notice:

Filesystem Size Used Avail Use% Mounted on
/cow 2.0G 434M 1.5G 23% /

This means that the tmpfs file system needs 400 MB RAM for snaps. Xenial didn't have snaps and needed only 14 MB tmpfs. Snap shouldn't consume tmpfs RAM space, it should only use squashfs disk space.

I think this is caused by function Install() in snap/squashfs/squashfs.go, which hardlinks /var/lib/snapd/seed/snaps/* to /var/lib/snapd/snaps/*.
Hardlinks cause overlayfs copy-ups in live CDs and LTSP.

Maybe the next option, symlinks, should be preferred in overlayfs.

[Alkis Georgopoulos (alkisg)]

I think an acceptable solution is to just completely remove the "try to hardlink" logic from squashfs.go, and default to the "try to symlink" logic instead. Symlinks work in a lot more cases than hardlinks.

I cross-posted this in snap forums:
https://forum.snapcraft.io/t/hardlinking-snaps-wastes-400-mb-tmpfs-ram-in-live-cds/15980

I also applied a temporary workaround in LTSP:
https://github.com/ltsp/ltsp/blob/master/ltsp/client/init/55-snap.sh#L31

[Dimitri John Ledkov (xnox)]

I think we have a few options here:

1) For Desktop, Run /usr/lib/snapd/snap-preseed on the desktop images, this should complete most seeding on the base squashfs, without wasting tmpfs space and speed up boot

2) For Server, we might need to run /usr/lib/snapd/snap-preseed on the up-most layer, such that install boot is without symlinks, whilst the in-target first boot is still slow. Because server uses multi-layer squashfs.

[Michael Vogt (mvo)]

A fix for this is now merged into the 2.44 release branch.

[Alkis Georgopoulos (alkisg)]

Would it make sense to somehow backport this to bionic for subsequent 18.04.x live CDs?

[Michael Hudson-Doyle (mwhudson)]

snapd releases are backported to stable releases as a matter of course so it will get there sooner or later.

[Alkis Georgopoulos (alkisg)]

@mvo, I still see the issue in the official 20.04 isos.

I boot ubuntu-20.04-desktop-amd64.iso, I run `df -h`, and again I see:

Filesystem Size Used Avail Use% Mounted on
/cow 2.0G 434M 1.5G 23% /

The snapd version in the CD is 2.44.3+20.04, is the fix supposed to be included there?

[Ian Johnson (anonymouse67)]

Note that the PR that fixes this is https://github.com/snapcore/snapd/pull/8279, which git says should be in 2.44.3. It's possible the detection logic is slightly wrong, we will need to look into this more I think.

[Michael Vogt (mvo)]

There was a bug in the original symlink code that was utilized in 2.44. The fix itself was doing the overlayfs correction correctly. But the (unchanged) symlink in:
https://github.com/snapcore/snapd/pull/8279/files#diff-f440ed51e6e7c93f4920731345213192R114
has a bug: instead of `filepath.Dir(s.path) == dirs.SnapSeedDir ` it must use strings.HasPrefix(). The code in snapd 2.45 has this fix so the next respin of the desktop CD will fix it once the 2.45 SRU is approved. I tested this manually and a stock 2.45 snapd DTRT in the livecd environment.

[Zygmunt Krynicki (zyga)]

Snapd 2.45 has been released to stable, marking as released.
The Ubuntu package is in the SRU pipe.

[Alkis Georgopoulos (alkisg)]

The 20.04.1 live CDs appear not to be affected anymore, /cow only uses 50M RAM now.
Thumbs up! 👍