Enable the installation of snaps in a classic chroot

[Dan Watkins]

Cloud images are built using livecd-rootfs and then modified for specific clouds by using chroot to modify the contents of the image (without ever booting it).

In order to use snaps to deliver functionality required by clouds on first boot (generally speaking, the agents that they provide), we need some way of installing snaps within these chroots.

Currently, running `snap install ...` within a chroot fails because snapd is (of course) not running.

[Dan Watkins (oddbloke)]

2016-08-04 17:46:33 slangasek we're going to have a 'snap fetch', I think?
2016-08-04 17:46:53 Odd_Bloke Yeah, I've heard about that in the context of caching a snap for later installation.
2016-08-04 17:47:33 slangasek Odd_Bloke: right. So I think that we may need to do a 'snap fetch', followed by a bit of manual fiddling to put the snap bits in the right place as part of the image build
2016-08-04 17:47:55 slangasek niemeyer: ^^ maybe you'd be able to comment on this problem (pre-installing snaps as part of a classic image build, which happens in a chroot with no running snapd service)

[Jamie Bennett (jamiebennett)]

From 2.12 we have a mechanism in place that allows snaps to be installed on first boot using a seed.yaml file and a seed directory. Would this be enough?

[Dan Watkins (oddbloke)]

Provided this happened early enough in boot, this could work. As an example of why we need this to be early in boot, on Azure we can't get networking until walinuxagent has run for the first time.

As a side note, how much slower is installation of a seeded snap than boot-time mounting etc. of it? Clouds are very, very focused on boot time at the moment, so that's something we'll definitely need to be sensitive to.

[Jamie Bennett (jamiebennett)]

The seed functionality has just landed in snappy so we will test the impact of installing seeded packages at boot time for reference. Maybe it will be fast enough for your use case but until we have the data it is hard to say.

[Michael Vogt (mvo)]

I started looking into this today. We have two options:

1. add seed.yaml support for classic
- you can put your snaps into seed.yaml and put the needed assertions into seed/assertions
- on first boot snapd will install them
- this will slow down booting
- we need to figure out what to do about seeding on "normal" classic systems, i.e. should we wait until we see a seed.yaml ? should we consider the system seeded if we don't see a seed.yaml?

2. make snapd work in chroots
- you run snapd inside a chroot
- you install everything inside a chroot
- boot is fast
- we need to figure out how to do that. right now snapd requires a running systemd for various operations including just mounting the snaps.

[Michael Vogt (mvo)]

Ok, here is a slightly gross idea if install inside the chroot is a hard requirement.
This will *only* work for a limited subset of snap and only currently and I have not tested it yet carefully:
- get a chroot of the system
- ensure system /dev is bind mounted for /dev/loop? access
- override /bin/systemctl with a shell script that is smart enough to understand snapd mount units and mount the squashfs snaps to /snap/$snap/$rev, return "sure, will do" (exit code 0) for everything else
- override /bin/udevadm with "printf "sure"; exit 0"
- run snapd inside the chroot manually via: sudo /lib/systemd/systemd-activate -E SNAP_REEXEC=0 /run/snapd.socket -l /run/snapd-snap.socket /usr/lib/snapd/snapd&
- run snap install core
- cross fingers
- snapd will install core and trigger a restart of itself
- run snapd (same cmdline as above) again
- snap install yoursnap [1]
- stop snapd

This will probably work, it is obviously hacky and seeding is much cleaner. However if running inside a chroot is a use-case we want to support we can improve snapd here, it will still be much more limited because of the lack of systemd, i.e. a snapd inside a chroot would not be able to run services. But for putting bits on disk we can improve snapd to behave better.

[1] the snap can probably not have hooks or anything else fancy

[Michael Vogt (mvo)]

seed.yaml support is currently landing for classic now so this should allow us to go with option (1) and we can figure out how bad/good it is for boot speed.

[Dan Watkins (oddbloke)]

So seeding as it currently works isn't sufficient for what we need to do, because snapd runs too late in boot. To maintain parity with images as they ship now, we need commands installed by snapd to be available to, at the very least, cloud-init user-data; we may also want them to be available to services that run during the boot process.

I've been experimenting with moving snapd earlier in the boot process, just to see if that would address our issues. I've included the modified systemd unit files I was using at the bottom of this comment.

At a high level, it looks like we might be able to use this. I've been testing on GCE (because we have a snap of their SDK which we would eventually like to install instead of the Ubuntu package), and it looks like the seeding process might have minimal impact on boot; the time that apparmor, networking and the metadata collection phase of cloud-init take to start is roughly the amount of time that seeding takes. (I've attached a plot of a boot with my unit files, though you can't really see the seeding on here.)

Not being able to see the seeding from systemd's point-of-view is an issue; for a reliable boot, we would need to be able to specify that units shouldn't start until seeding is complete (i.e. After=...).

So, to summarise: in order to start using this, we will need two things:

- snapd to be configured to start as early as it possibly can, and
- snapd to expose seed-completion in a way that systemd units can relate to.

---

$ cat /lib/systemd/system/snapd.service
[Unit]
Description=Snappy daemon
Requires=snapd.socket

DefaultDependencies=no
Wants=network-pre.target
After=systemd-remount-fs.service
Before=NetworkManager.service
Before=network-pre.target
Before=shutdown.target
Before=sysinit.target
Conflicts=shutdown.target

[Service]
ExecStart=/usr/lib/snapd/snapd
EnvironmentFile=/etc/environment
Restart=always

[Install]
WantedBy=multi-user.target

$ cat /lib/systemd/system/snapd.socket
[Unit]
Description=Socket activation for snappy daemon
DefaultDependencies=no
Before=sockets.target

[Socket]
ListenStream=/run/snapd.socket
ListenStream=/run/snapd-snap.socket
SocketMode=0666
# these are the defaults, but can't hurt to specify them anyway:
SocketUser=root
SocketGroup=root

[Install]
WantedBy=sockets.target

[Boris Rybalkin (ribalkin)]

Out of curiosity, why not just boot the image for snap installation?
It must be easier to clean rootfs after the boot to make it look like first boot again.
Tools like docker or systemd-nspawn should be easy to automate depending on your build infrastructure.

[Zygmunt Krynicki (zyga)]

I'm marking this as triaged as it's not a new bug for sure.

Is this still an issue? I think the current design is that snaps are dropped into a specific place and then on boot the system is "seeded" and then things just work.

[Dan Watkins (oddbloke)]

So pre-seeding is what we've gone ahead with; it would still be nice to be able to perform a proper installation before first boot, but I don't think it's a high priority.

As a side note, the lack of a "seeded" synchronisation point is being worked on in https://github.com/snapcore/snapd/pull/5124 as it's now a hot issue in bionic.

[shasheene (shasheene)]

Beyond just cloud images, other use cases for Linux root filesystems configured using chroots are Linux live distribution ISOs and embedded Internet of Things devices. These filesystems are often embedded in read-only environments typically through the use of a squashfs-compressed archive (historically residing on read-only media such as optical disks). In this common situation, disk writes are handled by a tmpfs and OverlayFS which means any changes do not persistent between boots. This approach is often used as it helps with reliability of IoT devices in the face of sudden power interruptions.

Avoiding the snap pre-seeding workaround by providing the ability to install a snap from within a chroot means that the complete package environment of the live distribution can be fully configured at build-time. This does not just save time during boot up, it also prevents RAM wastage because the tmpfs overlay may get modified through work that would traditionally be done in the chroot and thus reside on the root filesystem.

This issue is already almost 4 years old. If it's possible to implement installation of snap-packages in chroot environments then I think it should be done. Otherwise in the coming years when/if snap-based packaging becomes more common the overhead of wasted CPU cycles and RAM will become more significant.

[Zygmunt Krynicki (zyga)]

A terse comment from sandbox POV:

- chroot is incompatible with apparmor - multiple chroots would overwrite each other’s sandbox system.
- apparmor is incompatible with overlayfs, the special case for live media is poorly tested and fragile

I don’t see any way to support chroots.

[Jamie Strandboge (jdstrand)]

> A terse comment from sandbox POV:
>
> - chroot is incompatible with apparmor - multiple chroots would overwrite each other’s sandbox system.
> - apparmor is incompatible with overlayfs, the special case for live media is poorly tested and fragile

This is overstated. apparmor works fine with chroots. What must be accounted for is that chroot paths are mediated as the full path by default. Eg, if in the chroot the access is /etc/fstab, the mediation happens at /path/to/chroot/etc/fstab. The chroot_relative profile flag could help depending on what is needed. AppArmor aliases are another tool that could be leveraged.

As for overlayfs, it mostly works on newer kernels but there are additional rules that need to be added that are dependent on how the overlayfs filesystem is mounted. What makes overlayfs problematic is snapd can't easily determine the mount options automatically (which is why we currently have special handling for specific livecd environments) and snapd is expected to work on older kernels where the overlayfs support is not as good.

@zyga - IMO it would be possible to support chroots (as opposed to overlayfs) in certain circumstances provided snapd knew about it. The use cases would need to be clearly outlined to design a proper solution.

[PJSingh5000 (pjsingh5000)]

Is there any more information or progress on this issue?

[Steve Langasek (vorlon)]

This bug was originally filed in support of building cloud images containing snaps. We have had support for preseeding of snaps as part of image builds for a number of years. There will be no support for installing and running snaps in a chroot, for the reasons given in this bug report.

Therefore closing this bug.

[regs (regs-pochta)]

What's the modern way of baking ISO then, so the could have preinstalled snaps?

[Rudra Saraswat (rs2009)]

This is going to prevent us from releasing Ubuntu Web 22.04's Brave edition. We want to remove the Firefox snap from it, but looks like packages can't be removed from a seed.yaml file (to replace it with the Brave snap). Are there any plans to implement option 2? (basic support for installation and removal of snap packages inside a chroot, other operations don't need to be supported for most usecases inside a chroot, like mounting and running snaps).

This doesn't only affect flavors and remixes like Ubuntu Unity, Ubuntu Web, Ubuntu Cinnamon etc, but also affects people who

1) build ISOs with tools like the Ubuntu Remix Builder (https://gitlab.com/ubuntu-unity/ubuntu-remixes) and @PJSingh5000's Cubic project for their regular setups

2) develop downstream distros which remove snap packages when building inside a chroot

3) build images for their embedded devices using their own builders (not Ubuntu's builder) which build inside a chroot

I think this issue is really important and should be re-opened. (seed.yaml is only a temporary workaround for very few usecases)

@vorlon Thanks.

[Jani Jaakkola (jj-lousa)]

I'm building Linux images in chroot to deploy in our Linux network at University of Helsinki. Since this is has been closed as "won't fix" it seems I'm forced to remove the whole snap system from the prebuild images to get a nice clean working install image. I could just build the images in a virtual machine, but that would be a hassle. And these kind of problems are the reason why I really just want to get rid of snap packages completely.

[Cubic PPA (cubic-wizard)]

I agree that this should be reopened. It should be straightforward to convert some of the snapd code into scripts that perform simple functions such as `snap install`, `snap list`, `snap remove`. These could be installable as a separate package from snapd.

[Zygmunt Krynicki (zyga)]

Im no longer actively involved in upstream snaps development but I came to comment that the claim that it should be straightforward to convert snaps to to scripts that perform simple steps such as snap install, and so on, is entirely detached from reality of how snapd operates.

It is anything but straightforward and in the current design of snaps, is mostly impossible. The only thing you can do is to create a seed and that is indeed supported, but snaps will be installed form that seed on system startup.

The basics of seeding are described on the snapcraft forum: https://discourse.ubuntu.com/t/seeding-a-classic-ubuntu-image/19756

[Cubic PPA (cubic-wizard)]

@Zygmunt Krynicki,

Thanks for chiming in.

I realize this is a closed bug, so I don't want to prolong the discussion here.

I certainty can appreciate when something looks "easy" to outsiders (such as myself), when it fact, it is not straightforward. I did not realize this was the case with Snaps.

I will look at the link you shared. (Thanks for that).

Of course, I still hold out the hope that chroot support is offered for Snaps in the future.

[TQ (nothinrandom)]

It looks like there's still not a way to fully pre-seed snap packages. On the first boot of a non live-cd, the system is installing the necessary snap packages that are specified in the seed folder, and this adds between 90s and 300s (depending on hardware). Is there a way to sync the snap packages directly to the target system? I noticed that during installation (ubiquity), the snap packages are installed because if you kill the gui and log into the system, you can run snap list, which shows all of the installed snaps. Could we "sync" these installed snaps to the target system via ubiquity/success_command? Thanks folks.