seeding a snap with higher assumes than deb pkg of snapd on classic fails

[Ian Johnson]

When building a seed and putting it onto an image, if the version of snapd installed on the system through the distro packaging, (i.e. deb package snapd on ubuntu server) is less than the version specified by a seeded snap with `assumes`, the seeding process never seems to finish and no changes appear on the system.

My basic understanding of this is that:
1. image has snapd 2.37 as a debian package installed
2. image has seed created with new snapd (i.e. 2.39) and a snap with `assumes: [snapd2.38]`
3. image boots up, snapd 2.37 from debian package starts running
4. there is no snapd in /snap/core/, so snapd doesn't re-exec
5. snapd starts scanning the snaps in /var/lib/snapd/seed/...
6. snapd sees a snap that declares `assumes: [snapd2.38]` which is higher than the currently installed snapd version and suggests refreshing core snap to get a higher version
7. snapd waits forever until the newer snapd is available, which never happens

The only way I was able to move past this was essentially to upgrade the deb of snapd installed package because I couldn't `snap refresh core`, since the core snap hadn't been installed yet.

A possible resolution here would be if snapd detects that the following things are true:
1. snapd is in a seeding task
2. the currently running version of snapd is too low for one of the snaps in the seed
3. there is a version of snapd in the seed that is sufficient to finish the seeding

then we re-exec into the version of snapd from the seed.

Some additional context in case it's relevant:
There were 2 situations that the deb of snapd could be upgraded with this image since it was a pre-installed image:
1. After the image has been booted up and a user logs in they can run `apt update && apt install snapd` (to upgrade only snapd)
2. By booting the pre-installed image in runlevel 1, setting up networking, masking the snapd service and then running `apt update && apt install snapd`

For this image, I went with 2 because this prevents the other "first boot" tasks on an Ubuntu Server image from running such as cloud-init, resizing the SD card partitions (this is on a Raspberry Pi), etc. The reason I had to mask the snapd service was because upon installing the new version of snapd, apt will start the snapd service running and then snapd will attempt to setup the seed and eventually hang somewhere because some other service was necessary for it to start (I waited for over 30 minutes for the serial console to come back and it never did - the seeding of these snaps only takes 5 minutes when setup normally). There's probably another bug right there in the hang, but I think it's reasonable to not fix/investigate that more right now since we are running in runlevel 1 so something like snapd probably isn't going to entirely work anyways.

Another reason I couldn't do 1, because on this image (which is a cloud-image) the user doesn't get created until snapd is fully seeded (cloud-init sits waiting with `snap wait system seed.loaded`), so the user couldn't ever actually login until the seed was done, which never will happen because snapd is too old. To move past this, the user would have to bootup with runlevel 1 and add another user, then reboot and login with that user to run the upgrade.

So this seems to be a case where snapd would stand in the way of "booting" an image because cloud-init is waiting for it to finish seeding. Perhaps cloud-init should have a timeout waiting for snapd to seed the system?

[Zygmunt Krynicki (zyga)]

I think this is a missing stage in the validation of seed and construction of the seed structure itself. It would be useful to extend the interface for creating the seed directory to know which version of snapd is preinstalled on the system image. This could be used to detect "assumes" stanzas that cannot be fulfilled during the seeding process.

I'm marking it as confirmed.

[Samuele Pedroni (pedronis)]

A reexecing snapd in the case at lesat of the snapd snap should properly reexec to the snapd inside the snapd snapd early and let it proceed with the seeding.