Always preseed core and snapd snap in server seed

[Philip Roche]

In removing the LXD snap from preseeding in the server seed for Ubuntu 24.04 as part LP #2051346 [1] we also removed the snapd snap and the core22 snap.

This means that are subsequent snap install, like LXD, will take much longer than expected for a non minimized image.

Time taken to install LXD snap using the lxd-installer package without snapd and core22 preinstalled/seeded

```
ubuntu@cloudimg:~$ time sudo lxd --version
Installing LXD snap, please be patient.
5.19

real 0m29.107s
user 0m0.006s
sys 0m0.005s
```

Time taken to install LXD snap using the lxd-installer package with snapd and core22 already installed.

```
ubuntu@cloudimg:~$ time sudo lxd --version
Installing LXD snap, please be patient.
5.19

real 0m15.034s
user 0m0.005s
sys 0m0.005s
```

This is a significant difference and for a workload we intend to remain as a core tested and tracked workload. As such I propose we re-introduce core22 and snapd snaps to our seed.

LXD do intend to move to the core24 snap as their base as I'm sure snapd does too so when that does happen we need to update the preseeded core snap.

This bug is to track the work of making that change in the server seed @ https://git.launchpad.net/~ubuntu-core-dev/ubuntu-seeds/+git/ubuntu/tree/server#n69

[1] https://bugs.launchpad.net/ubuntu/+source/ubuntu-meta/+bug/2051346

[Steve Langasek (vorlon)]

> This is a significant difference and for a workload we intend to remain as a core tested
> and tracked workload. As such I propose we re-introduce core22 and snapd snaps to our seed.

I disagree that the image should be optimized by default to prioritize the one-time startup performance of an optional use case.

15 seconds vs 30 seconds, on a thing that won't affect most cloud customers, and happens at most once per image, AND should be weighed against the first-boot speed improvements in clouds resulting from having a smaller image?

Also, statically seeding a particular base snap is bad form, as soon as lxd upgrades its base you lose your performance benefit and have to play catch-up in a stable release.

If "time to initialize lxd" is your metric, I think you're measuring the wrong thing :)

[Philip Roche (philroche)]

Good points

I'll measure boot speed with and without core snap preseeded and add it here.

time to initialize any snap was my goal but with lxd as an example as it is such a popular snap.

[Philip Roche (philroche)]

If we don't preseed a core snap and snapd it feels like we're failing to prioritise the performance of snaps on server/cloud.

But if we acknowledge that knowing that we are prioritising boot speed then that's fine and we can add it to the noble release notes.

[Philip Roche (philroche)]

> 15 seconds vs 30 seconds, on a thing that won't affect most cloud customers

I'll see if I can find the data to back this

[John Chittum (jchittum)]

I agree with vorlon in general. It is a bit odd to not be seeding snapd by default, since snaps are a recognized package format. preseeding snapd will bring in its base, but there's no guarantee that base will match any other snaps.

Flip side is that somewhere back in history, it was decided that in a system without any other snaps preseeded, that having the snapd deb, which exists to bootstrap itself into a snapd snap, was the right choice. so let's narrow the cases we know of:

* pre-installed systems with no other snaps pre-seeded

That limits itself, right now, to cloud-images "downloads" (the qcow, ova, etc). Right now the cloud team sees a fairly long time in running `lxd` commands, as we specifically have a mandate to ensure `lxd` is operating well on our cloud-images.

what we don't know right now

* boot time differences with adding the snaps back in (from old reading it looks anywhere from 3-5s, which is a nice little gain)
* the amount of time/slow down for `lxd` booting from `lxd-installer` in a "no snaps pre-seeded" setting
* the _usage_ and _expectation_ about the speed here. We definitely support running lxd on cloud-images. and we know it is a use case. but we don't know if this is a "many users" or "a couple users," and we don't know their expectations regarding speed. We may be able to roll out with good documentation letting everyone know what the change is, what to expect, etc.

The more info we can gather on our side for time cost the better. getting some data comparisons across the following would be good:

* boot times w/ and w/o preseeded snaps
* lxd launch time with no preseeded, snapd + it's core snap preseeded, and "other cloud cases that have preseeded snaps" (thinking like ec2 or oracle that have snapped cloud agents)

[Simon Déziel (sdeziel)]

FYI, snapd is a "base-less" snap:

$ lxc launch ubuntu-minimal-daily:22.04 c1
$ lxc shell c1
root@c1:~# snap list
No snaps are installed yet. Try 'snap install hello-world'.
root@c1:~# snap install snapd
2024-02-15T21:17:09Z INFO Waiting for automatic snapd restart...
snapd 2.61.1 from Canonical✓ installed
root@c1:~# snap list
Name Version Rev Tracking Publisher Notes
snapd 2.61.1 20671 latest/stable canonical✓ snapd

This is also visible by not having any `base:` while LXD currently uses core22:

root@c1:~# snap info --verbose snapd | grep base:
root@c1:~# snap info --verbose lxd | grep base:
base: core22

[Philip Roche (philroche)]

> "other cloud cases that have preseeded snaps" (thinking like ec2 or oracle that have snapped cloud agent

This isn't something we need to worry about as there will be no change in this case. If any agent snaps are preseeded then so too will a core snap and snapd snap.

[Philip Roche (philroche)]

Thank you for the detail sdeziel

[Christian Ehrhardt  (paelzer)]

It was nice to have LXD around and ready for many test/dev workloads,
and I feel it was worth it back then.
But we already replaced it with lxd-installer in minimal environments and it was fine there too.
I never heard someone complaining that LXD takes a bit there, but every second of boot time seems to be valued highly.

Now that we had to reduce this to the lxd-installer everywhere (Due to LP #2051346) it is really worth to be re-revaluated. Thank you for driving this Phil!

IMHO now that your first LXD command will take a bit longer already (due to fetching LXD snap), the exact amount of that "a bit longer" (as being more by also fetching snapd and base) seems almost irrelevant as long as it is in the same ballpark.

On one hand those dev/test environments that use it most, can easily be made to tolerate the bit of extra time - they usually start with a barrage of other "install this" anyway that has the same "wait for network and install" characteristic.

On the other hand reducing size and the initialization effort of it will save transfer and startup time for everyone - the guessed 3-5 seconds mentioned/assumed above would be totally worth it IMHO.

---

Furthermore as Simon showed (thanks), by snapd being a baseless snap we'd not even gain something by having that around already for the latter fetch of lxd by lxd-installer.

---

I further appreciate John's comment that we should back up some of our current assumptions (how much will this slow down lxc interactions, how much will the boot speed gain) with some actual data.

But if that data will not totally upset what we expect, then I very much agree with Steve in comment #1 and would not optimize for it at the cost of all others and thereby I'd be fine to not preseed the other bits there.

---

P.S. I wanted to mention that our perception might also be biased. I believe (no data) that the closer to Ubuntu development itself you are, the more likely you use LXD heavily in testing. But that same ratio likely does not apply to any user of Ubuntu images in the world.

[Philip Roche (philroche)]

@vorlon

> Also, statically seeding a particular base snap is bad form, as soon as lxd upgrades its base you lose your performance benefit and have to play catch-up in a stable release.

yes I don't like this either. Even if we do change it later to core24 then the expectations people have for their snap startup/install time for snaps, not LXD, which use core22 will change/break.

> If "time to initialize lxd" is your metric, I think you're measuring the wrong thing :)

Well it isn't just LXD though - any snap install/init will be affected with this change

@jchittum

> * the amount of time/slow down for `lxd` booting from `lxd-installer` in a "no snaps pre-seeded" setting

We do have this. In initial tests it 15 seconds vs 29 seconds.

> * the _usage_ and _expectation_ about the speed here.

I will try get this info from LXD team

> * boot times w/ and w/o preseeded snaps

I will gather this info

@paelzer

> I never heard someone complaining that LXD takes a bit there, but every second of boot time seems to be valued highly.
> Now that we had to reduce this to the lxd-installer everywhere (Due to LP #2051346) it is really worth to be re-revaluated. Thank you for driving this Phil!

Yes I really want this to be a decision we know we are making and consciously making.

>IMHO now that your first LXD command will take a bit longer already (due to fetching LXD snap), the exact amount of that "a bit longer" (as being more by also fetching snapd and base) seems almost irrelevant as long as it is in the same ballpark.

is 15 seconds vs 30 seconds really in the same ballpark?

> Furthermore as Simon showed (thanks), by snapd being a baseless snap we'd not even gain something by having that around already for the latter fetch of lxd by lxd-installer.

Not true. It still takes time to fetch and install the snapd snap - 6.738s in my test in qemu

> I further appreciate John's comment that we should back up some of our current assumptions (how much will this slow down lxc interactions, how much will the boot speed gain) with some actual data.

I will continue to gather data on the boot speed implications.

[Philip Roche (philroche)]

> * boot times w/ and w/o preseeded snaps

Without preseeded snaps:
```
ubuntu@cloudimg:~$ systemd-analyze
Startup finished in 3.609s (kernel) + 11.026s (userspace) = 14.636s
graphical.target reached after 10.642s in userspace.
```

With preseeded snapd and core22 snaps:

```
ubuntu@cloudimg:~$ systemd-analyze
Startup finished in 3.733s (kernel) + 12.566s (userspace) = 16.300s
graphical.target reached after 12.175s in userspace.
```

This is on a powerful AMD threadripper machine using qemu. I ran this 3 times for each image with similar results so we are definitely seeing a boot speed improvement without the snaps preseeded.

I will try with only snapd snap too for comparison.

[Philip Roche (philroche)]

With only snapd snap preseeded I get boot times very similar to

```
ubuntu@cloudimg:~$ systemd-analyze
Startup finished in 3.757s (kernel) + 12.458s (userspace) = 16.216s
graphical.target reached after 12.061s in userspace.
```

Which shows we are still taking a boot time hit of ~1.5 seconds... however with snapd snap preseeded we are seeing a much faster "first snap" install. From ~21 seconds down to 17 seconds.

To summarise.

* Definitely faster boot without any snaps preseeded @ ~1.5 seconds in my tests
* Still slower boot with only snapd snap preseeded but faster first snap install time by about ~4 seconds

I have been unable to get any data on LXD usage numbers on cloud/server to help.

So the question remains do we want faster boot by ~1.5 seconds (based on my test environment) at the expense of first snap install speed improvements of between 4 (if only snapd snap is preseeded) seconds and up to ~15 seconds (if a core snap is also preseeded)?

@vorlon @jchittum @paelzer given the above findings are you still -1 on any snap preseeding? Based on the data, I vote not to preseed any snaps.

[Steve Langasek (vorlon)]

On Fri, Feb 16, 2024 at 06:51:46PM -0000, Philip Roche wrote:
> @vorlon @jchittum @paelzer given the above findings are you still -1 on
> any snap preseeding? Based on the data, I vote not to preseed any snaps.

That is my position. Thanks for bringing data to this!

[John Chittum (jchittum)]

Based on the data, I'm in the "No" camp as well.

[Christian Ehrhardt  (paelzer)]

On Fri, Feb 16, 2024 at 06:51:46PM -0000, Philip Roche wrote:
> @vorlon @jchittum @paelzer given the above findings are you still -1 on
> any snap preseeding? Based on the data, I vote not to preseed any snaps.

I was already leaning that way and thank you for adding the data.
I agree to not to preseed any snap (in images where no mandatory snaps are present, i.e. not those agent examples you brought up above - these would stay as is right?).

[Philip Roche (philroche)]

Thanks all. Marking as "Won't Fix" and marked MP as Rejected.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.