snapd.seeded.service has considerably slowed down

[Chloé Smith]

[SRU] 2.68.5: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/2098137

[ Impact ]

Missing preseed files when performing preseeding in a LXD container. This also results in the sense that preseeding is slowed down, which is a side effect and not the actual bug.

[ Test Plan ]

1. Produce error with snapd 2.67.1+20.04 deb (existing evidence is fine)

2. Switch to snapd 2.68.5 deb

3. Proof the preseeding works and preseeding files are not missing.

Repeat for all targeted Ubuntu releases.

---original---

In both Jammy and Focal CPC (cloud) images we are seeing failures across the latest serials. Closer inspection seems to point at the `snapd.seeded.service` service. Below is the output of several images along with their type, version and snapd versions.

GCE, focal image serial 20250501 [AMD64], snapd: 2.68.4 24505 [latest/stable], snapd .deb: 2.67.1+20.04
```
ubuntu@kajiya-focal-zvzdqrxefi:~$ systemd-analyze blame

7min 45.546s snapd.seeded.service
2min 19.838s dev-loop4.device
5.765s accounts-daemon.service
...
...
...
```

GCE, jammy image serial 20250501 [ARM64], snapd: 2.68.4 24509 [latest/stable], snapd .deb: 2.67.1+22.04
```
ubuntu@kajiya-cit-jj-base-accelerator-pkgvalidation-aozokbndxy:~$ systemd-analyze blame

7min 30.139s snapd.seeded.service
2min 18.601s dev-loop4.device
      5.576s google-guest-agent.service
      5.140s cloud-init.service
      2.962s nvidia-imex.service
...
...
...
```

AWS, focal image serial 20250502 [AMD64], snapd: 2.68.4 24505 [latest/stable], snapd .deb: 2.67.1+20.04
```
root@ip-172-31-42-57:/home/ubuntu# systemd-analyze critical-chain

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @7min 48.781s
└─multi-user.target @7min 48.779s
  └─snapd.seeded.service @12.654s +7min 35.091s
    └─basic.target @12.576s
      └─sockets.target @12.575s
        └─snap.lxd.daemon.unix.socket @7min 45.279s
          └─snap-lxd-32662.mount @7min 40.510s +17ms
            └─local-fs-pre.target @2.097s
              └─multipathd.service @2.046s +48ms
                └─systemd-udev-settle.service @950ms +1.091s
                  └─systemd-udev-trigger.service @814ms +132ms
                    └─systemd-udevd-kernel.socket @611ms
                      └─system.slice @469ms
                        └─-.slice @469ms
```

Download image, serial 20250501 [AMD64], snap:snapd stable 24505, https://people.canonical.com/~simpoir/jammy_20250501.img

```
$ sudo systemd-analyze critical-chain

The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @7min 19.569s
└─multi-user.target @7min 19.569s
  └─snapd.seeded.service @4.059s +7min 15.508s
    └─basic.target @4.036s
      └─sockets.target @4.035s
        └─snap.lxd.user-daemon.unix.socket @7min 18.248s
          └─snap-lxd-31333.mount @7min 16.161s +2ms
            └─local-fs-pre.target @496ms
              └─multipathd.service @422ms +73ms
                └─systemd-remount-fs.service @413ms +8ms
                  └─systemd-fsck-root.service @377ms +33ms
                    └─systemd-journald.socket @336ms
                      └─-.mount @304ms
                        └─-.slice @304ms
```

### More verbose information

This particular system (Jammy GCE image serial 20250501) has currently 4 IDs according to `snap changes`. Here's the verbose output for each ID's `debug timings`:

```
ubuntu@kajiya-cit-jj-base-accelerator-pkgvalidation-aozokbndxy:~$ snap debug timings 1 --verbose
ID Status Doing Undoing Label Summary
36 Hold - - run-hook Run default-configure hook of "lxd" snap if present
37 Hold - - start-snap-services Start snap "lxd" (31335) services
38 Hold - - run-hook Run configure hook of "lxd" snap if present
39 Hold - - run-hook Run health check of "lxd" snap
61 Hold - - run-hook Run install hook of "google-cloud-cli" snap if present
62 Hold - - run-hook Run default-configure hook of "google-cloud-cli" snap if present
63 Hold - - start-snap-services Start snap "google-cloud-cli" (329) services
64 Hold - - run-hook Run configure hook of "google-cloud-cli" snap if present
65 Hold - - run-hook Run health check of "google-cloud-cli" snap
66 Hold - - mark-seeded Mark system seeded
35 Error 120070ms - run-hook Run install hook of "lxd" snap if present
51 Undone 4ms 4ms run-hook Run health check of "core22" snap
50 Undone 9ms 10ms start-snap-services Start snap "core22" (1912) services
49 Undone 5ms 4ms run-hook Run install hook of "core22" snap if present
25 Undone 4ms 4ms run-hook Run health check of "core20" snap
24 Undone 14ms 10ms start-snap-services Start snap "core20" (2503) services
23 Undone 4ms 5ms run-hook Run install hook of "core20" snap if present
2 Undone 18ms 5ms run-hook Run configure hook of "core" snap if present
13 Undone 10ms 10ms start-snap-services Start snap "snapd" (24509) services
12 Undone 4ms 4ms run-hook Run install hook of "snapd" snap if present
60 Undone 25ms 5ms setup-aliases Setup snap "google-cloud-cli" aliases
59 Undone 32ms 4ms set-auto-aliases Set automatic aliases for snap "google-cloud-cli"
58 Undone 62ms 0ms auto-connect Automatically connect eligible plugs and slots of snap "google-cloud-cli"
57 Undone 82ms 22ms link-snap Make snap "google-cloud-cli" (329) available to the system
 ^ 23ms - generate-wrappers generate wrappers for snap google-cloud-cli
56 Undone 181ms 8ms setup-profiles Setup snap "google-cloud-cli" (329) security profiles
 ^ 78ms - setup-security-backend setup security backend "seccomp" for snap "google-cloud-cli"
 ^ 74ms - setup-security-backend[many] setup security backend "apparmor" for 1 snaps
  ^ 23ms - load-profiles[changed-many] load changed security profiles of 1 snaps
55 Undone 21ms 6ms copy-snap-data Copy snap "google-cloud-cli" data
54 Undone 740ms 183ms mount-snap Mount snap "google-cloud-cli" (329)
 ^ 665ms - check-snap check snap "google-cloud-cli"
 ^ 38ms - setup-snap setup snap "google-cloud-cli"
53 Undone 20ms 0ms prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/google-cloud-cli_329.snap" (329)
34 Undone 24ms 5ms setup-aliases Setup snap "lxd" aliases
33 Undone 26ms 4ms set-auto-aliases Set automatic aliases for snap "lxd"
67 Undone 1013ms 225ms setup-profiles Setup snap "lxd" (31335) security profiles for auto-connections
 ^ 150ms - setup-security-backend setup security backend "seccomp" for snap "lxd"
 ^ 823ms - setup-security-backend[many] setup security backend "apparmor" for 2 snaps
  ^ 599ms - load-profiles[changed-many] load changed security profiles of 2 snaps
 ^ - 7ms setup-security-backend setup security backend "seccomp" for snap "lxd"
 ^ - 142ms setup-security-backend setup security backend "udev" for snap "snapd"
 ^ - 49ms setup-security-backend[many] setup security backend "apparmor" for 2 snaps
  ^ - 26ms load-profiles[unchanged-many] load unchanged security profiles 2 snaps
71 Undone 35ms 10ms connect Connect lxd:network-bind to snapd:network-bind
69 Undone 23ms 10ms connect Connect lxd:lxd-support to snapd:lxd-support
70 Undone 22ms 10ms connect Connect lxd:network to snapd:network
68 Undone 25ms 10ms connect Connect lxd:system-observe to snapd:system-observe
32 Undone 30ms 0ms auto-connect Automatically connect eligible plugs and slots of snap "lxd"
31 Undone 154ms 188ms link-snap Make snap "lxd" (31335) available to the system
 ^ 63ms - generate-wrappers generate wrappers for snap lxd
30 Undone 846ms 7ms setup-profiles Setup snap "lxd" (31335) security profiles
 ^ 235ms - setup-security-backend setup security backend "seccomp" for snap "lxd"
 ^ 8ms - setup-security-backend setup security backend "mount" for snap "lxd"
 ^ 576ms - setup-security-backend[many] setup security backend "apparmor" for 1 snaps
  ^ 450ms - load-profiles[changed-many] load changed security profiles of 1 snaps
29 Undone 26ms 7ms copy-snap-data Copy snap "lxd" data
28 Undone 120ms 176ms mount-snap Mount snap "lxd" (31335)
 ^ 36ms - check-snap check snap "lxd"
 ^ 43ms - setup-snap setup snap "lxd"
27 Undone 24ms 0ms prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/lxd_31335.snap" (31335)
48 Undone 27ms 5ms setup-aliases Setup snap "core22" aliases
47 Undone 22ms 5ms set-auto-aliases Set automatic aliases for snap "core22"
46 Undone 23ms 0ms auto-connect Automatically connect eligible plugs and slots of snap "core22"
45 Undone 69ms 15ms link-snap Make snap "core22" (1912) available to the system
44 Undone 18ms 5ms setup-profiles Setup snap "core22" (1912) security profiles
43 Undone 21ms 6ms copy-snap-data Copy snap "core22" data
42 Undone 287ms 173ms mount-snap Mount snap "core22" (1912)
 ^ 207ms - check-snap check snap "core22"
 ^ 39ms - setup-snap setup snap "core22"
41 Undone 28ms 0ms prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/core22_1912.snap" (1912)
22 Undone 19ms 4ms setup-aliases Setup snap "core20" aliases
21 Undone 20ms 4ms set-auto-aliases Set automatic aliases for snap "core20"
20 Undone 21ms 0ms auto-connect Automatically connect eligible plugs and slots of snap "core20"
19 Undone 62ms 15ms link-snap Make snap "core20" (2503) available to the system
18 Undone 21ms 5ms setup-profiles Setup snap "core20" (2503) security profiles
17 Undone 23ms 6ms copy-snap-data Copy snap "core20" data
16 Undone 265ms 178ms mount-snap Mount snap "core20" (2503)
 ^ 175ms - check-snap check snap "core20"
 ^ 38ms - setup-snap setup snap "core20"
15 Undone 20ms 0ms prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/core20_2503.snap" (2503)
11 Undone 22ms 4ms setup-aliases Setup snap "snapd" aliases
10 Undone 24ms 4ms set-auto-aliases Set automatic aliases for snap "snapd"
9 Undone 26ms 0ms auto-connect Automatically connect eligible plugs and slots of snap "snapd"
8 Undone 65ms 15ms link-snap Make snap "snapd" (24509) available to the system
7 Error 152ms 27ms setup-profiles Setup snap "snapd" (24509) security profiles
 ^ 124ms - setup-security-backend[many] setup security backend "apparmor" for 1 snaps
52 Done 22ms - prerequisites Ensure prerequisites for "google-cloud-cli" are available
40 Done 10ms - prerequisites Ensure prerequisites for "core22" are available
14 Done 10ms - prerequisites Ensure prerequisites for "core20" are available
1 Done 51ms - mark-preseeded Mark system pre-seeded
26 Done 24ms - prerequisites Ensure prerequisites for "lxd" are available
6 Undone 23ms 8ms copy-snap-data Copy snap "snapd" data
5 Undone 106ms 173ms mount-snap Mount snap "snapd" (24509)
 ^ 23ms - check-snap check snap "snapd"
 ^ 44ms - setup-snap setup snap "snapd"
4 Undone 16ms 0ms prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/snapd_24509.snap" (24509)
3 Done 11ms - prerequisites Ensure prerequisites for "snapd" are available

$ snap debug timings 2 --verbose
ID Status Doing Undoing Label Summary
72 Done 3336ms - generate-device-key Generate device key
 ^ 3319ms - generate-rsa-key generating device key pair
73 Done 292ms - request-serial Request device serial
 ^ 233ms - get-serial get device serial
  ^ 121ms - prepare-serial-request prepare device serial request
  ^ 111ms - submit-serial-request submit device serial request
 ^ 45ms - fetch-keys fetch signing key chain

$ snap debug timings 3 --verbose
ID Status Doing Undoing Label Summary
75 Done 5ms - prerequisites Ensure prerequisites for "snapd" are available
76 Done 12ms - prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/snapd_24509.snap" (24509)
77 Done 222ms - mount-snap Mount snap "snapd" (24509)
 ^ 14ms - check-snap check snap "snapd"
 ^ 188ms - setup-snap setup snap "snapd"
78 Done 12ms - copy-snap-data Copy snap "snapd" data
79 Done 209ms - setup-profiles Setup snap "snapd" (24509) security profiles
 ^ 194ms - setup-security-backend[many] setup security backend "apparmor" for 1 snaps
80 Done 24ms - link-snap Make snap "snapd" (24509) available to the system
81 Done 25ms - auto-connect Automatically connect eligible plugs and slots of snap "snapd"
82 Done 13ms - set-auto-aliases Set automatic aliases for snap "snapd"
83 Done 13ms - setup-aliases Setup snap "snapd" aliases
84 Done 6ms - run-hook Run install hook of "snapd" snap if present
85 Done 13ms - start-snap-services Start snap "snapd" (24509) services
74 Done 16ms - run-hook Run configure hook of "core" snap if present
86 Done 6ms - prerequisites Ensure prerequisites for "core20" are available
87 Done 12ms - prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/core20_2503.snap" (2503)
88 Done 345ms - mount-snap Mount snap "core20" (2503)
 ^ 138ms - check-snap check snap "core20"
 ^ 183ms - setup-snap setup snap "core20"
89 Done 12ms - copy-snap-data Copy snap "core20" data
90 Done 16ms - setup-profiles Setup snap "core20" (2503) security profiles
91 Done 26ms - link-snap Make snap "core20" (2503) available to the system
92 Done 12ms - auto-connect Automatically connect eligible plugs and slots of snap "core20"
93 Done 13ms - set-auto-aliases Set automatic aliases for snap "core20"
94 Done 12ms - setup-aliases Setup snap "core20" aliases
95 Done 6ms - run-hook Run install hook of "core20" snap if present
96 Done 13ms - start-snap-services Start snap "core20" (2503) services
97 Done 6ms - run-hook Run health check of "core20" snap
112 Done 6ms - prerequisites Ensure prerequisites for "core22" are available
113 Done 13ms - prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/core22_1912.snap" (1912)
114 Done 370ms - mount-snap Mount snap "core22" (1912)
 ^ 165ms - check-snap check snap "core22"
 ^ 183ms - setup-snap setup snap "core22"
115 Done 14ms - copy-snap-data Copy snap "core22" data
116 Done 16ms - setup-profiles Setup snap "core22" (1912) security profiles
117 Done 28ms - link-snap Make snap "core22" (1912) available to the system
118 Done 13ms - auto-connect Automatically connect eligible plugs and slots of snap "core22"
119 Done 13ms - set-auto-aliases Set automatic aliases for snap "core22"
120 Done 13ms - setup-aliases Setup snap "core22" aliases
121 Done 6ms - run-hook Run install hook of "core22" snap if present
122 Done 13ms - start-snap-services Start snap "core22" (1912) services
123 Done 6ms - run-hook Run health check of "core22" snap
98 Done 13ms - prerequisites Ensure prerequisites for "lxd" are available
99 Done 14ms - prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/lxd_31335.snap" (31335)
100 Done 231ms - mount-snap Mount snap "lxd" (31335)
 ^ 18ms - check-snap check snap "lxd"
 ^ 190ms - setup-snap setup snap "lxd"
101 Done 13ms - copy-snap-data Copy snap "lxd" data
102 Done 864ms - setup-profiles Setup snap "lxd" (31335) security profiles
 ^ 106ms - setup-security-backend setup security backend "seccomp" for snap "lxd"
 ^ 736ms - setup-security-backend[many] setup security backend "apparmor" for 1 snaps
  ^ 688ms - load-profiles[changed-many] load changed security profiles of 1 snaps
103 Done 214ms - link-snap Make snap "lxd" (31335) available to the system
 ^ 187ms - generate-wrappers generate wrappers for snap lxd
104 Done 18ms - auto-connect Automatically connect eligible plugs and slots of snap "lxd"
143 Done 14ms - connect Connect lxd:network-bind to snapd:network-bind
141 Done 16ms - connect Connect lxd:lxd-support to snapd:lxd-support
142 Done 14ms - connect Connect lxd:network to snapd:network
140 Done 15ms - connect Connect lxd:system-observe to snapd:system-observe
139 Done 1006ms - setup-profiles Setup snap "lxd" (31335) security profiles for auto-connections
 ^ 65ms - setup-security-backend setup security backend "seccomp" for snap "lxd"
 ^ 913ms - setup-security-backend[many] setup security backend "apparmor" for 2 snaps
  ^ 841ms - load-profiles[changed-many] load changed security profiles of 2 snaps
105 Done 14ms - set-auto-aliases Set automatic aliases for snap "lxd"
106 Done 14ms - setup-aliases Setup snap "lxd" aliases
107 Done 516ms - run-hook Run install hook of "lxd" snap if present
108 Done 7ms - run-hook Run default-configure hook of "lxd" snap if present
109 Done 615ms - start-snap-services Start snap "lxd" (31335) services
 ^ 189ms - enable-services enable services ["snap.lxd.daemon.unix.socket" "snap.lxd.user-daemon.unix.socket" "snap.lxd.activate.service"]
 ^ 403ms - start-services start services
  ^ 5ms - start-service start service "snap.lxd.daemon.unix.socket"
  ^ 5ms - start-service start service "snap.lxd.user-daemon.unix.socket"
  ^ 393ms - start-service start service "snap.lxd.activate.service"
110 Done 124ms - run-hook Run configure hook of "lxd" snap if present
111 Done 7ms - run-hook Run health check of "lxd" snap
124 Done 13ms - prerequisites Ensure prerequisites for "google-cloud-cli" are available
125 Done 13ms - prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/google-cloud-cli_329.snap" (329)
126 Done 746ms - mount-snap Mount snap "google-cloud-cli" (329)
 ^ 520ms - check-snap check snap "google-cloud-cli"
 ^ 203ms - setup-snap setup snap "google-cloud-cli"
127 Done 13ms - copy-snap-data Copy snap "google-cloud-cli" data
128 Done 87ms - setup-profiles Setup snap "google-cloud-cli" (329) security profiles
 ^ 30ms - setup-security-backend setup security backend "seccomp" for snap "google-cloud-cli"
 ^ 38ms - setup-security-backend[many] setup security backend "apparmor" for 1 snaps
  ^ 17ms - load-profiles[changed-many] load changed security profiles of 1 snaps
129 Done 37ms - link-snap Make snap "google-cloud-cli" (329) available to the system
 ^ 9ms - generate-wrappers generate wrappers for snap google-cloud-cli
130 Done 13ms - auto-connect Automatically connect eligible plugs and slots of snap "google-cloud-cli"
131 Done 15ms - set-auto-aliases Set automatic aliases for snap "google-cloud-cli"
132 Done 14ms - setup-aliases Setup snap "google-cloud-cli" aliases
133 Done 7ms - run-hook Run install hook of "google-cloud-cli" snap if present
134 Done 7ms - run-hook Run default-configure hook of "google-cloud-cli" snap if present
135 Done 14ms - start-snap-services Start snap "google-cloud-cli" (329) services
136 Done 7ms - run-hook Run configure hook of "google-cloud-cli" snap if present
137 Done 7ms - run-hook Run health check of "google-cloud-cli" snap
138 Done 13ms - mark-seeded Mark system seeded

$ snap debug timings 4 --verbose
ID Status Doing Undoing Label Summary
144 Done 16ms - prerequisites Ensure prerequisites for "go" are available
145 Done 2965ms - download-snap Download snap "go" (10881) from channel "stable"
 ^ 2949ms - download download snap "go"
146 Done 483ms - validate-snap Fetch and check assertions for snap "go" (10881)
147 Done 406ms - mount-snap Mount snap "go" (10881)
 ^ 181ms - check-snap check snap "go"
 ^ 200ms - setup-snap setup snap "go"
148 Done 15ms - copy-snap-data Copy snap "go" data
149 Done 57ms - setup-profiles Setup snap "go" (10881) security profiles
 ^ 13ms - setup-security-backend setup security backend "seccomp" for snap "go"
 ^ 20ms - setup-security-backend[many] setup security backend "apparmor" for 1 snaps
  ^ 9ms - load-profiles[changed-many] load changed security profiles of 1 snaps
150 Done 35ms - link-snap Make snap "go" (10881) available to the system
151 Done 15ms - auto-connect Automatically connect eligible plugs and slots of snap "go"
152 Done 15ms - set-auto-aliases Set automatic aliases for snap "go"
153 Done 15ms - setup-aliases Setup snap "go" aliases
154 Done 8ms - run-hook Run install hook of "go" snap if present
155 Done 7ms - run-hook Run default-configure hook of "go" snap if present
156 Done 15ms - start-snap-services Start snap "go" (10881) services
157 Done 7ms - run-hook Run configure hook of "go" snap if present
158 Done 7ms - run-hook Run health check of "go" snap
```

[Chloé Smith (kajiya)]

This seems to be isolated to just Focal and Jammy at the moment, I don't have the privs to change the series though unfortunately.

[Chloé Smith (kajiya)]

ubuntu@kajiya-cit-jj-base-accelerator-pkgvalidation-aozokbndxy:~$ snap change 1
Status Spawn Ready Summary
Done yesterday at 13:59 UTC today at 12:08 UTC Ensure prerequisites for "snapd" are available
Undone yesterday at 13:59 UTC today at 12:08 UTC Prepare snap "/var/lib/snapd/seed/snaps/snapd_24509.snap" (24509)
Undone yesterday at 13:59 UTC today at 12:08 UTC Mount snap "snapd" (24509)
Undone yesterday at 13:59 UTC today at 12:08 UTC Copy snap "snapd" data
Error yesterday at 13:59 UTC today at 12:08 UTC Setup snap "snapd" (24509) security profiles
Undone yesterday at 13:59 UTC today at 12:08 UTC Make snap "snapd" (24509) available to the system
Undone yesterday at 13:59 UTC today at 12:08 UTC Automatically connect eligible plugs and slots of snap "snapd"
Undone yesterday at 13:59 UTC today at 12:08 UTC Set automatic aliases for snap "snapd"
Undone yesterday at 13:59 UTC today at 12:08 UTC Setup snap "snapd" aliases
Undone yesterday at 13:59 UTC today at 12:08 UTC Run install hook of "snapd" snap if present
Undone yesterday at 13:59 UTC today at 12:08 UTC Start snap "snapd" (24509) services
Undone yesterday at 13:59 UTC today at 12:08 UTC Run configure hook of "core" snap if present
Done yesterday at 13:59 UTC today at 12:08 UTC Ensure prerequisites for "core20" are available
Undone yesterday at 13:59 UTC today at 12:08 UTC Prepare snap "/var/lib/snapd/seed/snaps/core20_2503.snap" (2503)
Undone yesterday at 13:59 UTC today at 12:08 UTC Mount snap "core20" (2503)
Undone yesterday at 13:59 UTC today at 12:08 UTC Copy snap "core20" data
Undone yesterday at 13:59 UTC today at 12:08 UTC Setup snap "core20" (2503) security profiles
Undone yesterday at 13:59 UTC today at 12:08 UTC Make snap "core20" (2503) available to the system
Undone yesterday at 13:59 UTC today at 12:08 UTC Automatically connect eligible plugs and slots of snap "core20"
Undone yesterday at 13:59 UTC today at 12:08 UTC Set automatic aliases for snap "core20"
Undone yesterday at 13:59 UTC today at 12:08 UTC Setup snap "core20" aliases
Undone yesterday at 13:59 UTC today at 12:08 UTC Run install hook of "core20" snap if present
Undone yesterday at 13:59 UTC today at 12:08 UTC Start snap "core20" (2503) services
Undone yesterday at 13:59 UTC today at 12:08 UTC Run health check of "core20" snap
Done yesterday at 13:59 UTC today at 12:08 UTC Ensure prerequisites for "core22" are available
Undone yesterday at 13:59 UTC today at 12:08 UTC Prepare snap "/var/lib/snapd/seed/snaps/core22_1912.snap" (1912)
Undone yesterday at 13:59 UTC today at 12:08 UTC Mount snap "core22" (1912)
Undone yesterday at 13:59 UTC today at 12:08 UTC Copy snap "core22" data
Undone yesterday at 13:59 UTC today at 12:08 UTC Setup snap "core22" (1912) security profiles
Undone yesterday at 13:59 UTC today at 12:08 UTC Make snap "core22" (1912) available to the system
Undone yesterday at 13:59 UTC today at 12:08 UTC Automatically connect eligible plugs and slots of snap "core22"
Undone yesterday at 13:59 UTC today at 12:08 UTC Set automatic aliases for snap "core22"
Undone yesterd...

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

It seems that for some reason preseeding is not populating /var/lib/snapd/cgroup/

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

It looks like this change:

https://github.com/canonical/snapd/commit/75e6f1d02d9fb204ee27b6ef4a56133b41c7e84c

is causing the problem, as it removes udev backend (which creates the files in /var/lib/snapd/cgroup/) from the list of backends if running in a container, so when preseeding is run in an LP container it does not generate these files.

This is confirmed by looking at the snapd state.json file. This should appear if preseeding is correct, but it is not the case in the state file from the cloud image:

        "timings": [
          ...
          {
            "label": "setup-security-backend",
            "summary": "setup security backend \"udev\" for snap \"lxd\"",
            "duration": 16212930
          },

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Possible fix: https://github.com/canonical/snapd/pull/15411

[Chloé Smith (kajiya)]

From a focal GCE image built with the workaround in livecd-rootfs:

```
ubuntu@focal-quik-check:~$ systemd-analyze blame
12.647s dev-loop1.device
12.647s dev-loop0.device
12.634s dev-loop2.device
12.591s dev-loop3.device
12.579s dev-loop4.device
 9.637s snapd.seeded.service
 6.585s snapd.service
 5.008s accounts-daemon.service
 3.313s apport.service
 3.282s polkit.service
 2.996s pollinate.service
```

and

```
ubuntu@focal-quik-check:~$ snap list
Name Version Rev Tracking Publisher Notes
core20 20250213 2501 latest/stable canonical✓ base
core22 20250408 1963 latest/stable canonical✓ base
google-cloud-cli 520.0.0 328 latest/stable/… google-cloud-sdk✓ classic
lxd 4.0.10-d342340 32662 4.0/stable/… canonical✓ -
snapd 2.69+g113.58f8e17 24665 latest/edge canonical✓ snapd
```

so what's in edge rn is working :D

[Ernest Lotter (ernestl)]

Thanks for verifying the fix
We have created a snapd hotfix branch: latest/stable/hotfix-preseeding-lp2109843

[Chloé Smith (kajiya)]

Thanks Ernest! I'll try that in a PPA build first before merging into ~cloudware's livecd-rootfs fork

[Chloé Smith (kajiya)]

Can confirm that the new hotfix channel indeed still fixes the issue for GCE focal and jammy in my testing :D I'm going to merge the livecd-rootfs MPs soon and follow the builds across the clouds, assuming it's still all green I'll report back again :salute:

[Chloé Smith (kajiya)]

Just a heads up, I can't give a proper validation of `latest/stable/hotfix-preseeding-lp2109843` in production just yet as we need the branch/channel to be available in arm64, armhf, ppc64el, RISCV and s390x as well. Alfonso will give me a heads up when those are ready and I'll re-kick the builds on our and and let you know

[Ernest Lotter (ernestl)]

arm64, armhf, ppc64el, riscv64 and s390x now also available on the hotfix branch.

[Chloé Smith (kajiya)]

Can confirm that the fix in `latest/stable/hotfix-preseeding-lp2109843` works in production as well :) All affected focal and jammy images are building again and passing tests

[Ernest Lotter (ernestl)]

We are working on snapd 2.68.5 as non-hotfix solution to this bug. Furthermore, the hotfix version of the bug caused regression relating to udev in containers, that is also addressed in the version of the fix in snapd 2.68.5: https://github.com/canonical/snapd/pull/15439

See snapd roadmap for date estimated timeline: https://forum.snapcraft.io/t/the-snapd-roadmap/1973

[Chloé Smith (kajiya)]

ACK, thank for the update Ernest! :D

[Ernest Lotter (ernestl)]

Snapd 2.68.5 is now available in beta. Would appreciate a re-test.

[Chloé Smith (kajiya)]

I can confirm that `latest/beta` also fixes the issues we were seeing - thanks again everyone!

Just an FYI though that we'll continue to use `stable/hotfix-preseeding-lp2109843` for now, as when that branch expires it will fallback to `stable` which is what we want in images that are already out in the wild :)

[Ernest Lotter (ernestl)]

Re-released hotfix 2.68.4.1 to "stable/hotfix-preseeding-lp2109843", effectively just extending the expiry to 2025-07-04T00:00:00Z.

Snapd 2.68.5 is already in candidate, if all goes as expected, it should reach stable by next week Monday after which we will leave the "stable/hotfix-preseeding-lp2109843" to expire.

[Ernest Lotter (ernestl)]

Hi @kajiya, we have the snapd 2.68.5 deb available in questing-proposed, would appreciated yet another test as per SRU test template provided in description, to confirm this issue is also solved when using the deb as well.

[Chloé Smith (kajiya)]

@ernestl Thanks for your patience! I've added a comment to the SRU bug [0] with the results of my testing in questing :)

[0]: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/2098137

[Ernest Lotter (ernestl)]

Duplicating questing-proposed test result here:
================================================

### Questing testing ###

[Note that the bug in question was not seen in GCP questing 25.10 images, so this test is to check there are no regressions]

* Created a new questing GCP image with `snapd` installed from `-proposed` in the image build:
```
ubuntu@cit-questing-snapd-sru:~$ dpkg -s snapd
Package: snapd
Status: install ok installed
Priority: optional
Section: devel
Installed-Size: 125332
Maintainer: Ubuntu Developers <email address hidden>
Architecture: arm64
Version: 2.68.5+ubuntu25.10.2
```

* Verified that our internal testing worked (CTF and CIT)

* Verified that `snapd.seeded.service` wasn't taking longer than expected:
```
ubuntu@cit-questing-snapd-sru:~$ systemd-analyze critical-chain
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.

graphical.target @22.949s
└─multi-user.target @22.949s
  └─snapd.seeded.service @12.754s +10.193s
    └─basic.target @12.639s
      └─sockets.target @12.637s
        └─snapd.socket @12.608s +25ms
          └─sysinit.target @12.562s
          ...
   ...

ubuntu@cit-questing-snapd-sru:~$ systemd-analyze blame
10.193s snapd.seeded.service
 9.081s snapd.service
 6.451s google-guest-agent.service
 4.818s cloud-config.service
 4.775s cloud-init-main.service
...
...
```

* Verified that pre-seeding was successful:
```
ubuntu@cit-questing-snapd-sru:~$ snap debug timings 1 --verbose
ID Status Doing Undoing Label Summary
3 Done 8ms - prerequisites Ensure prerequisites for "snapd" are available
4 Done 12ms - prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/snapd_24509.snap" (24509)
5 Done 110ms - mount-snap Mount snap "snapd" (24509)
 ^ 37ms - check-snap check snap "snapd"
 ^ 49ms - setup-snap setup snap "snapd"
6 Done 12ms - copy-snap-data Copy snap "snapd" data
7 Done 152ms - setup-profiles Setup snap "snapd" (24509) security profiles
 ^ 9ms - setup-security-backend setup security backend "udev" for snap "snapd"
 ^ 127ms - setup-security-backend[many] setup security backend "apparmor" for 1 snaps
8 Done 32ms - link-snap Make snap "snapd" (24509) available to the system
9 Done 11ms - auto-connect Automatically connect eligible plugs and slots of snap "snapd"
10 Done 11ms - set-auto-aliases Set automatic aliases for snap "snapd"
11 Done 10ms - setup-aliases Setup snap "snapd" aliases
14 Done 6ms - prerequisites Ensure prerequisites for "core22" are available
15 Done 11ms - prepare-snap Prepare snap "/var/lib/snapd/seed/snaps/core22_1984.snap" (1984)
16 Done 295ms - mount-snap Mount snap "core22" (1984)
 ^ 215ms - check-snap check snap "core22"
 ^ 56ms - setup-snap setup snap "core22"
17 Done 15ms - copy-snap-data Copy snap "core22" data
18 Done 22ms - setup-profiles Setup snap "core22" (1984) security profiles
 ^ 5ms - setup-security-backend setup security backend "udev" for snap "core22"
19 Done 41ms - link-snap Make snap "core22" (1984) available to the system
20 Done 10ms - auto-connect Automatically connect eligible plugs and slots of snap "core22"
21 Done 10ms - set-auto-aliases Set automa...