Snapd 2.59 prepare-image not fetching assertions on arm64

[William Wilson]

Somewhere between snapd 2.58.3 and snapd 2.59 there is a regression introduced regarding fetching assertions/declarations while running snap prepare-image. I have so far only recreated this on arm64, but can do so reliably. I cannot make it happen on amd64, and haven't tried any other architectures.

In the following logs I have used these steps to reproduce this issue:

1) create two identical chroots, preseedjawn and preseedjawn2
2) run snap prepare-image from version 2.58.3 on preseedjawn
3) run snap prepare-image from version 2.59 on preseedjawn2
4) compare the contents of the <chroot>/var/lib/snapd/seed/assertions directories

```
 ubuntu@raspi:~$ snap info snapd
name: snapd
summary: Daemon and tooling that enable snap packages
publisher: Canonical✓
store-url: https://snapcraft.io/snapd
license: GPL-3.0
description: |
  Install, configure, refresh and remove snap packages. Snaps are
  'universal' packages that work across many different Linux systems,
  enabling secure distribution of the latest apps and utilities for
  cloud, servers, desktops and the internet of things.

  Start with 'snap list' to see installed snaps.
type: snapd
snap-id: PMrrV4ml8uWuEUDBT8dSGnKUYbevVhc4
tracking: latest/stable
refresh-date: 2 days ago, at 18:43 UTC
channels:
  latest/stable: 2.58.3 2023-03-20 (18600) 45MB -
  latest/candidate: 2.58.3 2023-03-07 (18600) 45MB -
  latest/beta: 2.59 2023-03-10 (18748) 48MB -
  latest/edge: 2.59+git709.g7dbd962 2023-03-22 (18872) 48MB -
installed: 2.58.3 (18600) 45MB snapd
 ubuntu@raspi:~$ sudo /snap/snapd/18600/usr/bin/snap prepare-image --classic --snap=firefox=latest/stable/ubuntu-23.04 --snap=bare --snap=core22 --snap=snapd --snap=snap-store=latest/stable/ubuntu-23.04 --snap=gtk-common-themes=latest/stable/ubuntu-23.04 --snap=gnome-42-2204=latest/stable/ubuntu-23.04 --snap=snapd-desktop-integration=latest/stable/ubuntu-23.04 --arch=arm64 test-classic.model preseedjawn/
Fetching firefox
firefox 2% 7.70MB/s 28.9sFetching bare
Fetching core22
Fetching snapd
Fetching snap-store
Fetching gtk-common-themes
Fetching gnome-42-2204
Fetching snapd-desktop-integration
 ubuntu@raspi:~$ ls preseedjawn/var/lib/snapd/seed/assertions
16,3wdHCAVyZEmYsCMFDE9qt92UV8rC8Wdk.snap-declaration
16,EISPgh06mRh1vordZY9OZ34QHdd7OrdR.snap-declaration
16,IrwRHakqtzhFRHJOOPxKVPU0Kk7Erhcu.snap-declaration
16,PMrrV4ml8uWuEUDBT8dSGnKUYbevVhc4.snap-declaration
16,amcUKQILKXHHTlmSa7NMdnXSx02dNeeT.snap-declaration
16,gjf3IPXoRiipCu9K0kVu52f0H56fIksg.snap-declaration
16,jZLfBRzf1cYlYysIjD2bwSzNtngY0qit.snap-declaration
16,lATO8HzwVvrAPrlZRAWpfyrJKlAJrZS3.snap-declaration
49jadSjo8x8qkrbA7YkTySJmMm5cKK4HLBT2j8-eryC1c0sYR5xREb4ZW9fpws0t.snap-revision
62rRwo_VHrOdajhLlnNTqUJ8gfWoZZNSFWhxn2AszCKyUAD0d8guxddMFeW5LVKP.snap-revision
BWDEoaqyr25nF5SNCvEv2v7QnM9QsfCc0PBMYD_i2NGSQ32EF2d4D0hqUel3m8ul.account-key
OgeoZuqQpVvSr9eGKJzNCrFGSaKXpkey.account
SI2bNbgfn98zy_t8zwPPDclZH4G_jGOBZG6inoX1VCYrSEyO6lBBPQ3-Xt6JCITU.snap-revision
_e4-wu7p76lfZfnk9D-5uc6mKXNYxMFIW_4O7yf_2hXQEsLRiYWc4OkqyplFeW5r.snap-revision
d-JcZF9nD9eBw7bwMnH61x-bklnQOhQud1Is6o_cn2wTj8EYDi9musrIT9z2MdAa.account-key
ecx0ulPGLHJUbE3jEYcBm_XT4NDYmm7GU5zP_DdQhg-lorbAKwMjIa2zwYoJ7MI5.snap-revision
generic.account
hpKwDJNvb0bpBjpJPecpStgoWmx93Z3oKB8BOMcqnWGG-VAEcyqEDo1lW6w0AJcy.snap-revision
model
olZfDYKrkDShlUIFgp2u72Xd10XY8JBBO9Pi5bqYmSPxuiH5RL25eY8vy4fSt08g.snap-revision
ssqXT1-13QfJ0xfZIq88CkcaPC8vcKrVlgGfBJSBr4WD5RFvxeuQ2E5y9U52VpyX.snap-revision
 ubuntu@raspi:~$ snap info snapd
name: snapd
summary: Daemon and tooling that enable snap packages
publisher: Canonical✓
store-url: https://snapcraft.io/snapd
license: GPL-3.0
description: |
  Install, configure, refresh and remove snap packages. Snaps are
  'universal' packages that work across many different Linux systems,
  enabling secure distribution of the latest apps and utilities for
  cloud, servers, desktops and the internet of things.

  Start with 'snap list' to see installed snaps.
type: snapd
snap-id: PMrrV4ml8uWuEUDBT8dSGnKUYbevVhc4
tracking: latest/beta
refresh-date: today at 21:09 UTC
channels:
  latest/stable: 2.58.3 2023-03-20 (18600) 45MB -
  latest/candidate: 2.58.3 2023-03-07 (18600) 45MB -
  latest/beta: 2.59 2023-03-10 (18748) 48MB -
  latest/edge: 2.59+git709.g7dbd962 2023-03-22 (18872) 48MB -
installed: 2.59 (18748) 48MB snapd
 ubuntu@raspi:~$ sudo /snap/snapd/18748/usr/bin/snap prepare-image --classic --snap=firefox=latest/stable/ubuntu-23.04 --snap=bare --snap=core22 --snap=snapd --snap=snap-store=latest/stable/ubuntu-23.04 --snap=gtk-common-themes=latest/stable/ubuntu-23.04 --snap=gnome-42-2204=latest/stable/ubuntu-23.04 --snap=snapd-desktop-integration=latest/stable/ubuntu-23.04 --arch=arm64 test-classic.model preseedjawn2/
Fetching firefox ({2449})
Fetching bare ({5})
Fetching core22 ({548})
Fetching snapd ({18600})
Fetching snap-store ({937})
Fetching gtk-common-themes ({1535})
Fetching gnome-42-2204 ({66})
Fetching snapd-desktop-integration ({58})
 ubuntu@raspi:~$ ls preseedjawn2/var/lib/snapd/seed/assertions
d-JcZF9nD9eBw7bwMnH61x-bklnQOhQud1Is6o_cn2wTj8EYDi9musrIT9z2MdAa.account-key generic.account model
```

As you can see, with version 2.59 many of the files that should be in assertions are missing. The contents of my model are the genericClassicModel from snapd's asserts/sysdb/generic.go file.

[Samuele Pedroni (pedronis)]

what's the content of test-classic.model used here?

[William Wilson (jawn-smith)]

I sourced test-classic model from snapd's asserts/sysdb/generic.go file. It's what ubuntu-image uses if a model is not provided for a classic image. I used this exact same file on amd64 and this behavior did not appear. Here are the contents:

```
type: model
authority-id: generic
series: 16
brand-id: generic
model: generic-classic
classic: true
timestamp: 2017-07-27T00:00:00.0Z
sign-key-sha3-384: d-JcZF9nD9eBw7bwMnH61x-bklnQOhQud1Is6o_cn2wTj8EYDi9musrIT9z2MdAa

AcLBXAQAAQoABgUCWYuXiAAKCRAdLQyY+/mCiST0D/0XGQauzV2bbTEy6DkrR1jlNbI6x8vfIdS8
KvEWYvzOWNhNlVSfwNOkFjs3uMHgCO6/fCg03wGXTyV9D7ZgrMeUzWrYp6EmXk8/LQSaBnff86XO
4/vYyfyvEYavhF0kQ6QGg8Cqr0EaMyw0x9/zWEO/Ll9fH/8nv9qcQq8N4AbebNvNxtGsCmJuXpSe
2rxl3Dw8XarYBmqgcBQhXxRNpa6/AgaTNBpPOTqgNA8ZtmbZwYLuaFjpZP410aJSs+evSKepy/ce
+zTA7RB3384YQVeZDdTudX2fGtuCnBZBAJ+NYlk0t8VFXxyOhyMSXeylSpNSx4pCqmUZRyaf5SDS
g1XxJet4IP0stZH1SfPOwc9oE81/bJlKsb9QIQKQRewvtUCLfe9a6Vy/CYd2elvcWOmeANVrJK0m
nRaz6VBm09RJTuwUT6vNugXSOCeF7W3WN1RHJuex0zw+nP3eCehxFSr33YrVniaA7zGfjXvS8tKx
AINNQB4g2fpfet4na6lPPMYM41WHIHPCMTz/fJQ6dZBSEg6UUZ/GiQhGEfWPBteK7yd9pQ8qB3fj
ER4UvKnR7hcVI26e3NGNkXP5kp0SFCkV5NQs8rzXzokpB7p/V5Pnqp3Km6wu45cU6UiTZFhR2IMT
l+6AMtrS4gDGHktOhwfmOMWqmhvR/INF+TjaWbsB6g==
```

I also tried a different custom model, but saw the same behavior. The contents of the custom model are:

```
type: model
authority-id: Pp6hQz8bUzbsiL3hRvxZ22lrCDlsKJAJ
series: 16
brand-id: Pp6hQz8bUzbsiL3hRvxZ22lrCDlsKJAJ
model: ubuntu-raspi-classic
architecture: arm64
base: core18
classic: true
display-name: Ubuntu Raspi Classic
distribution: lunar
grade: dangerous
snaps:
  -
    default-channel: latest/stable/ubuntu-23.04
    id: 3wdHCAVyZEmYsCMFDE9qt92UV8rC8Wdk
    name: firefox
    type: app
  -
    default-channel: latest/stable/ubuntu-23.04
    id: lATO8HzwVvrAPrlZRAWpfyrJKlAJrZS3
    name: gnome-42-2204
    type: app
  -
    default-channel: latest/stable/ubuntu-23.04
    id: gjf3IPXoRiipCu9K0kVu52f0H56fIksg
    name: snap-store
    type: app
  -
    default-channel: latest/stable/ubuntu-23.04
    id: IrwRHakqtzhFRHJOOPxKVPU0Kk7Erhcu
    name: snapd-desktop-integration
    type: app
  -
    default-channel: latest/stable/ubuntu-23.04
    id: jZLfBRzf1cYlYysIjD2bwSzNtngY0qit
    name: gtk-common-themes
    type: app
  -
    default-channel: latest/stable
    id: EISPgh06mRh1vordZY9OZ34QHdd7OrdR
    name: bare
    type: base
  -
    default-channel: latest/stable
    id: amcUKQILKXHHTlmSa7NMdnXSx02dNeeT
    name: core22
    type: base
  -
    default-channel: latest/stable
    id: PMrrV4ml8uWuEUDBT8dSGnKUYbevVhc4
    name: snapd
    type: snapd
timestamp: 2021-11-24T10:40:41+00:00
sign-key-sha3-384: SNdXhT977udsZN6CWRN7cTuSUGKhbXbID6iBMyykLzHBxYhQTUj6Jk9FDumLPUBe

AcLBcwQAAQoAHRYhBG53C20bnafIZaAIpl3EKJzwBddxBQJkG2ewAAoJEF3EKJzwBddxbQIP+QEC
Bjqznx4jmcIUFXc2F0xKl++JaoZhcArNSfEMa4CPemvn589JL5nAytQI2zNDjCPV2Os51tqXBUMx
nUHlqgOAAf9QtwHJ6i01nRAz/i7T07c1Inj5ixaF/KKEwPwF7dDEcCPOkxmQz9Nb8kmGY/XtM57E
vtQ2s5G9lfRc29DjQ7LgPA3jmS8ttS3EJ5ZH6wjNqM4jjxfT+wqB2ZSl8S28QR42dc/84puRqqai
6PZ36Id/nc71/k7oh6Yyq5xM5OhZwq3JaSS1T+bJi5mssCV+1RUeNb50iivJ6u8p7uGGd5psJkna
8Rne7cQDqeZjeYjtZh8jCckMdXNsEhTH/brlFyOy8m3ITDdC+4vU/q4bBltWATaN87ypG+4Cz9ft
KlZ4eCzNwKeJzdAdy9XrLxdICf8Nwhbfg...

[Samuele Pedroni (pedronis)]

unrelated, distribution: lunar should be distribution: ubuntu, it's a distribution id, we discussed whether to add version constraints but those would be on top of that

[Samuele Pedroni (pedronis)]

I think I understand the error but I'm not sure why it would apply to your 2nd model, as the snapd snap with snapd type should trigger assertion fetching in the logic...

I understand why that is not the case with the generic model though

[Samuele Pedroni (pedronis)]

I created https://github.com/snapcore/snapd/pull/12669 to try address this