Refresh Control is not supported by ubuntu-image

[Kyle Nitzsche]

Building an Ubuntu Core image that respects Refresh Control is very difficult or impossible.

ubuntu-image does not obtain validated snap revisions: it just pulls the current revision on the specified channel. After first boot and refresh, the validated revisions are obtained & installed, but the first boot does not use validated revisions because they are not seeded.

If one has the validated snaps and their assertions in the current working dir when running ubuntu-image, one could install them into the image, asserted, using '--snap SNAPFILE'. But, there are problems getting the validated snaps/assertions. Notably, 'snap download SNAP --revision=FOO` is only supported for accounts with "developer access" (see the tool's help).

Refresh Control is most useful to control snaps that the brand specifically does *not* control, but it is precisely these snaps that cannot be downloaded with .snap download SNAP --revision...'.

(Not really sure if this issue should be filed against snapd or ubuntu-image...)

[Kyle Nitzsche (knitzsche)]

Note that the possible work around of building an image using sideloaded snaps of the validated revisions is not allowed when the model assertion uses "grade: signed" or "grade: secured" but only works with "grade: dangerous". Since "grade: dangerous" is not a preferred choice for production images, building a production image with validated snaps is currently difficult.

[Samuele Pedroni (pedronis)]

I'm working through this ATM. We will have to introduce an option to control the prepare-image behavior with respect to this. The initial setting will be backward compatible, with warnings that we will switch the default around at a later point.

[Hao Wang (soccerhaotian)]

Hi there, got a quick question for the release schedule, when will this fix be merged to the stable channel for snapd and the next version of ubuntu-image? Thanks.

[Ian Johnson (anonymouse67)]

The PR in question for snap prepare-image landed as part of https://github.com/snapcore/snapd/pull/10590 which will be included in snapd 2.53. I'm not sure of the release schedule of ubuntu-image support, since ubuntu-image also needs to learn about the option before it is useful from ubuntu-image directly.

[Kyle Nitzsche (knitzsche)]

Thanks Samuele and Ian.

Can someone from the ubuntu-image tool team please provide guidance on when the fix will be released?

Cheers

[Kyle Nitzsche (knitzsche)]

@sertac: Can you please indicate when this fix is likely to be released in ubuntu-image? thanks.

[Kyle Nitzsche (knitzsche)]

There is a path to build an image that honors Refresh Control validations before this bug fix is released.

1) Before testing a new snap revision that is current on the channel, run snap download SNAP.
2) If the test passes, save the downloaded snapfile
3) When building an image where the model has "grade: signed" and where the snap is declared in the model, you can use '--snap downloaded.snap' arguments on ubuntu-image. Apparently, ubuntu-image obtains the related assertions and is able to add the downloded snap asserted into the image, even if it is not the current revisoin on the channel specified in the mode, and even if you do not have the downloaded assertion file.

[Kyle Nitzsche (knitzsche)]

Looks like the merged fix for ubuntu-image is here: https://github.com/CanonicalLtd/ubuntu-image/pull/202

[Hao Wang (soccerhaotian)]

Hi Kyle,

Is the test case you mentioned used for uc18 images? I remember uc20 images cannot take "--snap" parameter so I saved all the snaps in assert files along with "grade: signed" setting. So for uc20 images, we may need to wait snapd 2.53 to be released on stable channel, then we can give it a test. :)

Thanks,
Hao

[Kyle Nitzsche (knitzsche)]

Hi Hao,

I found while experimenting with a uc20 model with "grade: signed" that the assertion was obtained and added to the image when --snap pointed at a downloaded snap file with no assertion in the current directory. (This surprised me.)

[Ian Johnson (anonymouse67)]

Note that snapd does not require the assertion file to be present, this is a persistent misunderstanding of how ubuntu-image / `snap prepare-image` works.

When you use --snap (or --extra-snaps) with ubuntu-image (and thus `snap prepare-image`), snapd will hash that .snap file and then query the store for a snap-revision assertion matching that hash, and this is how snapd figures out to include the assertions and thus build the image with the snap without the assertion files present. It has never looked at the local assert files, it has always used this hashing method.

[Hao Wang (soccerhaotian)]

Hi Ian,

Thanks for the details. It works to build an image. It is also found that when the image started the revision of docker will change to the current one on stable channel (1125) even though I installed the revision 796 in the image. Will try gating assertion command and see if 796 will be used when OS starts.

Thanks,
Hao

[Hao Wang (soccerhaotian)]

Hmm it works for a dangerous grade but got an error for a signed one:

ubuntu-image snap -O vm-image uc20-prod-signed.assert --snap docker_796.snap
Error: Error preparing image: cannot override channels, add devmode snaps, local snaps, or extra snaps with a model of grade higher than dangerous

Regards,
Hao

[Ian Johnson (anonymouse67)]

Does the model assertion you are using include the docker snap in the snaps key? If not, it needs to in order to be able to provide that snap locally through the --snap option

[Hao Wang (soccerhaotian)]

yeah, the assertion file includes docker snap:
  -
    default-channel: latest/stable
    id: sLCsFAO8PKM5Z0fAKNszUOX0YASjQfeZ
    name: docker
    type: app

Thanks,
Hao

[Ian Johnson (anonymouse67)]

Is that docker_796.snap asserted from the store or has it been changed?

[Hao Wang (soccerhaotian)]

Sorry, how can we assert a snap from the store? Is the above assertion file used for that purpose?

Thanks,
Hao

[Ian Johnson (anonymouse67)]

In order for a snap to be asserted, it just has to have been uploaded and accepted into the store. You don't need to download the assertion, as long as the exact same bits were uploaded to the store, the store will create an assertion for it and associate the hash of that file with the assertion.

Snapd when preparing an image will hash the snap file provided with `--snap` and ask the store if there is an assertion for that hash, and if the store says there is such an assertion, then snapd considers the snap file to be asserted.

Typically you would get an asserted snap file by running `snap download ...` and not modifying the resulting .snap file at all. If you unsquash it and modify it at all, the hash no longer matches and thus the .snap file is no longer asserted.

[Hao Wang (soccerhaotian)]

Hi Ian,

Thanks for the details. The snap being included is docker which we don't have the control:

```
#ubuntu-image snap -O vm-image uc20-prod-signed.assert --snap docker_796.snap
```

Regards,
Hao

[Ian Johnson (anonymouse67)]

@Hao did you modify the docker_796.snap at all?

Also what version of ubuntu-image are you using this with?

[Hao Wang (soccerhaotian)]

Hi Ian,

I didn't modify the file, and the version of ubuntu-image is "ubuntu-image 2.0+snap2".

Thanks,
Hao

[Ian Johnson (anonymouse67)]

For the purposes of testing, can you try using the stable channel of ubuntu-image instead?

[Hao Wang (soccerhaotian)]

Cool, it is building now :) There is a big difference between 1.11 and 2. Can you please shed some light on which commits or features cause the difference? Thanks. :)

[Ian Johnson (anonymouse67)]

Ubuntu-image was totally rewritten in Go for 2.0, so there are a lot of internal changes and it is clear to us now that there was not adequate test coverage for the features from the 1.11 version to accept the 2.0 implementation, but AIUI the ubuntu-image maintainers are working on it and are possibly very close to finishing the fixes so that 2.0 can be released.

[Łukasz Zemczak (sil2100)]

There is a lot of features and a lot of corner cases that we probably did not take into consideration, but we are continuously working on getting those fixed.

The 2.0 implementation is written in go and the main difference is that ubuntu-image no longer calls `snap prepare-image` but instead directly uses image.Prepare() from snapd. So possibly any additional handling that the actual `snap prepare-image` command is doing that is outside of image.Prepare() and outside of what we actually do now, will have potential to being broken.

Let me look into again how we're executing image.Prepare() and feeding the list of extra snaps to it, trying to see if there's anything worryingly different that we do in comparison to what `snap prepare-image` used to do.

[Łukasz Zemczak (sil2100)]

Okay, after some quick investigation, I think this now should be fixed after landing the recent changes from William. This issue could have been caused by us always passing a channel (using stable as default) to the list of snaps to include in the image.Prepare() call.

@Hao could you try the latest ubuntu-image snap from the edge channel and tell us if it's still a problem?

[Hao Wang (soccerhaotian)]

Hi Łukasz,

Sorry for the late reply. I tested the version 2 of ubuntu-image and it works :) The image was built with correct revisions of docker snap. I run into the issue when the OS was booted up. The docker snap on candidate channel was also gated. If not mistaken, the gated revision should be only for stable channel. Is this a known issue in snapd?

Thanks,
Hao