Snaps are broken in daily images

[Iain Lane]

[ Description ]

eoan:

livecd-rootfs | 2.599

bionic:

livecd-rootfs | 2.525.27

Images currently being built have non-functional snaps. Seeding never finishes, and the journal contains:

Jul 15 13:52:16 ubuntu snapd[944]: stateengine.go:102: state ensure error: devicemgr: cannot proceed without seeding "core"

[ Fix ]

Explicitly seed 'core', since snapd still requires it currently.

[ QA ]

Boot a desktop ISO to the live session. Snap seeding is slow, so wait a couple of minutes. Check that the seeded snaps (e.g. gnome-calculator) are available in the Shell.

[ Regression potential ]

Image builds could start failing. They'll now have core where they didn't before, so that is a size increase.

[ Original report (rest of) ]

seed.yaml doesn't contain core:

ubuntu@ubuntu:~$ cat /var/lib/snapd/seed/seed.yaml
snaps:
  -
    name: core18
    channel: stable
    file: core18_1055.snap
  -
    name: snapd
    channel: stable
    file: snapd_3646.snap
  -
    name: gnome-3-28-1804
    channel: stable/ubuntu-19.10
    file: gnome-3-28-1804_67.snap
  -
    name: gnome-calculator
    channel: stable/ubuntu-19.10
    file: gnome-calculator_406.snap
  -
    name: gnome-characters
    channel: stable/ubuntu-19.10
    file: gnome-characters_296.snap
  -
    name: gnome-logs
    channel: stable/ubuntu-19.10
    file: gnome-logs_61.snap
  -
    name: gnome-system-monitor
    channel: stable/ubuntu-19.10
    file: gnome-system-monitor_100.snap
  -
    name: gtk-common-themes
    channel: stable/ubuntu-19.10
    file: gtk-common-themes_1313.snap

I think it is due to this change in livecd-rootfs:

laney@raleigh (ubuntu/master↑3|✔)> git show 05e71b38409b5a7cef41ebc864223466292268bb -- live-build/functions ~/dev/canonical/release/livecd-rootfs
commit 05e71b38409b5a7cef41ebc864223466292268bb
Author: Tobias Koch <email address hidden>
Date: Thu Apr 11 17:46:00 2019 +0200

    Do not include any snaps in ubuntu-cpc:minimized builds

diff --git a/live-build/functions b/live-build/functions
index c0dfcdca..6cc87d91 100644
--- a/live-build/functions
+++ b/live-build/functions
@@ -436,6 +436,17 @@ _snap_preseed() {
         return
     fi

+ # Determine if and what core snap is needed
+ local core_snap=$(/usr/share/livecd-rootfs/snap-tool info \
+ --cohort-key="${COHORT_KEY:-}" \
+ --channel="$CHANNEL" "$SNAP_NAME" | \
+ grep '^base:' | awk '{print $2}'
+ )
+
+ if [ -n "$core_snap" ]; then
+ _snap_preseed $CHROOT_ROOT $core_snap stable
+ fi
+
     sh -c "
         set -x;
         cd \"$CHROOT_ROOT/var/lib/snapd/seed\";
@@ -544,9 +555,6 @@ snap_prepare() {
     local CUSTOM_BRAND_MODEL=${2:-generic:generic-classic}

     snap_prepare_assertions "$CHROOT_ROOT" "$CUSTOM_BRAND_MODEL"
-
- # Download the core snap
- _snap_preseed $CHROOT_ROOT core stable
 }

 snap_preseed() {

Which stopped unconditionally seeding core. But this can't be the whole story, since disco was built with that livecd-rootfs and it *does* have core.

[Iain Lane (laney)]

I guess bringing it back in snap_prepare except for ubuntu-cpc:minimized builds would work?

[Iain Lane (laney)]

I'm probably going to upload a fix along the lines of comment #1, and we can see if it fixes tomorrow's builds or breaks horribly.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.600

---------------
livecd-rootfs (2.600) eoan; urgency=medium

  [ Tobias Koch ]
  * Do proper error checking when calling snap-tool info to determine
    the base of a snap.

  [ Iain Lane ]
  * Seed core for non minimized builds, as it is still required (LP:
    #1836594).

 -- Tobias Koch <email address hidden> Mon, 15 Jul 2019 16:13:31 +0100

[Iain Lane (laney)]

Seems to have worked. We need to SRU this now.

[Andy Whitcroft (apw)]

Hello Iain, or anyone else affected,

Accepted livecd-rootfs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.525.28 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andy Whitcroft (apw)]

Hello Iain, or anyone else affected,

Accepted livecd-rootfs into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.578.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

All autopkgtests for the newly accepted livecd-rootfs (2.578.6) for disco have finished running.
There have been regressions in tests triggered by the package. Please visit the sru report page and investigate the failures.

https://people.canonical.com/~ubuntu-archive/pending-sru.html#disco

[Iain Lane (laney)]

Comment #7 is not correct.

I tested a bionic desktop ISO built with this livecd-rootfs and built successfully and got core, so snaps work.

That is what this bug was for, so I'm marking v-done for bionic. (disco not tested)

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.525.28

---------------
livecd-rootfs (2.525.28) bionic; urgency=medium

  [ Tobias Koch ]
  * Do proper error checking when calling snap-tool info to determine
    the base of a snap. (LP: #1828500)

  [ Michael Vogt ]
  * Run "snap debug validate-seed" during preseeding to do basic validation of
    the generated seed.yaml

  [ Iain Lane ]
  * Seed core for non minimized builds, as it is still required (LP:
    #1836594).

 -- Iain Lane <email address hidden> Tue, 16 Jul 2019 13:20:18 +0100

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Timo Aaltonen (tjaalton)]

disco still not tested?

[Iain Lane (laney)]

Done now. I built a new ISO with -proposed enabled (label 20190913). The core snap is seeded, and the seeded snaps work.

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.578.7

---------------
livecd-rootfs (2.578.7) disco; urgency=medium

  [ Tobias Koch ]
  * Add retry logic to snap-tool to make downloads more resilient.
    (LP: #1837871)

 -- Tobias Koch <email address hidden> Mon, 26 Aug 2019 13:36:36 +0200

[Timo Aaltonen (tjaalton)]

This bug was fixed in the package livecd-rootfs - 2.578.7

---------------
livecd-rootfs (2.578.7) disco; urgency=medium

  [ Tobias Koch ]
  * Add retry logic to snap-tool to make downloads more resilient.
    (LP: #1837871)

 -- Tobias Koch <email address hidden> Mon, 26 Aug 2019 13:36:36 +0200