snapd fails always in Optimised Ubuntu Desktop images available in Microsoft Hyper-V gallery

Bug #1828500 reported by Peter Goodall on 2019-05-10
42
This bug affects 7 people
Affects Status Importance Assigned to Milestone
livecd-rootfs (Ubuntu)
Critical
Unassigned
Bionic
Undecided
Unassigned
Disco
Undecided
Unassigned

Bug Description

[Impact]

* Failing to do error checks on `snap-tool info` calls while determining the bases of snaps can lead to inconsistently-seeded images as reported in the original bug report below.

* This affects all stable releases which use automatic detection of the base snap during image builds.

* The changes proposed introduce extra error checking which should result in a build error, if the `snap-tool info` call fails, thus preventing distribution of a broken image.

[Test Case]

* It may be tricky to reproduce the exact error on demand. But it should be clear by looking at the code path in

https://code.launchpad.net/~tobijk/livecd-rootfs/+git/livecd-rootfs/+merge/370096

that the scenario described below may arise.

* A way to test that the changes have no *negative* impact on functionality is to run a script such as

    https://pastebin.ubuntu.com/p/wQknctr6ys/

and compare it against the expected result

    https://pastebin.ubuntu.com/p/N8PXRJJV9b/

(In this case both core and core18 should have been seeded.) and

[Regression Potential]

* The regression potential for these changes is low, as they only add extra error checking but do not change the existing logic.

--- Original Bug Report ---

Alway fails in both 18.04.2 and 19.04
Reporting the bug failed too yesterday.
As a consequence LivePatch and everything else fails that is based on snapd.
This makes system unusable. This is a supported configuration.

https://blog.ubuntu.com/2018/09/17/optimised-ubuntu-desktop-images-available-in-microsoft-hyper-v-gallery
https://blog.ubuntu.com/2019/05/07/19-04-disco-dingo-now-available-as-optimised-desktop-image-for-hyper-v

xx@xenial-Virtual-Machine:~$ sudo snap install vscode
[sudo] password for peterg:
error: too early for operation, device not yet seeded or device model not acknowledged
xx@xenial-Virtual-Machine:~$ journalctl -b -u snapd --full --no-pager
-- Logs begin at Thu 2019-05-09 20:06:58 AEST, end at Fri 2019-05-10 13:20:46 AEST. --
May 10 13:19:59 xenial-Virtual-Machine systemd[1]: Starting Snappy daemon...
May 10 13:20:00 xenial-Virtual-Machine snapd[566]: AppArmor status: apparmor is enabled and all features are available
May 10 13:20:00 xenial-Virtual-Machine snapd[566]: helpers.go:717: cannot retrieve info for snap "gnome-calculator": cannot find installed snap "gnome-calculator" at revision 352: missing file /snap/gnome-calculator/352/meta/snap.yaml
May 10 13:20:00 xenial-Virtual-Machine snapd[566]: daemon.go:379: started snapd/2.38+18.04 (series 16; classic) ubuntu/18.04 (amd64) linux/4.15.0-48-generic.
May 10 13:20:00 xenial-Virtual-Machine systemd[1]: Started Snappy daemon.
May 10 13:20:46 xenial-Virtual-Machine snapd[566]: api.go:1071: Installing snap "vscode" revision unset

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: snapd 2.38+18.04
ProcVersionSignature: Ubuntu 4.15.0-48.51-generic 4.15.18
Uname: Linux 4.15.0-48-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.6
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Fri May 10 13:21:25 2019
ProcEnviron:
 LANGUAGE=en_AU:en
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_AU.UTF-8
 SHELL=/bin/bash
SourcePackage: snapd
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Peter Goodall (pjgoodall) wrote :
Peter Goodall (pjgoodall) wrote :

To my limited understanding system appears to have not completed its start:

peterg@xenial-Virtual-Machine:~$ systemd-analyze blame
Bootup is not yet finished. Please try again later.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in snapd (Ubuntu):
status: New → Confirmed

'systemd-analyze blame' suggests to run 'systemctl list-jobs', this is its output:

JOB UNIT TYPE STATE
148 snapd.autoimport.service start waiting
 89 systemd-update-utmp-runlevel.service start waiting
  2 multi-user.target start waiting
129 snapd.seeded.service start running
  1 graphical.target start waiting

5 jobs listed.

After a long and fruitless digging around systemd unit and seed.loaded I have removed (purged) and installed snapd package and it fixed this issue for me.

Something along the lines of:
$ sudo apt purge snapd
$ sudo apt install snapd
$ sudo snap install core

Doug Lincicome (turbodoug) wrote :

I've had the same problem and purging/re-installing via apt package manager is the only way I could figure out to fix it.

Dan Pam (dm1795) wrote :

This is a pretty bad first experience for 19.04 gallery image as its supposed to be 'out of the box'. Must be lots of people trying this first time in hyper-v gallery and giving up. Took few hours to find this bug post. I can verify that the #5 commands corrected problem. purge was needed. uninstall didn't work.

Peter Feerick (pfeerick) wrote :

I can confirm this bug is still present in the optimised 18.04.2 image, and also verify that the commands from post #5 above worked for me to fix it. I actually had to purge snapd twice, as the first time it failed due to being unable to delete a folder with `rm` since it was a folder, not a file. `sudo rm -r /var/cache/snapd/aux` fixed it for me, and then the second purge completed properly. I had no issues after that with reinstalling snap or the core snap package, or a subsequent dist-upgrade, which had failed completely on a previous spinup of the 18.04.2 LTS image.

John Lenton (chipaca) wrote :

I suspect these images have a broken seed.yaml. Could somebody attach that?

Daniel Llewellyn (diddledan) wrote :

@chipaca: seed.yaml from the 18.04.2 image in the Hyper-V Gallery

$ cat /var/lib/snapd/seed/seed.yaml
snaps:
  -
    name: core
    channel: stable
    file: core_6673.snap
  -
    name: gnome-3-26-1604
    channel: stable/ubuntu-18.04
    file: gnome-3-26-1604_82.snap
  -
    name: gnome-calculator
    channel: stable/ubuntu-18.04
    file: gnome-calculator_352.snap
  -
    name: gnome-characters
    channel: stable/ubuntu-18.04
    file: gnome-characters_206.snap
  -
    name: gnome-logs
    channel: stable/ubuntu-18.04
    file: gnome-logs_57.snap
  -
    name: gnome-system-monitor
    channel: stable/ubuntu-18.04
    file: gnome-system-monitor_70.snap
  -
    name: gtk-common-themes
    channel: stable/ubuntu-18.04
    file: gtk-common-themes_1198.snap

John Lenton (chipaca) wrote :

yeah. That seed.yaml is missing core18, and those snaps need it AFAIK.

John Lenton (chipaca) wrote :

To confirm, (from memory), sudo snap info --verbose /var/lib/snapd/seed/snaps/*.snap | grep base:

Daniel Llewellyn (diddledan) wrote :

followup:

$ sudo snap info --verbose /var/lib/snapd/seed/snaps/*.snap | grep base:
[sudo] password for dllewellyn:
base: core18
base: core18
base: core18
base: core18

John Lenton (chipaca) wrote :

Yeah. The images are broken, because they ship a broken seed.yaml. Who creates the images? That's the project this bug needs to be assigned to.

John Lenton (chipaca) on 2019-07-12
affects: snapd (Ubuntu) → livecd-rootfs (Ubuntu)
Michael Vogt (mvo) wrote :

I think we need to add validation of the seed.yaml to livecd-rootfs:
https://code.launchpad.net/~mvo/livecd-rootfs/+git/livecd-rootfs/+merge/370065

We also need to do more validation in "snap debug validate-seed":
https://github.com/snapcore/snapd/pull/7105

Peter Goodall (pjgoodall) wrote :

I have no understanding of the test process used to check a system livecd build, but I imagine installing a snap that confirmed itself to the test-system would be useful on each platform.

Tobias Koch (tobijk) wrote :

I ran the following test on the current version of livecd-rootfs for Bionic:

https://pastebin.ubuntu.com/p/wQknctr6ys/

with a positive result:

https://pastebin.ubuntu.com/p/N8PXRJJV9b/

So I think in principle it should work, but maybe there are silent download failures. We definitely need mvo's sanity checks on seed.yaml.

Tobias Koch (tobijk) wrote :

My mistake that I didn't do proper error checking while determining the base of a snap. This here

https://code.launchpad.net/~tobijk/livecd-rootfs/+git/livecd-rootfs/+merge/370096

will hopefully prevent this from happening again.

Peter Goodall (pjgoodall) wrote :

Thanks everyone!

tags: added: rls-bb-incoming
Changed in livecd-rootfs (Ubuntu):
importance: Undecided → High
tags: added: rls-dd-incoming
Changed in livecd-rootfs (Ubuntu):
importance: High → Critical
Michael Vogt (mvo) wrote :

Just as a heads up: the seed.yaml for the azure image also need gnome-3-28-1804 in the seed. The `snap debug validate-image` will validate all this once its landed so hopefully this will at least be easy to detect fix (build-time instead of runtime).

Tobias Koch (tobijk) wrote :

Maybe also useful: modifications to snap-tool to make it more resilient to connection failures of all sorts:

https://code.launchpad.net/~tobijk/livecd-rootfs/+git/livecd-rootfs/+merge/370139

Tobias Koch (tobijk) wrote :
description: updated
Tobias Koch (tobijk) wrote :

The attachment "Backport for Bionic" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch

Hello Peter, or anyone else affected,

Accepted livecd-rootfs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.525.28 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in livecd-rootfs (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
Changed in livecd-rootfs (Ubuntu Disco):
status: New → Fix Committed
tags: added: verification-needed-disco
Andy Whitcroft (apw) wrote :

Hello Peter, or anyone else affected,

Accepted livecd-rootfs into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.578.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted livecd-rootfs (2.578.6) for disco have finished running.
There have been regressions in tests triggered by the package. Please visit the sru report page and investigate the failures.

https://people.canonical.com/~ubuntu-archive/pending-sru.html#disco

Iain Lane (laney) wrote :

Comment #27 is not correct.

I tested a bionic desktop ISO built with this livecd-rootfs and built successfully and got core, so snaps work.

Not sure if that is sufficient validation for this bug, so I won't (yet) change the tags.

Tobias Koch (tobijk) wrote :

I believe that comment #27 is correct, because after that the autopkgtests for ubuntu-image succeeded (triggered by a qemu update) using the previous revision of livecd-rootfs on disco. But I'm not able to tell how the changes made to livecd-rootfs affect the operation/testing of ubuntu-image.

Iain Lane (laney) wrote :

I should have been more explicit. When I wrote comment #28, the tests *had not finished running yet* so it was false at that point.

I don't know about this regression, let me retry it and if it happens again, please investigate.

Iain Lane (laney) wrote :

It passed now.

FYI: Comment #8 (with comment #5) fixed the problem for me :-)

Tobias Koch (tobijk) wrote :

I have verified (bionic 2.525.28 and disco 2.578.6) that the changes to snap_preseed do not break existing functionality and that an error is thrown when the helper _snap_preseed is called with an invalid snap name, resulting in snap-info failing to retrieve the base.

tags: added: verification-done-bionic verification-done-disco
removed: verification-needed-bionic verification-needed-disco
tags: removed: rls-bb-incoming rls-dd-incoming
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.525.28

---------------
livecd-rootfs (2.525.28) bionic; urgency=medium

  [ Tobias Koch ]
  * Do proper error checking when calling snap-tool info to determine
    the base of a snap. (LP: #1828500)

  [ Michael Vogt ]
  * Run "snap debug validate-seed" during preseeding to do basic validation of
    the generated seed.yaml

  [ Iain Lane ]
  * Seed core for non minimized builds, as it is still required (LP:
    #1836594).

 -- Iain Lane <email address hidden> Tue, 16 Jul 2019 13:20:18 +0100

Changed in livecd-rootfs (Ubuntu Bionic):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.578.7

---------------
livecd-rootfs (2.578.7) disco; urgency=medium

  [ Tobias Koch ]
  * Add retry logic to snap-tool to make downloads more resilient.
    (LP: #1837871)

 -- Tobias Koch <email address hidden> Mon, 26 Aug 2019 13:36:36 +0200

Changed in livecd-rootfs (Ubuntu Disco):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers