bionic-live-server images stuck seeding snapd at early boot

Bug #1883156 reported by Paride Legovini
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu CD Images
Fix Released
Undecided
Unassigned
subiquity
Invalid
Undecided
Unassigned
livecd-rootfs (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
High
Michael Hudson-Doyle

Bug Description

[impact]
The bionic-live-server-amd64 images get stuck at early boot of the installer system waiting for snapd to be seeded. The seeding never happens. The problem can be easily reproduced by running:

  kvm -m 2048 -boot d -cdrom ./bionic-live-server-amd64.iso

on a recent pending image. I just tested it with:

Ubuntu-Server 18.04.4 LTS "Bionic Beaver" - Release amd64 (20200610)

Note that the problem happens early, before subiquity sets up the network, so I don't think it's a problem accessing the store.

[test case]
boot the live server ISO successfully

[regression potential]
We should check that we don't regress the other flavours that seed snaps: ubuntu desktop, budge and mate. I think the seed.yaml in the main squashfs will be unchanged, which will be easy to check.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

I wonder if this is because subiquity & lxd have switched to core18 base.

Changed in livecd-rootfs (Ubuntu):
status: New → Fix Released
Changed in livecd-rootfs (Ubuntu Bionic):
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

So on bionic:
- filesystem layer has core snap seeded, yet it's unused by anything (ie. no lxd seeded)

- installer layer has core+subiquity snaps seeded, yet subiquity needs core18 snap, and ideally snapd snap.

I think we need to backport lots of snapseeding related changes to bionic.

Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

There's nothing to change in subiquity here is there? I guess we _could_ still have base: core for 18.04 but well. Let's not do that :)

Changed in livecd-rootfs (Ubuntu Bionic):
milestone: none → ubuntu-18.04.5
tags: added: id-5f15f568604db45e7b7e52cf
Changed in subiquity:
status: New → Invalid
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :
Changed in livecd-rootfs (Ubuntu Bionic):
status: Confirmed → In Progress
assignee: nobody → Michael Hudson-Doyle (mwhudson)
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Paride, or anyone else affected,

Accepted livecd-rootfs into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.525.45 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in livecd-rootfs (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Paride Legovini (paride) wrote :

Hi,

While the new livecd-rootfs unblocked the snapd seeding, it seems that it triggered another problem later in the subiquity install process.

The 20200729.1 bionic-live-server-amd64 image now fails to install because /media/filesystem/var/lib/snapd/seed/seed.yaml is missing. The full install logs are attached, however the interesting bits should be at [1].

The MP [2] with the tentative fix for this bug does touch how seed.yaml is generated, so the two things are likely to be related.

Big warning: this issue is NOT hit by the by automated ISO test as they skip the SnapList subiquity step because of [3]. This means that images are getting promoted but they are to be considered broken.

[1] https://paste.ubuntu.com/p/tYCM4bx6yq/
[2] https://code.launchpad.net/~mwhudson/livecd-rootfs/+git/livecd-rootfs/+merge/387802
[2] https://bugs.launchpad.net/subiquity/+bug/1864868

Changed in livecd-rootfs (Ubuntu Bionic):
status: Fix Committed → New
Revision history for this message
Paride Legovini (paride) wrote :

Note: the bionic-live-server 20200729.1 images have been built with -proposed enabled. Build log:

https://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu-server/bionic/daily-live-20200729.1.log

Revision history for this message
Ubuntu QA Website (ubuntuqa) wrote :

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1883156

tags: added: iso-testing
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

So, livecd-rootfs changes look good.

The newly reported subiquity issue, is of completely different nature. And I will open a new bug about it.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

One option is to fix subiquity to not assume that filesystem has anything seeded.

Another option is to provide an empty seed.yaml in the filesystem layer.

Or seed something into the filesystem layer.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :
Revision history for this message
Paride Legovini (paride) wrote :

OK, I set this bug back to Fix Committed, thanks for looking into this.

I like the idea of patching subiquity so it doesn't assume a seed.yaml is there, as it's not a broken configuration by itself.

Changed in livecd-rootfs (Ubuntu Bionic):
status: New → Fix Committed
Revision history for this message
Paride Legovini (paride) wrote :

I tested an ISO image with bionic-proposed enabled [1] and I can confirm livecd-rootfs 2.525.45 fixes this bug.

[1] https://people.canonical.com/~ubuntu-archive/cd-build-logs/ubuntu-server/bionic/daily-live-20200729.1.log

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package livecd-rootfs - 2.525.45

---------------
livecd-rootfs (2.525.45) bionic; urgency=medium

  [ Francis Ginther ]
  * Use the linux-raspi-hwe-18.04 kernel for raspi3 classic image builds.
    (LP: #1889188)

  [ Michael Hudson-Doyle ]
  * Backport snap seeding and other fixes for live-server ISOs. (LP: #1883156)
  * Stop building and mounting the mass-rack and maas-region squashfses for
    the server-live build as subiquity does not use them any more.

  [ Dimitri John Ledkov ]
  * Run snap info on the downloaded snap, rather than against the
    store. First of all snap info doesn't have --channel argument, thus
    queries the wrong channel, and depening on the cohort, a different
    snap might be visible too. Thus seed the base of the snap revision we
    dowanloaded, rather than some random one from the store.
  * Use snap-tool to seed subiquity snap.
  * Subiquity: install linux-firmware in the installer layer, to make
    firmware blobs available in the live session. LP: #1847835

  [ Robert C Jennings ]
  * Fix logic to ensure snapd is seeded in core18-only images (LP: #1871919)

 -- Michael Hudson-Doyle <email address hidden> Wed, 22 Jul 2020 14:44:38 +1200

Changed in livecd-rootfs (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Norbert (nrbrtx)
Changed in ubuntu-cdimage:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.