snapd fails to process corrupted seed.yaml on Ubuntu 19.04

[Doug McMahon]

Fresh install on latest daily 19.04 image (04/17/2019

Ex.
$ snap list
error: cannot list snaps: cannot communicate with server: Get http://localhost/v2/snaps: dial unix /run/snapd.socket: connect: connection refused

ProblemType: Bug
DistroRelease: Ubuntu 19.04
Package: snapd 2.38+19.04
ProcVersionSignature: Ubuntu 5.0.0-13.14-generic 5.0.6
Uname: Linux 5.0.0-13-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.10-0ubuntu27
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Thu Apr 18 18:21:11 2019
InstallationDate: Installed on 2019-04-17 (1 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Alpha amd64 (20190413)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: snapd
UpgradeStatus: No upgrade log present (probably fresh install)

[John Lenton (chipaca)]

What's the output of "journalctl --system -u snapd"?

[Doug McMahon (mc3man)]

Attaching output
(edited to just oe "reboot" as it's the exact same per each reboot

[Zygmunt Krynicki (zyga)]

Thank you for providing the service logs. It seems snapd is crashing on a NULL pointer. I'm marking the bug as confirmed and bumping priority to critical.

[Zygmunt Krynicki (zyga)]

Can you please provide the contents of /var/lib/snapd/seed/seed.yaml as an attachment?

[Zygmunt Krynicki (zyga)]

I've started a tentative workaround for this issue under: https://github.com/snapcore/snapd/pull/6756

[Maciej Borzecki (maciek-borzecki)]

Keep in mind that apparently the ISO image is not the final relase one. The attached bug data has:

InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Alpha amd64 (20190413)

[Samuele Pedroni (pedronis)]

snapd should error on this, not panic, but seed.yaml is intended to be strict a format, nulls shouldn't be there

[Samuele Pedroni (pedronis)]

fwiw, at least the base seed in ubuntu-19.04-desktop-amd64.iso Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416) looks correct

[Doug McMahon (mc3man)]

Hmm, I thought this was the final, maybe should try again the install.
Anyway this is the requested file..

[Zygmunt Krynicki (zyga)]

HA

Thank you! This explains a lot.

So the short story is:

- the seed is broken
- snapd is confused by the seed and doesn't install the required snaps
- snap dependencies are not satisfied
- all hell breaks loose.

[Zygmunt Krynicki (zyga)]

Do you perhaps have logs from the system installer?

[Ken VanDine (ken-vandine)]

I can't reproduce this with the released image. Any idea what scenario it creates the broken seed?

[Zygmunt Krynicki (zyga)]

I hope it is this image only:

InstallationDate: Installed on 2019-04-17 (1 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Alpha amd64 (20190413)

[Maciej Borzecki (maciek-borzecki)]

I did a clean install with the Alpha image and did not notice any issues, snapd did not panic, seed yaml looked correct.

However, the format of seed yaml is somewhat weird, it looks like this (with empty line following -):

snaps:
 -
   name: foo
   file: bar.snap

If by any chance (a bug in the installer or any other piece), the seed file became this:

snaps:
 -
   name: foo
   file: bar.snap

 -

Then the problem would be triggered.

[Doug McMahon (mc3man)]

Tried once with current release mage, works ok.
So maybe it was just the image 2 day prior to release though that seems curious..

Maybe if time permits will try release image a couple of times to see if the issue arises.
For info sake the installer log's snapd mentions attached from 'bad' install.

[Michael Vogt (mvo)]

From what I understand the immediate crisis is over, correct? We had an incorrect image right before the release but for the final release things got fixed. I would love to know what exactly happend with the image and if that was an installer or an image builder issues. I think we need to improve the tooling there to ensure we don't write incorrect seed.yaml. I also thing snapd should error and not ignore this issue.

[Michael Vogt (mvo)]

I looked at this a bit more and the following change broke the installer:

https://git.launchpad.net/livecd-rootfs/commit/?id=f4fa240ed7a43ed927634d5829812e37955d4b4c

this was uploaded to livecd-rootfs at apr-12 so it ended up on the medium on 20190413.

The following commit fixed it:

https://git.launchpad.net/livecd-rootfs/commit/?id=39f2558b96a35a77988f762f941d1e5bf581f8a4

And indeed:
```
$ cat > /tmp/test-seed.yaml <<EOF
-
   name: core
   channel: foo
   file: bar
-
   name: snapd
   channel: stable/ubuntu-19.04
   file: snapd_941.snap
-
   name: gnome-foo
   channel: foo
   name: xxx
EOF
# broken sed command will produce exactly what we saw:
$ sed -e'/name: snapd/,+2d' /tmp/test-seed.yaml -
   name: core
   channel: foo
   file: bar
-
-
   name: gnome-foo
   channel: foo
   name: xxx
# and the fixed sed command actually fixes things:
$ sed -e'N;/name: snapd/,+2d' /tmp/test-seed.yaml
-
   name: core
   channel: foo
   file: bar
-
   name: gnome-foo
   channel: foo
   name: xxx
```

[Michael Vogt (mvo)]

I am adding a livecd-rootfs task - it would be nice if seed.yaml manipulation would add an extra check at the end. Ideally we should call a new "snap debug validate-seed <path>" during the build process. livecd-rootfs already build-depends on snapd so that should be ok, yes? The PR for this is https://github.com/snapcore/snapd/pull/6771

[Dimitri John Ledkov (xnox)]

On Wed, 24 Apr 2019 at 16:00, Michael Vogt <email address hidden> wrote:
>
> I am adding a livecd-rootfs task - it would be nice if seed.yaml
> manipulation would add an extra check at the end. Ideally we should call
> a new "snap debug validate-seed <path>" during the build process.
> livecd-rootfs already build-depends on snapd so that should be ok, yes?
> The PR for this is https://github.com/snapcore/snapd/pull/6771
>

Is my snap different to yours?

# snap debug validate-seed ./seed.yaml
error: unknown command "validate-seed", see 'snap help debug'.

--
Regards,

Dimitri.

[John Lenton (chipaca)]

Yes, mvo lives in a future in which he's already implemented that feature, whereas us mere mortals are stuck waiting for it to arrive.

[Dimitri John Ledkov (xnox)]

Oh come on! Avengers Endgame comes out in 15 minutes in the UK, don't spoil it for me mvo!

[John Lenton (chipaca)]

just don't read the commit messages and you'll be fine

[Michael Vogt (mvo)]

The snapd part of this has landed in https://github.com/snapcore/snapd/pull/6771 and will be part of the 2.39 upload.