seeded snaps fail to launch after OEM install

[Jeremy Bícha]

I just did an OEM install of the Ubuntu 22.04.1 Desktop candidate.

I used the instructions at
http://iso.qa.ubuntu.com/qatracker/milestones/437/builds/254993/testcases/1305/results

I was unable to start Firefox. Here is the command line output.

$ firefox
sed: can't read /home/jeremy/.config/user-dirs.dirs: Permission denied
/snap/firefox/1635/snap/command-chain/desktop-launch: line 261: /home/jeremy/.config/user-dirs.dirs: Permission denied
cp: cannot open '/home/jeremy/.config/user-dirs.locale' for reading: Permission denied
/snap/firefox/1635/snap/command-chain/desktop-launch: line 266: /home/jeremy/.config/user-dirs.locale: Permission denied
Error: cannot open display: :0

This is strange because I was able to launch Firefox after completing a non-OEM install.

/etc/passwd shows
jeremy:x:1000:1000:Jeremy Bicha,,,:/home/jeremy:/bin/bash

And my user-dirs.dirs looks normal
-rw------- 1 jeremy jeremy 633 Aug 3 20:09 user-dirs.dirs

snap-store
==========
I have the same problem with running the snap-store

$ snap-store
sed: can't read /home/jeremy/.config/user-dirs.dirs: Permission denied
/snap/snap-store/582/snap/command-chain/desktop-launch: line 261: /home/jeremy/.config/user-dirs.dirs: Permission denied
cp: cannot open '/home/jeremy/.config/user-dirs.locale' for reading: Permission denied
/snap/snap-store/582/snap/command-chain/desktop-launch: line 266: /home/jeremy/.config/user-dirs.locale: Permission denied
Warning: Schema \u201corg.gnome.system.locale\u201d has path \u201c/system/locale/\u201d. Paths starting with \u201c/apps/\u201d, \u201c/desktop/\u201d or \u201c/system/\u201d are deprecated.
Warning: Schema \u201corg.gnome.system.proxy\u201d has path \u201c/system/proxy/\u201d. Paths starting with \u201c/apps/\u201d, \u201c/desktop/\u201d or \u201c/system/\u201d are deprecated.
Warning: Schema \u201corg.gnome.system.proxy.http\u201d has path \u201c/system/proxy/http/\u201d. Paths starting with \u201c/apps/\u201d, \u201c/desktop/\u201d or \u201c/system/\u201d are deprecated.
Warning: Schema \u201corg.gnome.system.proxy.https\u201d has path \u201c/system/proxy/https/\u201d. Paths starting with \u201c/apps/\u201d, \u201c/desktop/\u201d or \u201c/system/\u201d are deprecated.
Warning: Schema \u201corg.gnome.system.proxy.ftp\u201d has path \u201c/system/proxy/ftp/\u201d. Paths starting with \u201c/apps/\u201d, \u201c/desktop/\u201d or \u201c/system/\u201d are deprecated.
Warning: Schema \u201corg.gnome.system.proxy.socks\u201d has path \u201c/system/proxy/socks/\u201d. Paths starting with \u201c/apps/\u201d, \u201c/desktop/\u201d or \u201c/system/\u201d are deprecated.
00:40:12:0952 GLib-GIO g_app_info_get_name: assertion 'G_IS_APP_INFO (appinfo)' failed
00:40:12:0961 Gtk cannot open display: :0

[Jeremy Bícha (jbicha)]

This works:

snap install ohmygiraffe && snap run ohmygiraffe

snap install chromium && snap run chromium

So then I tried:
snap refresh --channel=beta firefox
snap refresh --channel=latest/stable/ubuntu-22.04 firefox
to reinstall the snap.

Now it works fine. The snap-store also works fine after I use the similar trick to reinstall it.

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1983528

[Jeremy Bícha (jbicha)]

I did not experience this bug when I tried with the original 22.04 LTS OEM install. I did not install updates in the OEM stage. Some security updates were probably installed since I chose the "Install updates" option in the installer.

Here's the temporary OEM user's account details:

oem:x:29999:29999:OEM Configuration (temporary user),,,:/home/oem:/bin/bash

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Aaron Rainbolt (arraybolt3)]

Successfully reproduced bug on latest Ubuntu 22.04.1 ISO.

[Simon Quigley (tsimonq2)]

[ 329.424135] audit: type=1400 audit(1659575973.283:88): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/home/user/.config/user-dirs.dirs" pid=35369 comm="sed" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 329.425766] audit: type=1400 audit(1659575973.283:89): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/home/user/.config/user-dirs.dirs" pid=35370 comm="desktop-launch" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 329.428821] audit: type=1400 audit(1659575973.287:90): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/home/user/.config/user-dirs.locale" pid=35371 comm="cp" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 329.430225] audit: type=1400 audit(1659575973.287:91): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/home/user/.config/user-dirs.locale" pid=35372 comm="desktop-launch" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 329.519008] audit: type=1400 audit(1659575973.379:92): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/usr/share/icons/" pid=35312 comm="desktop-launch" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 329.519383] audit: type=1400 audit(1659575973.379:93): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/var/lib/snapd/desktop/icons/" pid=35312 comm="desktop-launch" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 329.697600] audit: type=1400 audit(1659575973.555:94): apparmor="DENIED" operation="capable" profile="snap.firefox.firefox" pid=35312 comm="firefox" capability=21 capname="sys_admin"
[ 344.522960] audit: type=1400 audit(1659575988.383:95): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/home/user/.config/user-dirs.dirs" pid=35504 comm="sed" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 344.523850] audit: type=1400 audit(1659575988.383:96): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/home/user/.config/user-dirs.dirs" pid=35505 comm="desktop-launch" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 344.526464] audit: type=1400 audit(1659575988.383:97): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/home/user/.config/user-dirs.locale" pid=35506 comm="cp" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 344.527433] audit: type=1400 audit(1659575988.387:98): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/home/user/.config/user-dirs.locale" pid=35507 comm="desktop-launch" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 344.586773] audit: type=1400 audit(1659575988.447:99): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/usr/share/icons/" pid=35460 comm="desktop-launch" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 344.586908] audit: type=1400 audit(1659575988.447:100): apparmor="DENIED" operation="open" profile="snap.firefox.firefox" name="/var/lib/snapd/desktop/icons/" pid=35460 comm="desktop-launch" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

[Jeremy Bícha (jbicha)]

I can also duplicate with the original 22.04 LTS OEM install.

In the OEM stage:
sudo apt update
sudo apt dist-upgrade
reboot
snap refresh

[Brian Murray (brian-murray)]

As another point of information after performing a Kubuntu OEM install of a candidate 22.04.1 image dated 20220801 the firefox snap launched without any issues.

[James Henstridge (jamesh)]

One thing to help debug would be to run "snap changes" on the system when its in the broken state right after seeding.

If you can see the change where it tried and failed to install the seed snaps, run "snap change $id" to get a log of what it tried to do and where it failed.

[Steve Langasek (vorlon)]

The breakage happens before oem-config has run. Switching to a terminal after second reboot (ubiquity->oem-firstboot->oem-config), 'snap changes' shows nothing new, but 'snap connections' returns empty(!) and /var/lib/snapd/apparmor/profiles/snap.firefox.firefox is in a broken state.

[Jeremy Bícha (jbicha)]

Here's a snap changes log from my broken system that I updated from 22.04 in the OEM stage.

[Steve Langasek (vorlon)]

/var/lib/snapd/state.json from the broken system.

[James Henstridge (jamesh)]

Attached is the journalctl log from a VM booted with the OEM. In particular, while snapd is starting, it can't find the mounted snaps:

    snapmgr.go:363: cannot read snap info of snap "core20" at revision 1587: cannot find installed snap "core20" at revision 1587: missing file /snap/core20/1587/meta/snap.yaml
    snapmgr.go:363: cannot read snap info of snap "firefox" at revision 1635: cannot find installed snap "firefox" at revision 1635: missing file /snap/firefox/1635/meta/snap.yaml

... and later decides to clean up the connections

    helpers.go:128: cannot add snap "core20" to interface repository: snap is broken: cannot find installed snap "core20" at revision 1587: missing file /snap/core20/1587/meta/snap.yaml
    helpers.go:128: cannot add snap "firefox" to interface repository: snap is broken: cannot find installed snap "firefox" at revision 1635: missing file /snap/firefox/1635/meta/snap.yaml
    ...
    helpers.go:268: removed stale connections: snapd-desktop-integration:desktop core:desktop, snapd-desktop-integration:snap-themes-control core:snap-themes-control, firefox:x11 core:x11, firefox:screen-inhibit-control core:screen-inhibit-control, snap-store:fwupd core:fwupd, firefox:browser-sandbox core:browser-support, firefox:desktop core:desktop, snap-store:network-status core:network-status, firefox:gsettings core:gsettings, snapd-desktop-integration:wayland core:wayland, firefox:desktop-legacy core:desktop-legacy, firefox:wayland core:wayland, snap-store:appstream-metadata core:appstream-metadata, snap-store:opengl core:opengl, snap-store:gsettings core:gsettings, snap-store:snapd-control core:snapd-control, firefox:u2f-devices core:u2f-devices, firefox:avahi-observe core:avahi-observe, firefox:hardware-observe core:hardware-observe, firefox:etc-firefox-policies core:system-files, firefox:audio-record core:audio-record, firefox:network core:network, firefox:upower-observe core:upower-observe, snap-store:upower-observe core:upower-observe, firefox:camera core:camera, firefox:home core:home, firefox:network-bind core:network-bind, snapd-desktop-integration:desktop-legacy core:desktop-legacy, snap-store:x11 core:x11, firefox:audio-playback core:audio-playback, firefox:removable-media core:removable-media, firefox:cups-control core:cups-control, snap-store:network core:network, snapd-desktop-integration:opengl core:opengl, firefox:joystick core:joystick, firefox:system-packages-doc core:system-packages-doc, firefox:opengl core:opengl, snap-store:desktop-legacy core:desktop-legacy, snap-store:system-observe core:system-observe, snapd-desktop-integration:x11 core:x11, firefox:unity7 core:unity7, snap-store:desktop core:desktop, snap-store:password-manager-service core:password-manager-service, snap-store:packagekit-control core:packagekit-control, firefox:dot-mozilla-firefox core:personal-files, snapd-desktop-integration:gsettings core:gsettings, snap-store:hostfs-usr-share-applications core:system-files, snap-store:wayland core:wayland

[James Henstridge (jamesh)]

From IRC conversation, this was one hypothesis:

1. when the system boots for OEM configuration, it boots to the oem-config.target systemd target rather than multi-user.target.

2. the systemd mount units that snapd generates only set WantedBy=multi-user.target, so they are not started at this point. While they set Before=snapd.service, that only orders things if both the mount unit and snapd.service are to be started.

3. The Ubuntu desktop now ships the snapd-desktop-integration snap, which includes a user session service. This is active during the OEM config desktop session.

4. the snapd-desktop-integration user session service hits the /run/snapd-snap.socket REST API socket to check whether the snaps corresponding to the current desktop theme are installed, which will cause snapd to start without the mount units (so the Before= directive does nothing).

There's an obvious hole in this hypothesis though: if the snaps haven't been mounted, how could snapd-desktop-integration even start?

Could the mount units have been shut down after the user session service started?

[Alex Murray (alexmurray)]

If the snapd mount units use `WantedBy=multi-user.target default.target` (or perhaps just default.target) this seems to be sufficient to resolve this.

To test:

Boot in OEM install mode, run the installer and then reboot back into the installed system live environment
Edit each of the mount units to also specify default.target and make sure these get enabled to install the new symlinks

sudo sed -i "s/WantedBy=multi-user.target/WantedBy=multi-user.target default.target/" /etc/systemd/system/snap*.mount
for m in /etc/systemd/system/snap*.mount; do sudo systemctl enable $(basename $m); done

# you should now see a heap of output saying symlink were created for the default.target.wants

Then run 'Prepare for shipping to end user' to update the default boot target to oem-config.target and reboot as normal

[Alberto Mardegan (mardy)]

I'm working on this in https://github.com/snapcore/snapd/pull/12011

[Łukasz Zemczak (sil2100)]

So to understand correctly: the reason for this to be a problem *only* on our Ubuntu Desktop images (and not Kubuntu images, for instance) is that we ship snapd-desktop-integration?

[Alberto Mardegan (mardy)]

@Lukasz: yes, and the analysis by James in comment #14 seems correct (though it's still unclear how the snapd-desktop-integration service, whose snap has not been mounted, is able to trigger the snapd startup).

[James Henstridge (jamesh)]

I posted these on IRC, but in case they help others here are the full journals from my test install:

System journal: https://paste.ubuntu.com/p/XcySs8tJmb/ (includes both before and after reboot).

OEM user journal: https://paste.ubuntu.com/p/8fpGmf99r3/ (i.e. everything uid 29999 was doing before it was deleted).

User journal: https://paste.ubuntu.com/p/RgSNcJYMG5/

[Alberto Mardegan (mardy)]

I ran some tests to understand what could have triggered the snapd activation, and I believe that snapd-desktop-integration is not responsible for this. What I did:

    sudo systemctl stop snapd.service
    # At this point the snapd socket is still active and managed by systemd

    # Now simulate the case where the mount units have not been loaded
    sudo umount /snap/snapd-desktop-integration/14

    systemctl --user start snap.snapd-desktop-integration.snapd-desktop-integration.service

After running this, snapd is still not started.

Still, it's very likely that snapd was activated by a connection on its socket (probably via snapctl), but it's not something easy to establish (I looked at systemd options for socket unit files, and I didn't find a way to increase verbosity). But maybe it's not even something that we need to know: the idea that snapd could be woken up at random moments is always a possibility, and we should cope with it. What is critical here is that snapd removes all the snap connections because it sees that the snaps are not mounted, so it assumed that they have been removed.

The solution in the PR linked above works by activating all the snap mount units when the default boot target is reached, and should help in this case. Another option, which I'm going to investigate, is making snapd not disconnect all the connections if the mount unit is not active but the corresponding snap binary file still exists in /var/lib/snapd/snaps/.

[Alberto Mardegan (mardy)]

While we make the snapd release (it will hopefully happen on Monday), it would be nice if someone is able (or teaches me how to do it) generate a new OEM image including the snapd from the latest/edge channel, so that we can be sure that the workaround works.

[Steve Langasek (vorlon)]

Not at all trivial to build an image with the snapd from the edge channel, we simply don't have support for channel selection for classic images in the image build scripts.

Instead I've done the following:

 - boot the image in OEM install mode
 - install
 - reboot
 - in a terminal, run:
 - for snap in snapd-desktop-integration snap-store firefox gnome-3-38-2004 gtk-common-themes core20 bare snapd; do snap remove $snap; done
 - snap install --channel=edge snapd
 - for snap in gtk-common-themes gnome-3-38-2004 snapd-desktop-integration snap-store firefox; do snap install --channel=latest/stable/ubuntu-22.04 $snap; done
 - verify all snap mount units (except for the one for snapd!) have WantedBy=default.target multi-user.target
 - complete oem-config-prepare
 - reboot, complete oem-config
 - launch firefox

Firefox fails to launch. I believe this is because the snapd snap's WantedBy is set by the snapd deb which does not have this fix, and we need a fixed snapd in the deb as well as in the snap.

If after the 'snap install --channel=edge snapd' I additionally do:
  cp /snap/snapd/current/usr/lib/snapd/snapd /usr/lib/snapd/snapd
  snap remove snapd
  snap install --channel=edge snapd

then *all* of the mount units in /etc/systemd/system have the expected WantedBy, all of the connections are present after oem-config, and launching 'firefox' succeeds.

[Alberto Mardegan (mardy)]

Hi Steve, and thanks for trying this!

I'm not sure that the deb package is needed; I suspect that the manual update you did occurred too late in the process, when part of the damage was already done. If my understanding is correct (also using the system logs that James posted in comment #19), what happens is:

1. System boots, with snapd 2.55 installed as a deb, and 2.56 installed as a snap. I guess that at this point the boot target is "multi-user", because:
2. "Aug 04 11:07:29 [...] Mounting Mount unit for snapd, revision 16292..." (the snapd snap gets mounted)
3. System reboots, the boot target is oem-config
4. All snap mount units are *not* loaded
5. The snapd startup is triggered
6. Since the mount unit for snapd was not loaded, the deb version of the snapd is run
7. snapd cannot read the yaml file for the installed snaps, and marks them as invalid
8. snapd disconnects all the snap connections

My understanding is that in your test you intervened after point 3. Your installation of the newer snapd and the reinstallation of all the snaps fixed the "WantedBy" line, but by that time the boot target was already set. Since the commands you entered obviously required a running snapd, it means that points 5 to 8 described above have also been executed before the snaps had been reinstalled.

It's not clear to me why the reinstalling of the snaps did not help to restore the connections (if you still have the logs, they could be helpful), but my main point is that, with a fixed snapd, point 4 above would not happen, therefore the snapd deb would not enter into the picture. But again, mine is just speculation, that's why having an image with the newest snapd built-in would be helpful.

[Alex Murray (alexmurray)]

Alberto, I think an updated deb of snapd is needed otherwise this will still be broken for offline installations (ie when there is no network available to refresh to a newer snapd snap).

[Łukasz Zemczak (sil2100)]

Hello Jeremy, or anyone else affected,

Accepted snapd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snapd/2.56.2+22.04ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Alberto Mardegan (mardy)]

Hi Lukasz and all,
  I tried today's daily image which contains the snapd snap from the edge channel (but not the .deb package), and while the WantedBy line in the mount units is now updated and lists the default.target, still the problem persists. I'm attaching the syslog.

My impression is that the "OEM Configuration" target is never reached (I don't see any line from systemd about it). Maybe, as Dmitry said in the PR, we should not use default.target but something else (like local-fs).

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted snapd (2.56.2+22.04ubuntu1) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

systemd/249.11-0ubuntu3.4 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#snapd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Jeremy Bícha (jbicha)]

I did an OEM install with today's Ubuntu Desktop candidate image 20220808.3 which includes snapd 2.56.2+22.04ubuntu1

I was able to run both Firefox and the Snap Store at the end of the process.
I did a second install without Internet and I was able to run both Firefox and the Snap Store at the end successfully too.

[Aaron Rainbolt (arraybolt3)]

I also did an OEM install with the same ISO as jbicha - Firefox and Snap Store both started out-of-the-box, and I was able to browse Reddit with Firefox without problems. This was in a GNOME Boxes VM with SeaBIOS and Internet access.

[Alberto Mardegan (mardy)]

I replayed the whole flow with the same image (20220808.3) and indeed the bug does not occur. The snap connections are preserved, I still don't see a line from systemd about the OEM Configuration target being reached, but there's a line about activating it:

    systemd[1]: Queued start job for default target OEM Configuration.

It's probably normal this way.

[Łukasz Zemczak (sil2100)]

Seeing that many people mention being able to get a functioning desktop after OEM install, I'm marking this as verified. Thank you!

[Łukasz Zemczak (sil2100)]

Kinetic will be released as part of the next snapd 2.57 release.

[Launchpad Janitor (janitor)]

This bug was fixed in the package snapd - 2.56.2+22.04ubuntu1

---------------
snapd (2.56.2+22.04ubuntu1) jammy; urgency=medium

  * Cherry pick https://github.com/snapcore/snapd/pull/12011
    to fix launching snaps OEM install (LP: #1983528)

snapd (2.56.2+22.04) jammy; urgency=medium

  * New upstream release, LP: #1974147
    - o/snapstate: exclude services from refresh app awareness hard
      running check
    - cmd/snap: support custom apparmor features dir with snap
      prepare-image

 -- Michael Vogt <email address hidden> Mon, 08 Aug 2022 11:17:19 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for snapd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package snapd - 2.57.1+22.10

---------------
snapd (2.57.1+22.10) kinetic; urgency=medium

  * New upstream release, LP: #1983035
    - cmd/snap-update-ns: handle mountpoint removal failures with EBUSY
    - cmd/snap-update-ns: print current mount entries
    - cmd/snap-update-ns: check the unused mounts with a cleaned path
    - snap-confine: disable -Werror=array-bounds in __overflow tests to
      fix build error on Ubuntu 22.10
    - systemd: add `WantedBy=default.target` to snap mount units
      (LP: #1983528)

 -- <email address hidden> (Samuele Pedroni (Canonical Services Ltd.)) Wed, 10 Aug 2022 09:30:50 +0300