Lenovo T490S and X390 GM do not have Firefox.App store install errors out

[Vamshee Paduru]

The new GM Images for T490S and X390 do not have Firefox preinstalled. When I try to install from the Appstore, it errors out "Firefox : too early for operation, device not yet seeded or device model not acknowledged."

If I install using apt-get install firefox it seemed to install fine.

[Bin Li (binli)]

It should be a snap package issue.

[Bin Li (binli)]

I tried the official 18.04.2, installed the 66.05(64bit) snap successfully from 'Ubuntu Software' 3.28.1.

[Bin Li (binli)]

And it could also start up correctly. :)

[Bin Li (binli)]

$ snap list
Name Version Rev Tracking Publisher Notes
core 16-2.38.1 6818 stable canonical✓ core
firefox 66.0.5-1 212 stable mozilla✓ -

[Gabriel Zhi Chen (gabrielzchen)]

I can reproduce the issue. Please refer to the picture Appstore.png

[Gabriel Zhi Chen (gabrielzchen)]

Image : sutton-mccarthy-bionic-amd64-20190426-96.iso

[Gabriel Zhi Chen (gabrielzchen)]

U@Test:~$ dpkg -l | grep ubuntu-software
ii ubuntu-software 3.28.1-0ubuntu4.18.04.9 all Utility for browsing, installing, and removing software

[Bin Li (binli)]

Found the root cause, the snapd seeded service was waiting.

$ systemctl status snapd.seeded.service
● snapd.seeded.service - Wait until snapd is fully seeded
   Loaded: loaded (/lib/systemd/system/snapd.seeded.service; enabled; vendor preset: enabled)
   Active: activating (start) since Mon 2019-05-20 02:49:56 EDT; 5min ago
 Main PID: 1432 (snap)
    Tasks: 15 (limit: 4915)
   CGroup: /system.slice/snapd.seeded.service
           └─1432 /usr/bin/snap wait system seed.loaded

May 20 02:49:56 u-ThinkPad-X390-Yoga systemd[1]: Starting Wait until snapd is fully seeded...

[Bin Li (binli)]

Tried the old oem-image, I could not reproduce this issue.

[Rex Tsai (chihchun)]

- The default search result of firefox from software center is a snap[1].
- The issue is the user want to install a snap before the snapd on the device is initialized and to be ready for install snaps.
- It's not directly related to oem-image, but a timing issue of snapd, the error is from snapd. Depends on the network environment, some times snapd takes a bit longer to talk to the Ubuntu snap Store.
- Eventually, the snapd will finish the seeding process and ready for install, unless the Ubuntu Store API is banned from that network. "snap changes" can show you the latest progress of the snapd.

[1] https://snapcraft.io/firefox

[Bin Li (binli)]

@Rex,

The snapd couldn't finished the initialize, even after several days. I think something block the system seed.loaded. Thanks!

$ snap changes
ID Status Spawn Ready Summary
1 Doing 5 days ago, at 03:39 EDT - Initialize system state

$ snap info core
name: core
summary: snapd runtime environment
publisher: Canonical✓
license: unset
description: |
  The core runtime environment for snapd
type: core
snap-id: 99T7MUlRhtI3U0QFgl5mXXESAiSwt776
tracking: stable
refresh-date: 5 days ago, at 03:40 EDT
channels:
  stable: 16-2.39 2019-05-22 (6964) 92MB -
  candidate: 16-2.39 2019-05-07 (6964) 92MB -
  beta: 16-2.39 2019-05-03 (6964) 92MB -
  edge: 16-2.39+git1309.5707258 2019-05-27 (7103) 92MB -
installed: 16-2.38.1 (6818) 93MB core

$ snap version
snap 2.38.1
snapd 2.38.1
series 16
ubuntu 18.04
kernel 4.15.0-1039-oem

[Bin Li (binli)]

-- Reboot --
May 27 04:30:55 u-ThinkPad-X390-Yoga systemd[1]: Starting Snappy daemon...
May 27 04:30:56 u-ThinkPad-X390-Yoga snapd[1066]: AppArmor status: apparmor is enabled and all features are available
May 27 04:30:56 u-ThinkPad-X390-Yoga snapd[1066]: AppArmor status: apparmor is enabled and all features are available
May 27 04:30:56 u-ThinkPad-X390-Yoga snapd[1066]: helpers.go:717: cannot retrieve info for snap "gnome-calculator": cannot find installed snap "gnome-calculator" at revision 406: missing file /snap/gnome-calculator/406/meta/snap.yaml
May 27 04:30:56 u-ThinkPad-X390-Yoga snapd[1066]: daemon.go:379: started snapd/2.38.1 (series 16; classic) ubuntu/18.04 (amd64) linux/4.15.0-1039-oem.
May 27 04:30:56 u-ThinkPad-X390-Yoga systemd[1]: Started Snappy daemon.

[Bin Li (binli)]

Tried below workaround, but it couldn't work for our case.

https://askubuntu.com/questions/1107388/snaps-wont-install-in-ubuntu-18-04

sudo apt install jq
sudo cp /var/lib/snapd/state.json /var/lib/snapd/state.json-old
sudo systemctl stop snapd
sudo cat /var/lib/snapd/state.json | jq 'delpaths([["data", "auth", "device"]])' > state.json-new
sudo cp state.json-new /var/lib/snapd/state.json
sudo systemctl start snapd

[Bin Li (binli)]

In bugsy-config, I found the base image had their own hook to install snaps, I doubt it's related to this issue. I just try to build a new image to verify it.

chroot_local-hooks/70-default-snaps

[Rex Tsai (chihchun)]

Hi @binli

In case of #13, please run "snap change 1" to see what went wrong while snapd is doing "Initialize system state".

[Bin Li (binli)]

@Rex,

 Found it's hanging in 'Ensure prerequisites for "gnome-calculator" are available'. And in #14 there are error about the gnome-calculator, so it should be related to this issue. Thanks!

$ snap change 1
Status Spawn Ready Summary
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:39 EDT Ensure prerequisites for "core" are available
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:39 EDT Prepare snap "/var/lib/snapd/seed/snaps/core_6818.snap" (6818)
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:39 EDT Mount snap "core" (6818)
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:39 EDT Copy snap "core" data
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Setup snap "core" (6818) security profiles
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Make snap "core" (6818) available to the system
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Automatically connect eligible plugs and slots of snap "core"
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Set automatic aliases for snap "core"
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Setup snap "core" aliases
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Run install hook of "core" snap if present
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Start snap "core" (6818) services
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Run configure hook of "core" snap if present
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Ensure prerequisites for "gnome-3-26-1604" are available
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Prepare snap "/var/lib/snapd/seed/snaps/gnome-3-26-1604_82.snap" (82)
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Mount snap "gnome-3-26-1604" (82)
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Copy snap "gnome-3-26-1604" data
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Setup snap "gnome-3-26-1604" (82) security profiles
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Make snap "gnome-3-26-1604" (82) available to the system
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Automatically connect eligible plugs and slots of snap "gnome-3-26-1604"
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Set automatic aliases for snap "gnome-3-26-1604"
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Setup snap "gnome-3-26-1604" aliases
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Run install hook of "gnome-3-26-1604" snap if present
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Start snap "gnome-3-26-1604" (82) services
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Run configure hook of "gnome-3-26-1604" snap if present
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Ensure prerequisites for "gtk-common-themes" are available
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Prepare snap "/var/lib/snapd/seed/snaps/gtk-common-themes_1198.snap" (1198)
Done 6 days ago, at 03:39 EDT 6 days ago, at 03:40 EDT Mount snap "gtk-common-themes" (1198)
Done 6 days ago,...

[Bin Li (binli)]

I removed the gnome-calculator from seeded-snaps, building a new image now.

[Bin Li (binli)]

@Rex,

I removed all the package in oem image, after install the new image, the snap works fine, and I installed the gnome-calculator, I found it will install the gnome-3-28-1804, not gnome-3-26-1604.

So the gnome-calculator (406) based on the gnome-3-26-1604 is missed or wrong, do you know how to check it?

One solution is that we just changed the gnome base package version.

BTW, it's a general issue for all the timbuktu image.

[Bin Li (binli)]

Another solution is remove the gnome-calculator from preinstalled package list.

[Bin Li (binli)]

Based on comment #19, I build image, and I found there are still problems. After I removed the gnome-calculator, it will hang at gnome-characters. :)

Done today at 04:46 EDT today at 04:46 EDT Mount snap "gtk-common-themes" (1198)
Done today at 04:46 EDT today at 04:46 EDT Copy snap "gtk-common-themes" data
Done today at 04:46 EDT today at 04:46 EDT Setup snap "gtk-common-themes" (1198) security profiles
Done today at 04:46 EDT today at 04:46 EDT Make snap "gtk-common-themes" (1198) available to the system
Done today at 04:46 EDT today at 04:46 EDT Automatically connect eligible plugs and slots of snap "gtk-common-themes"
Done today at 04:46 EDT today at 04:46 EDT Set automatic aliases for snap "gtk-common-themes"
Done today at 04:46 EDT today at 04:46 EDT Setup snap "gtk-common-themes" aliases
Done today at 04:46 EDT today at 04:46 EDT Run install hook of "gtk-common-themes" snap if present
Done today at 04:46 EDT today at 04:46 EDT Start snap "gtk-common-themes" (1198) services
Done today at 04:46 EDT today at 04:46 EDT Run configure hook of "gtk-common-themes" snap if present
Doing today at 04:46 EDT - Ensure prerequisites for "gnome-characters" are available
Do today at 04:46 EDT - Prepare snap "/var/lib/snapd/seed/snaps/gnome-characters_258.snap" (258)
Do today at 04:46 EDT - Mount snap "gnome-characters" (258)
Do today at 04:46 EDT - Copy snap "gnome-characters" data
Do today at 04:46 EDT - Setup snap "gnome-characters" (258) security profiles
Do today at 04:46 EDT - Make snap "gnome-characters" (258) available to the system
Do today at 04:46 EDT - Automatically connect eligible plugs and slots of snap "gnome-characters"
Do today at 04:46 EDT - Set automatic aliases for snap "gnome-characters"
Do today at 04:46 EDT - Setup snap "gnome-characters" aliases
Do today at 04:46 EDT - Run install hook of "gnome-characters" snap if present
Do today at 04:46 EDT - Start snap "gnome-characters" (258) services
Do today at 04:46 EDT - Run configure hook of "gnome-characters" snap if present
Do today at 04:46 EDT - Ensure prerequisites for "gnome-logs" are available
Do today at 04:46 EDT - Prepare snap "/var/lib/snapd/seed/snaps/gnome-logs_61.snap" (61)

[Rex Tsai (chihchun)]

@binli can not see it from log and can not reproduce the issue, can you elaborate why gnome-characters is hanged?

[Bin Li (binli)]

I tried to install the 3-28-1804, but it's failed, so this solution also need to debug.

17:23:48 + _snap_preseed / gnome-3-28-1804 stable/ubuntu-18.04
17:23:48 + local CHROOT_ROOT=/
17:23:48 + local SNAP=gnome-3-28-1804
17:23:48 + local SNAP_NAME=gnome-3-28-1804
17:23:48 + local CHANNEL=stable/ubuntu-18.04
17:23:48 + local seed_dir=//var/lib/snapd/seed
17:23:48 + local snaps_dir=//var/lib/snapd/seed/snaps
17:23:48 + local seed_yaml=//var/lib/snapd/seed/seed.yaml
17:23:48 + local assertions_dir=//var/lib/snapd/seed/assertions
17:23:48 + local snap_download_failed=0
17:23:48 + chroot / sh -c '
17:23:48 set -x;
17:23:48 cd /var/lib/snapd/seed;
17:23:48 SNAPPY_STORE_NO_CDN=1 snap download --channel=stable/ubuntu-18.04 "gnome-3-28-1804"'
17:23:48 + cd /var/lib/snapd/seed
17:23:48 + SNAPPY_STORE_NO_CDN=1 snap download --channel=stable/ubuntu-18.04 gnome-3-28-1804
17:23:48 Fetching snap "gnome-3-28-1804"
17:23:49 error: cannot download snap "gnome-3-28-1804": no snap revision available as specified
17:23:49 + snap_download_failed=1

[Bin Li (binli)]

@Rex,

 Just found below information, and when I removed the gnome-calculator, I found gnome-characters also hang at 'Doing' status, so it might be related to lower level app.

 Doing today at 04:46 EDT - Ensure prerequisites for "gnome-characters" are available

[Bin Li (binli)]

Looks in our oem image we used the wrong channel.

$ snap download --channel=stable/18.04 gnome-3-28-1804
Fetching snap "gnome-3-28-1804"
error: cannot download snap "gnome-3-28-1804": no snap revision available as specified

$ snap download --channel=stable gnome-3-28-1804
Fetching snap "gnome-3-28-1804"
gnome-3-28-1804

[Bin Li (binli)]

gnome-3-26-1604 could be downloaded from stable/18.04.

$ snap download --channel=stable/18.04 gnome-3-26-1604
Fetching snap "gnome-3-26-1604"
gnome-3-26-1604

[Rex Tsai (chihchun)]

Reproduced with the sutton-mccarthy-bionic-amd64-20190426-96.iso, here is summary of the finding -

- When the image was built last year, the gnome-* snaps depends on the content snap - gnome-3-26-1604.
- But the build process will pull the new gnome-* snaps, which (eg, gnome-calculator) depends on new content snap - gnome-3-28-1804. and keep using the old gnome-3-26-1604.
- During the Initialize stage of snapd, snapd could not solve prerequisites for gnome-* snaps and stuck in the wrong state, which blocking user install any snap.

The bugsy-config of the image need to refreshed and using gnome-3-28-1804.

The workaround for user will be manually fixing /var/lib/snapd/seed/seed.yaml and run "snap abort --last=seed", "sudo systemctl restart snapd"

[Vamshee Paduru (vpaduru1)]

Looks like this bug expands to more applications from app store..Example I tried to install VLC player and get the same error. So it could be more...will try some common applications and update the is bug asap..

[John Lenton (chipaca)]

Yes, an invalid seed.yaml will trip up snapd. Please don't seed images with an invalid seed.yaml.

[Rex Tsai (chihchun)]

@chipaca agree. The build processes need to be fixed and improve the QA tests to capture the issue.

However, there are invalid seed.yaml in the images out there in th wild. The device enablement team needs to work on a solution to fix the issues using online-updates.

[Bin Li (binli)]

@Rex,

 I tried to install the gnome-3-28-1804 from stable, and built a new image, after testing, I found there were still same issue. And it was hang at gnome-3-28-1804.

$snap version
snap 2.39
snapd 2.39
series 16
ubuntu 18.04
kernel 4.15.0-1038-oem

$snap info core
name: core
summary: snapd runtime environment
publisher: Canonical*
license: unset
description: |
  The core runtime environment for snapd
type: core
snap-id: 99T7MUlRhtI3U0QFgl5mXXESAiSwt776
tracking: stable
refresh-date: today at 05:03 EDT
channels:
  stable: 16-2.39 2019-05-22 (6964) 92MB -
  candidate: 16-2.39 2019-05-07 (6964) 92MB -
  beta: 16-2.39.1 2019-05-29 (7121) 92MB -
  edge: 16-2.39.1+git1319.1f92067 2019-05-31 (7134) 92MB -
installed: 16-2.39 (6964) 92MB core

$ snap change 1
Status Spawn Ready Summary
Done today at 05:03 EDT today at 05:03 EDT Ensure prerequisites for "core" are available
Done today at 05:03 EDT today at 05:03 EDT Prepare snap "/var/lib/snapd/seed/snaps/core_6964.snap" (6964)
Done today at 05:03 EDT today at 05:03 EDT Mount snap "core" (6964)
Done today at 05:03 EDT today at 05:03 EDT Copy snap "core" data
Done today at 05:03 EDT today at 05:03 EDT Setup snap "core" (6964) security profiles
Done today at 05:03 EDT today at 05:03 EDT Make snap "core" (6964) available to the system
Done today at 05:03 EDT today at 05:03 EDT Automatically connect eligible plugs and slots of snap "core"
Done today at 05:03 EDT today at 05:03 EDT Set automatic aliases for snap "core"
Done today at 05:03 EDT today at 05:03 EDT Setup snap "core" aliases
Done today at 05:03 EDT today at 05:03 EDT Run install hook of "core" snap if present
Done today at 05:03 EDT today at 05:03 EDT Start snap "core" (6964) services
Done today at 05:03 EDT today at 05:03 EDT Run configure hook of "core" snap if present
Doing today at 05:03 EDT - Ensure prerequisites for "gnome-3-28-1804" are available
Do today at 05:03 EDT - Prepare snap "/var/lib/snapd/seed/snaps/gnome-3-28-1804_47.snap" (47)
....

[Bin Li (binli)]

I tried to remove all the gnome-* snaps in /var/lib/snapd/seed/seed.yaml and run "snap abort --last=seed", "sudo systemctl restart snapd".

$ sudo snap install gnome-3-28-1804

It prompt download the core18(20190508 970).

So this hang is related to the wrong version of ubuntu core.

[Rex Tsai (chihchun)]

Yes, you also need to pull core18.

On Fri, May 31, 2019 at 5:41 PM Bin Li <email address hidden> wrote:

> I tried to remove all the gnome-* snaps in /var/lib/snapd/seed/seed.yaml
> and run "snap abort --last=seed", "sudo systemctl restart snapd".
>
> $ sudo snap install gnome-3-28-1804
>
> It prompt download the core18(20190508 970).
>
> So this hang is related to the wrong version of ubuntu core.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1828604
>
> Title:
> Lenovo T490S and X390 GM do not have Firefox.App store install errors
> out
>
> Status in OEM Priority Project:
> New
> Status in snapd:
> New
> Status in snapd package in Ubuntu:
> New
>
> Bug description:
> The new GM Images for T490S and X390 do not have Firefox preinstalled.
> When I try to install from the Appstore, it errors out "Firefox : too
> early for operation, device not yet seeded or device model not
> acknowledged."
>
> If I install using apt-get install firefox it seemed to install fine.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/oem-priority/+bug/1828604/+subscriptions
>

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Bin Li (binli)]

-------------------------------------------------------------------------------
You have successfully initiated a package propagation request!
-------------------------------------------------------------------------------
URL : https://bugs.launchpad.net/bugs/1838973
Assignee : oem-archive
Urgency : High
Title : request of copy_package [sutton-base-timbuktu-bionic-devel]
[Description]
 Operation: copy_package
 Source: sutton-base-timbuktu-bionic-devel
 Destination: sutton-base-timbuktu-bionic-staging
 Package: sutton-base-meta (==3.0sutton6)
 Production: bionic-sutton-base-timbuktu
 ===== Description =====

 Please verify the staging archive,
 """
 deb https://warthogs:<password>@cesg.canonical.com/canonical sutton-base-timbuktu-bionic-staging public
 deb-src https://warthogs:<password>@cesg.canonical.com/canonical sutton-base-timbuktu-bionic-staging public
 """
 You can find <password> from https://wiki.canonical.com/PES/Infrastructure/Repository/CustomerMirrors,

 And then verify the production archive.
 """
 deb http://cesg.archive.canonical.com/updates bionic-sutton-base-timbuktu public
 deb-src http://cesg.archive.canonical.com/updates bionic-sutton-base-timbuktu public
 """

[Jason Yen (jasonyen)]

@Vamshee,

This issue is fixed. Please try to upgrade your system and try again. Thanks!