Live images with broken seed causes snapd high CPU usage and periodic short GUI freezes

[Zoltan The G]

ThinkPad 25, UEFI mode, Ubuntu 18.04 installed as a _live_ distro in FAT32 partition on SATA SSD in WWAN slot.

Grub options allow booting without persistence or with a 256MB persistence file. Only thing "persisted" is TERM in favorites.

Without persistence all is well. With persistence snapd uses up to 90% CPU - perhaps more - and cripples the system. Fan runs nearly full speed constantly.

I can eliminate the problem by removing snapd - and that persists. Can't kill snapd. PID is constantly changing.

0% fan and low CPU use at idle w/out snapd with persistence, or booted w/out persistence.

WORKAROUND:
sudo systemctl stop snapd

[Oliver Grawert (ogra)]

this could be a special case of bug 1729867

[Zygmunt Krynicki (zyga)]

First of all, kudos for using a nice machine :-)

Can you please provide some additional information about the system when the issue was observed:

- snap version
- snap changes

Can you explain how to re-create a similar environment (the persistence aspect is something I'm unfamiliar with). I have a T470 which is very similar to your 25-anniversary model and I could try yo reproduce it there.

[Zoltan The G (wdw)]

It is a nice machine :) This one was signed by David Hill, so it's a real treasure :)

I can try to dig up the info you requested, but first I have to state that I don't even know what snapd is :( I can say that "apt purge snapd ubuntu-core-launcher squashfs-tools" made it stop eating my CPU.

It's an easy thing to set up, but might require a 2nd drive. Primary drive in an NVMe SSD with Lenovo's Win 10 preload on it. Secondary drive is a SATA SSD in the WWAN slot. One FAT32 partition, one NTFS.

I put Ubuntu as a live "install" in the FAT32 partition for a couple of reasons - which may be off-base. 1st, it's intended as backup/fallback/recovery tool if the primary drive fails. Since an _installed_ Ubuntu wants to add its boot stuff to the EFI partition on the primary, that wouldn't be much of a fall-back if primary fails. The live install can boot directly from the T's boot menu w/out needing the primary drive. The other reason is that when it's in a FAT32 partition it can be manipulated from Windows and v/v without having to add any additional fs tools, and no additional partitions are required. I also use the grub in the FAT32 partition to boot things like memtest, Acronis, and some other stuff. It ends up looking like one of my "Swill-army" flash drives. Backup is also facilitated.

Back to how to set up: I just copy the contents of the Ubuntu ISO to the FAT32 partition. Then create a persistence file and edit grub.conf to allow the option of persistence or not. I've got an old writeup on how I make my flash drives. The process is the same, but _installing_ grub isn't required since it's a UEFI-only setup:

http://www.beezmo.com/geezblog/?p=428

I'm happy to dig for details if that will help - but will probably need some guidance.

Thanks!

[Zoltan The G (wdw)]

Hmm, is there an edit button here somewhere???

Adding: I haven't tried yet but my _guess_ is the same thing would happen if the above was run from a flash drive. Will try to test later.

Bill

[Zoltan The G (wdw)]

I've verified that this can be reproduced when booted from a flash drive. As above: extract the 18.04 ISO to a FAT32 drive. UEFI boot it. Create a persistence file and a persistence boot stanza. Boot that.

snapd and Xorg will use lots of CPU. Perhaps not as consistently or as high as when booted from SSD but basically the same behavior. Makes for lots of fan and laggy behavior.

Tested so far on the ThinkPad 25, ThinkPad X1 Yoga Gen 1, pre-production ThinkPad X280, and an IdeaPad Yoga 920 with the same results.

For the record the X280 needs "psmouse.proto=bare" added to the linux line in the boot stanza to have a functioning UltraNav device.

[Zoltan The G (wdw)]

I see that this but report is "incomplete". In case it's waiting on the snap version it's 2.32.5.

Since I still don't really know what it is or how to use it, I have to presume there have been no changes. (I find lots of docs on the interwebs but nothing so far that takes me back to the basics...)

I see the high CPU usage immediately after installing 18.04 as-is and adding persistence.

[Zoltan The G (wdw)]

*bug* report ;)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Chen-Han Hsiao (Stanley) (swem)]

After finishing the installation and reboot, this issue could be observed before connecting to internet. snapd restarts again and again and consuming high percentage of cpu.
(Daily Build image: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180508)%)

May 9 06:40:11 u-XPS-13-9360 systemd[1]: Starting Snappy daemon...
May 9 06:40:11 u-XPS-13-9360 systemd[1]: Started Snappy daemon.
May 9 06:40:17 u-XPS-13-9360 systemd[1]: snapd.service: Service hold-off time over, scheduling restart.
May 9 06:40:17 u-XPS-13-9360 systemd[1]: Stopped Snappy daemon.
May 9 06:40:17 u-XPS-13-9360 systemd[1]: Starting Snappy daemon...
May 9 06:40:18 u-XPS-13-9360 systemd[1]: Started Snappy daemon.
May 9 06:40:22 u-XPS-13-9360 systemd[1]: snapd.service: Service hold-off time over, scheduling restart.
May 9 06:40:22 u-XPS-13-9360 systemd[1]: Stopped Snappy daemon.
May 9 06:40:22 u-XPS-13-9360 systemd[1]: Starting Snappy daemon...
May 9 06:40:22 u-XPS-13-9360 systemd[1]: Started Snappy daemon.
May 9 06:40:29 u-XPS-13-9360 systemd[1]: snapd.service: Service hold-off time over, scheduling restart.
May 9 06:40:29 u-XPS-13-9360 systemd[1]: Stopped Snappy daemon.
May 9 06:40:29 u-XPS-13-9360 systemd[1]: Starting Snappy daemon...
May 9 06:40:30 u-XPS-13-9360 systemd[1]: Started Snappy daemon.

$ snap version
snap 2.32.6
snapd 2.32.6
series 16
ubuntu 18.04
kernel 4.15.0-20-generic

$ snap changes
ID Status Spawn Ready Summary
1 Error 2018-05-09T10:40:16Z 2018-05-09T10:40:23Z Initialize system state
2 Done 2018-05-09T10:40:16Z 2018-05-09T10:45:20Z Initialize device
3 Error 2018-05-09T10:40:27Z 2018-05-09T10:40:35Z Initialize system state
4 Error 2018-05-09T10:40:38Z 2018-05-09T10:40:45Z Initialize system state
5 Error 2018-05-09T10:40:48Z 2018-05-09T10:40:56Z Initialize system state
6 Error 2018-05-09T10:40:59Z 2018-05-09T10:41:07Z Initialize system state
7 Error 2018-05-09T10:41:10Z 2018-05-09T10:41:18Z Initialize system state
8 Error 2018-05-09T10:41:21Z 2018-05-09T10:41:30Z Initialize system state
9 Error 2018-05-09T10:41:33Z 2018-05-09T10:41:42Z Initialize system state
10 Error 2018-05-09T10:41:45Z 2018-05-09T10:41:54Z Initialize system state
11 Error 2018-05-09T10:41:57Z 2018-05-09T10:42:07Z Initialize system state
12 Error 2018-05-09T10:42:10Z 2018-05-09T10:42:19Z Initialize system state
13 Error 2018-05-09T10:42:22Z 2018-05-09T10:42:32Z Initialize system state
14 Error 2018-05-09T10:42:35Z 2018-05-09T10:42:45Z Initialize system state
15 Error 2018-05-09T10:42:48Z 2018-05-09T10:42:58Z Initialize system state
16 Error 2018-05-09T10:43:02Z 2018-05-09T10:43:13Z Initialize system state
17 Error 2018-05-09T10:43:16Z 2018-05-09T10:43:27Z Initialize system state
18 Error 2018-05-09T10:43:30Z 2018-05-09T10:43:42Z Initialize system state
19 Error 2018-05-09T10:43:46Z 2018-05-09T10:43:58Z Initialize system state
20 Error 2018-05-09T10:44:01Z 2018-05-09T10:44:13Z Initialize system state
21 Error 2018-05-09T10:44:16Z 2018-05-09T10:44:28Z Initialize system state
22 Error 2018-05-09T1...

[Chen-Han Hsiao (Stanley) (swem)]

seeded snap in Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180508)%

$ ll /var/lib/snapd/seed/snaps/
total 273560
drwxr-xr-x 2 root root 4096 May 8 04:23 ./
drwxr-xr-x 4 root root 4096 May 8 04:23 ../
-rw-r--r-- 2 root root 90828800 May 8 04:22 core_4571.snap
-rw-r--r-- 2 root root 146841600 May 8 04:22 gnome-3-26-1604_62.snap
-rw-r--r-- 2 root root 2428928 May 8 04:22 gnome-calculator_167.snap
-rw-r--r-- 2 root root 13594624 May 8 04:23 gnome-characters_86.snap
-rw-r--r-- 2 root root 22609920 May 8 04:23 gnome-logs_31.snap
-rw-r--r-- 2 root root 3813376 May 8 04:23 gnome-system-monitor_39.snap

[John Lenton (chipaca)]

That's very interesting!
Could you paste the output of "snap tasks --last=seed" ?

Thank you.

[Chen-Han Hsiao (Stanley) (swem)]

output of "snap tasks --last=seed"

This issue could be reproduced on every platform.

[Chen-Han Hsiao (Stanley) (swem)]

Note that this issue doesn't happen with official released image ubuntu-18.04-desktop-amd64.iso.

But does happen with daily build image http://cdimage.ubuntu.com/bionic/daily-live/pending/bionic-desktop-amd64.iso (For example, 20180508)

I think this issue is introduced between 2018-04-24 and 2018-04-30

[John Lenton (chipaca)]

it looks like the seed is missing gtk-common-themes

[John Lenton (chipaca)]

If you seed a snap, then you also need to seed that snap's base, and snaps providing any plugs that snap requires, otherwise it'll fail to seed.

A failure to seed it pretty bad, as snapd will be stuck in a loop trying to do that. I don't think this is a bug in snapd, because if you have network, this same retrying will mean that the missing bases and missing content interfaces that have default providers specified will be downloaded automatically, and seeding will succeed. There might be some things we can do to make it use less CPU, but they're a lot of work. The only *sensible* thing snapd could in this case is alert the user, although there's nothing the user can do beyond contact the creator of the seed so it's not very useful.

Marking as "Won't fix" for snapd. Feel free to point out why I'm wrong on that.

[Chen-Han Hsiao (Stanley) (swem)]

gtk-common-themes is not published in channel "stable/ubuntu-18.04", thus the image building will fail.

Hi, chipaca
Could you help to publish gtk-common-themes in channel "stable/ubuntu-18.04".

Ref:

$ SNAPPY_STORE_NO_CDN=1 snap download --channel=stable/ubuntu-18.04 gtk-common-themes
Fetching snap "gtk-common-themes"
error: cannot find snap "gtk-common-themes": snap not found

[Chen-Han Hsiao (Stanley) (swem)]

Code for fetching seeded snap while image building:

https://git.launchpad.net/ubuntu/+source/livecd-rootfs/tree/live-build/functions?id=af65a5affaecdf538c2d7e90775e9e5c4de8af09#n387

[John Lenton (chipaca)]

I am not a publisher for that snap, so I can't do that. I'll ask on #snapd to see if anybody is.

[Chen-Han Hsiao (Stanley) (swem)]

Also proposed code changes to ubuntu seeds:

https://code.launchpad.net/~swem/ubuntu-seeds/ubuntu.cosmic/+merge/345355
https://code.launchpad.net/~swem/ubuntu-seeds/ubuntu.bionic/+merge/345356

[Ken VanDine (ken-vandine)]

I've created channels for stable/ubuntu-18.04 and stable/ubuntu-18.10

[Dimitri John Ledkov (xnox)]

Both seed proposals merged. I believe nothing else needs doing, and tomorrow's dailies should be fixed then.

[Chen-Han Hsiao (Stanley) (swem)]

This issue fixed in daily build bionic-desktop-amd64.iso (20180522)

[Daniel van Vugt (vanvugt)]

Confirmed fixed in http://cdimages.ubuntu.com/daily-live/20180522/

[Zoltan The G (wdw)]

The above "confirmed fixed" link is broken.

If you'll pardon another ignorant question: when does this fix show up in distro?

Just tried 18.10 and persistence renders live Ubuntu completely unusable.

[Daniel van Vugt (vanvugt)]

If the problem still happens in the final release of 18.10:
http://releases.ubuntu.com/18.10/ubuntu-18.10-desktop-amd64.iso

then yes this bug needs to be reopened. Or another one logged. But since you are the original reporter, I would say reopen.

[Zoltan The G (wdw)]

Yes, this is with the 18.10 final release.

I'm unable to provide confirmation/diagnostics as the behavior is worse than with 18.04/04.1. With persistence enabled I can sometimes initially get to a desktop which has a moving mouse pointer but is unresponsive. It will at some point black-screen with a message about the Snappy daemon - which I wasn't able to capture.

Subsequent attempts to boot will either hang on the Ubuntu splash indefinitely, or eventually flash a desktop immediately followed by a black screen with only a flashing cursor.

I can exit that and initiate a shutdown with the three-fingered-salute. That brings a message about a stop job for the Snappy daemon, which also hangs for a very long time.

Sorry I can't provide any further information. I'm handcuffed ... and frustrated as *bleep*.

[Zygmunt Krynicki (zyga)]

Hello, there is a difference between the live installer session where snaps are supported to a limited degree and the persistent image where they are not supported at all. I'm very sorry for the inconvenience. I don't know if it is feasible to disable snapd in that context (through the persistence layer).

[Zoltan The G (wdw)]

So persistence is no longer possible? That's very disappointing :( Perhaps the option should be removed.

[John Lenton (chipaca)]

Moving it back to Won't fix, for snapd at least. I don't see an issue here that snapd itself _can_ fix.