snapd updates on focal never finish installing. Can't install any other updates.

[Daniel van Vugt]

snapd (2.33+18.10ubuntu3) cosmic never finishes installing. Can't install any other updates.

The first time I gave up waiting and killed it. Then...

$ sudo apt full-upgrade
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

$ sudo dpkg --configure -a
Setting up snapd (2.33+18.10ubuntu3) ...
snapd.snap-repair.service is a disabled or a static unit, not starting it.

<< never ends >>

All the while the snap and snapd process use about 0.3% CPU (which is more than usual).

WORKAROUND:

sudo killall apt dpkg
sudo dpkg -r snapd gnome-software-plugin-snap
sudo apt update
sudo apt full-upgrade

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[corrado venturini (corradoventu)]

problem does not occur on same hardware, different partition installed from
Ubuntu 18.10 "Cosmic Cuttlefish" - Alpha amd64 (20180509)
where snap list is:
corrado@corrado-p6-cc-0509:~$ snap list
Name Version Rev Tracking Developer Notes
core 16-2.32.8 4650 stable canonical core
gnome-3-26-1604 3.26.0 64 stable/… canonical -
gnome-calculator 3.28.1 178 stable/… canonical -
gnome-characters 3.28.2 101 stable/… canonical -
gnome-logs 3.28.2 37 stable/… canonical -
gnome-system-monitor 3.26.0 45 stable canonical -
gtk-common-themes 0.1 319 stable canonical -
corrado@corrado-p6-cc-0509:~$

corrado@corrado-p6-cc-0509:~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  snapd
1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/12,4 MB of archives.
After this operation, 705 kB of additional disk space will be used.
Do you want to continue? [Y/n]
(Reading database ... 177247 files and directories currently installed.)
Preparing to unpack .../snapd_2.33+18.10ubuntu3_amd64.deb ...
Unpacking snapd (2.33+18.10ubuntu3) over (2.32.6+18.10ubuntu2) ...
Setting up snapd (2.33+18.10ubuntu3) ...
Installing new version of config file /etc/apparmor.d/usr.lib.snapd.snap-confine.real ...
Installing new version of config file /etc/profile.d/apps-bin-path.sh ...
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.
Created symlink /etc/systemd/system/cloud-final.service.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.
snapd.snap-repair.service is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.8.3-2) ...
corrado@corrado-p6-cc-0509:~$

[Seth Arnold (seth-arnold)]

It might be worth grabbing a second or two of strace output from those processes to get a feeling for what they are trying to do.

Thanks

[Daniel van Vugt (vanvugt)]

Workaround:

sudo dpkg -r snapd gnome-software-plugin-snap
sudo apt update
sudo apt full-upgrade

[corrado venturini (corradoventu)]

Adding strace, I hope is useful

[corrado venturini (corradoventu)]

Problem does not happen on a new install from Ubuntu 18.10 "Cosmic Cuttlefish" - Alpha amd64 (20180620) on same PC
corrado@corrado-p13-cc-0620:~$ apt policy snapd
snapd:
  Installed: 2.33+18.10ubuntu3
  Candidate: 2.33+18.10ubuntu3
  Version table:
 *** 2.33+18.10ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu cosmic/main amd64 Packages
        100 /var/lib/dpkg/status
corrado@corrado-p13-cc-0620:~$

[Daniel van Vugt (vanvugt)]

A fresh install would probably work around the problem. Although the next update to snapd might then hit it again.

[Daniel van Vugt (vanvugt)]

Although a fresh install won't work on most systems because of bug 1775743.

[Daniel van Vugt (vanvugt)]

I tried installing snapd since a new version was released last week but still the same problem:

$ sudo apt install snapd
...
Setting up snapd (2.33.1+18.10ubuntu1) ...
snapd.snap-repair.service is a disabled or a static unit, not starting it.

<< hangs forever >>

[corrado venturini (corradoventu)]

after changing /var/lib/snapd/seed/seed.yaml as suggested in https://ubuntuforums.org/showthread.php?t=2393347&page=3 sudo dpkg --configure -a works fine.

[Doug McMahon (mc3man)]

This bug shouldn't be an issue on a fresh 18.10 install using latest image (07/23 or newer
(- the included gnome-calculator snap is completely unusable (will not start) but that's not this bug...

[Christopher Patrick (cpatrick08)]

sane problem upgrading to snapd 2.35~pre3+18.10.5 on 18.10

[Christopher Patrick (cpatrick08)]

Setting up snapd (2.35~pre3+18.10.5) ...
Installing new version of config file /etc/apparmor.d/usr.lib.snapd.snap-confine.real ...
snapd.failure.service is a disabled or a static unit, not starting it.
snapd.snap-repair.service is a disabled or a static unit, not starting it.

[Phillip Susi (psusi)]

Since ctrl-c doesn't seem to work to interrupt the upgrade, I had to ctrl-z and I checked pstree. It seems that the init script runs systemctl, which runs systemd-tty-ask-password-agent, and that hangs, never presenting a prompt. I tried to kill that and it became a zombie, never being reaped by systemctl.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[corrado venturini (corradoventu)]

again with new snapd from proposed

Setting up snapd (2.35.1+18.10) ...
snapd.failure.service is a disabled or a static unit, not starting it.
snapd.snap-repair.service is a disabled or a static unit, not starting it.

[Michael Vogt (mvo)]

Thanks for your bugreport. The output of "ps afx" when this hangs would be great to get an idea why the dpkg process blocks.

[corrado venturini (corradoventu)]

I added output from "ps afx" in my bug: https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1791087

[Alex (loxal)]

This happend to me repeatedly on Ubuntu Bionic too, making snaps uninstallable. How can such bugs come through. They can be easily catched with automated tests. I remember this bug happened to me after a fresh Ubuntu Bionic installation. An now again after updating snap. Please automate your tests!

alex@sun:~/my/project/intrafind/if-sitesearch/ops/al-provisioning$ ps afx
  PID TTY STAT TIME COMMAND
    2 ? S 0:00 [kthreadd]
    4 ? I< 0:00 \_ [kworker/0:0H]
    6 ? I< 0:00 \_ [mm_percpu_wq]
    7 ? S 0:01 \_ [ksoftirqd/0]
    8 ? I 2:28 \_ [rcu_sched]
    9 ? I 0:00 \_ [rcu_bh]
   10 ? S 0:00 \_ [migration/0]
   11 ? S 0:00 \_ [watchdog/0]
   12 ? S 0:00 \_ [cpuhp/0]
   13 ? S 0:00 \_ [cpuhp/1]
   14 ? S 0:00 \_ [watchdog/1]
   15 ? S 0:00 \_ [migration/1]
   16 ? S 0:02 \_ [ksoftirqd/1]
   18 ? I< 0:00 \_ [kworker/1:0H]
   19 ? S 0:00 \_ [cpuhp/2]
   20 ? S 0:00 \_ [watchdog/2]
   21 ? S 0:00 \_ [migration/2]
   22 ? S 0:01 \_ [ksoftirqd/2]
   24 ? I< 0:00 \_ [kworker/2:0H]
   25 ? S 0:00 \_ [cpuhp/3]
   26 ? S 0:00 \_ [watchdog/3]
   27 ? S 0:00 \_ [migration/3]
   28 ? S 0:02 \_ [ksoftirqd/3]
   30 ? I< 0:00 \_ [kworker/3:0H]
   31 ? S 0:00 \_ [cpuhp/4]
   32 ? S 0:00 \_ [watchdog/4]
   33 ? S 0:00 \_ [migration/4]
   34 ? S 0:01 \_ [ksoftirqd/4]
   36 ? I< 0:00 \_ [kworker/4:0H]
   37 ? S 0:00 \_ [cpuhp/5]
   38 ? S 0:00 \_ [watchdog/5]
   39 ? S 0:00 \_ [migration/5]
   40 ? S 0:01 \_ [ksoftirqd/5]
   42 ? I< 0:00 \_ [kworker/5:0H]
   43 ? S 0:00 \_ [cpuhp/6]
   44 ? S 0:00 \_ [watchdog/6]
   45 ? S 0:00 \_ [migration/6]
   46 ? S 0:03 \_ [ksoftirqd/6]
   48 ? I< 0:00 \_ [kworker/6:0H]
   49 ? S 0:00 \_ [cpuhp/7]
   50 ? S 0:00 \_ [watchdog/7]
   51 ? S 0:00 \_ [migration/7]
   52 ? S 0:01 \_ [ksoftirqd/7]
   54 ? I< 0:00 \_ [kworker/7:0H]
   55 ? S 0:00 \_ [kdevtmpfs]
   56 ? I< 0:00 \_ [netns]
   57 ? S 0:00 \_ [rcu_tasks_kthre]
   58 ? S 0:23 \_ [kauditd]
   61 ? S 0:00 \_ [khungtaskd]
   62 ? S 0:00 \_ [oom_reaper]
   63 ? I< 0:00 \_ [writeback]
   64 ? S 0:00 \_ [kcompactd0]
   65 ? SN 0:00 \_ [ksmd]
   66 ? SN 0:00 \_ [khugepaged]
   67 ? I< 0:00 \_ [crypto]
   68 ? I< 0:00 \_ [kintegrityd]
   69 ? I< 0:00 \_ [kblockd]
   76 ? I< 0:00 \_ [ata_sff]
   77 ? I< 0:00 \_ [md]
   78 ? I< 0:00 \_ [edac-poller]
   80 ? I< 0:00 \_ [devfreq_wq]
   81 ? I< 0:00 \_ [watchdogd]
   84 ? S 0:00 \_ [kswapd0]
   85 ? S 0:00 \_ [ecryptfs-kthrea]
...

[Michael Vogt (mvo)]

Hey Alex first, sorry for the trouble you have! We do run ~2800 tests which each branch merged or release. This includes an automatic upgrade test. Unfortunately the hang is not reproducible for everyone it seems, neither our automatic tests nor I in a VM have managed to trigger this hang. But we are very keen to learn what is missing to reproduce this issue. Is there anything special on the systems you experience this issue (unusual polkit configuration or special integration with pam or similar).

[Michael Vogt (mvo)]

What we see in the ps output is that for some reason it seems like systemctl prompts for a password. Which is unusual because apt/dpkg runs as root so there should be no need for this. Any further hints for us? Are these desktop or server installs?

[Daniel van Vugt (vanvugt)]

I think I've hit this with most/all cosmic desktop installs for the past few months.

My uncertainty only comes from the fact that you won't hit the bug until/unless the system needs to update snapd after that. So maybe a good test is to start with a cosmic image that is older than the last snapd update (2018-09-06). If you can't find such an image then install the current cosmic image (2018-09-08) and wait till the next snapd update in order to reproduce the bug.

Not sure if it's any different for automatic updates but I always manually update:

  sudo apt update
  sudo apt full-upgrade

which is why the cause of the bug was immediately obvious to me.

[Mario Vukelic (kreuzsakra)]

Michael, I am not Alex but I had the same on a completely fresh Cosmic install from the Desktop image to a new latop (Asus ZenBook Flip UX561UD). No special polkit/pam configs, after the install was finished I ran an update and there it was

[Mario Vukelic (kreuzsakra)]

Unfortunately the machine is a friend's and not in my hands anymore, but I can probably check next weekend if necessary

[corrado venturini (corradoventu)]

the ps output I addet to my bug https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1791087 is from a laptop install

[Michael Vogt (mvo)]

We found the root cause now and work on a fix. sorry for the trouble - Note that this is a "cosmic" specific issue. I'm exploring workarounds now to unblock you.

[Daniel van Vugt (vanvugt)]

Bug 1781513 is on bionic, so that's not the same issue as this?

[corrado venturini (corradoventu)]

workaround is easy: modify /var/lib/snapd/seed/seed.yaml and restart snapd
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1785388

[Mario Vukelic (kreuzsakra)]

Note that according to Oliver in the above linked bug #1785388 this is currently intentional in Cosmic

[Sergey Zolotarev (szx)]

Same issue here, I do "sudo pkill snapd" every time dpkg gets stuck on setting up snapd, at least this lets other packages to continue installing/updating.

[corrado venturini (corradoventu)]

On a new install from Ubuntu 18.10 "Cosmic Cuttlefish" - Alpha amd64 (20180913) the update+upgrade installed the new snapd (2.35.2+18.10) over (2.35.1+18.10)
and now i have all the default snap working

corrado@corrado-HP-p4-cc-0913:~$ snap list
Name Version Rev Tracking Publisher Notes
core 16-2.35 5328 stable canonical✓ core
gnome-3-26-1604 3.26.0 70 stable/… canonical✓ -
gnome-calculator 3.30.0 222 stable/… canonical✓ -
gnome-characters 3.28.2 117 stable/… canonical✓ -
gnome-logs 3.28.2 40 stable/… canonical✓ -
gnome-system-monitor 3.28.2 54 stable/… canonical✓ -
gtk-common-themes 0.1 319 stable/… canonical✓ -
corrado@corrado-HP-p4-cc-0913:~$

[Daniel van Vugt (vanvugt)]

Dropped severity.

While there are no updates to snapd newer than your ISO, this bug won't occur at all. Also there is a simple workaround documented at the top. And alternative workarounds are possible too.

[Daniel van Vugt (vanvugt)]

Was this fixed? I haven't encountered it in a while.

[Daniel van Vugt (vanvugt)]

Although that may just be because I am mainly using 19.04 and to a lesser extent 18.04.

[Launchpad Janitor (janitor)]

[Expired for snapd because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]

[Yves Dorfsman (dorfsmay)]

Experiencing the same issue running `apt upgrade` after a fresh Ubuntu 20.20 install.

Running https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1776622/comments/5 helped complete the install.

[Daniel van Vugt (vanvugt)]

Confirmed on 20.04 today.

[Daniel van Vugt (vanvugt)]

Further to comment #38, here's a workaround that works for me on 20.04:

sudo killall apt
sudo rm /var/lib/dpkg/lock
sudo dpkg -r snapd gnome-software-plugin-snap
sudo apt update
sudo apt full-upgrade

[Edward Hope-Morley (hopem)]

I can confirm i hit this with a fresh install of Focal Desktop today.

[Nathaniel Smith (njs)]

I also hit this today while running 'do-release-upgrade --dev' to upgrade a system from eoan to focal.

I was hesitant to kill the release upgrade and restart it, so I used a different workaround:

- opened a second terminal
- Run 'ps -A fwww', notice the following line:

   2629 ? Ssl 0:00 /usr/bin/snap wait system seed.loaded

- Kill that process

After doing this, the release upgrade continued.

[Balint Reczey (rbalint)]

This seems to be a snapd issue, not a systemd one.

[Claudio Matsuoka (cmatsuoka)]

Michael, could you confirm this was a cosmic-specific bug as there's a report of a similar problem with an upgrade from eoan to focal?

[Daniel van Vugt (vanvugt)]

This is a focal bug already.

[Michael Vogt (mvo)]

This problem is triggered by incorrect "/var/lib/snapd/seed/seed.yaml" files. At various points during the development cycles they were incorrect and that has (almost) no effect on the live-cd or the installed system. However on the next snapd refresh it would trigger the hang people are experiencing. The hang is caused by services waiting for "snapd" to finish seeding but that never happens because the seed.yaml is incorrect.

The best workaround is to

$ sudo mv /var/lib/snapd/seed/seed.yaml /var/lib/snapd/seed/seed.yaml .disabled

and re-run the upgrade. Versions of snapd from ~April 2020 will do this automatically for you if they detect a broken seed.

So I will close this bug as fixed because there is an automatic workaround in snapd now. The tooling around the livecd got also improved AIUI so that incorrect seed.yaml should be much harder to create now.

[Manoj Bhargav (manoj-bhargav333)]

Updating snap never finish-while I do not have/removed all snaps.