snapd won't start on dragonboard

[Pat McGowan]

It was working Friday, tried some snap commands this afternoon and snapd is not running. The system was up last three days without interaction, but it could have updated itself - I see an update Sunday

$ ls -lt /snap
total 20
drwxr-xr-x 5 root root 4096 Oct 9 17:33 ubuntu-core

This is on a Dragonboard.

Starting it fails as follows:

pat-mcgowan@localhost:~$ sudo systemctl status snapd
● snapd.service - Snappy daemon
   Loaded: loaded (/lib/systemd/system/snapd.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Tue 2016-10-11 19:34:19 UTC; 30s ago
  Process: 3434 ExecStart=/usr/lib/snapd/snapd (code=exited, status=1/FAILURE)
 Main PID: 3434 (code=exited, status=1/FAILURE)

Oct 11 19:34:18 localhost.localdomain systemd[1]: snapd.service: Unit entered failed state.
Oct 11 19:34:18 localhost.localdomain systemd[1]: snapd.service: Failed with result 'exit-code'.
Oct 11 19:34:19 localhost.localdomain systemd[1]: snapd.service: Service hold-off time over, scheduling restart.
Oct 11 19:34:19 localhost.localdomain systemd[1]: Stopped Snappy daemon.
Oct 11 19:34:19 localhost.localdomain systemd[1]: snapd.service: Start request repeated too quickly.
Oct 11 19:34:19 localhost.localdomain systemd[1]: Failed to start Snappy daemon.

From syslog:
Oct 11 19:39:38 localhost systemd[1]: snapd.service: Main process exited, code=exited, status=1/FAILURE
Oct 11 19:39:38 localhost systemd[1]: snapd.service: Unit entered failed state.
Oct 11 19:39:38 localhost systemd[1]: snapd.service: Failed with result 'exit-code'.
Oct 11 19:39:39 localhost systemd[1]: snapd.service: Service hold-off time over, scheduling restart.
Oct 11 19:39:39 localhost systemd[1]: Stopped Snappy daemon.
Oct 11 19:39:39 localhost systemd[1]: Started Snappy daemon.
Oct 11 19:39:39 localhost snapd[3538]: error: cannot downgrade: snapd is too old for the current system state (patch level 4)
Oct 11 19:39:39 localhost systemd[1]: snapd.service: Main process exited, code=exited, status=1/FAILURE
Oct 11 19:39:39 localhost systemd[1]: snapd.service: Unit entered failed state.
Oct 11 19:39:39 localhost systemd[1]: snapd.service: Failed with result 'exit-code'.
Oct 11 19:39:39 localhost systemd[1]: snapd.service: Service hold-off time over, scheduling restart.
Oct 11 19:39:39 localhost systemd[1]: Stopped Snappy daemon.
Oct 11 19:39:39 localhost systemd[1]: snapd.service: Start request repeated too quickly.
Oct 11 19:39:39 localhost systemd[1]: Failed to start Snappy daemon.
Oct 11 19:39:39 localhost systemd[1]: snapd.socket: Unit entered failed state.
Oct 11 19:41:37 localhost rsyslogd-2007: action 'action 11' suspended, next retry is Tue Oct 11 19:43:07 2016 [v8.16.0 try http://www.rsyslog.com/e/2007 ]

[Pat McGowan (pat-mcgowan)]

Why is current pointing to the old version

ls -lt /snap/ubuntu-core/
total 0
lrwxrwxrwx 1 root root 3 Oct 9 17:25 current -> 425
drwxr-xr-x 23 root root 308 Oct 9 04:38 852
drwxr-xr-x 23 root root 308 Oct 8 04:38 844
drwxr-xr-x 23 root root 332 Sep 2 04:37 425

[Pat McGowan (pat-mcgowan)]

I relinked current to 852, rebooted and same errors

[Pat McGowan (pat-mcgowan)]

Somehow 425 is a more recent updated than the other, was made later in the day on Oct 9 but the original folder is old.

$ ls -l /snap/ubuntu-core/
total 0
drwxr-xr-x 23 root root 332 Sep 2 04:37 425
drwxr-xr-x 23 root root 308 Oct 8 04:38 844
drwxr-xr-x 23 root root 308 Oct 9 04:38 852
lrwxrwxrwx 1 root root 3 Oct 11 20:27 current -> 425

$ ls -l /var/lib/snapd/snaps/
total 294960
-rw------- 1 pat-mcgowan pat-mcgowan 109998080 Sep 27 06:30 dragonboard-kernel_10.snap
-rw------- 1 pat-mcgowan pat-mcgowan 761856 Sep 27 06:30 dragonboard_23.snap
-rw------- 1 root root 2969600 Oct 7 20:11 snapweb_17.snap
-rw------- 1 root root 57589760 Oct 9 17:06 ubuntu-core_425.snap
-rw------- 1 root root 65359872 Oct 8 09:43 ubuntu-core_844.snap
-rw------- 1 root root 65359872 Oct 9 05:08 ubuntu-core_852.snap

[Pat McGowan (pat-mcgowan)]

this shows a series of restarts every 5 mins, and a lot of
Generate device key
Request device serial
2016-10-09T05:07:27Z ERROR internal error: cannot find device key pair

[Pat McGowan (pat-mcgowan)]

Similar symptoms reported at lp:1632306 starting with the same update

[Oliver Grawert (ogra)]

(note that ths is actually two bugs ... bug 1632306 for the log spam about device initialization, bug 1631791 for the switch to 425 which prevents snapd from working at all.