Latest snapd in Trusty is broken after reboot because of systemd units start ordering

[Rafael David Tinoco]

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty

Linux systemdlivetnew 4.4.0-96-generic #119~14.04.1-Ubuntu SMP Wed Sep 13 08:40:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

snap-confine 2.27.5~14.04
snapd 2.27.5~14.04

--------------

root@systemdlivetnew:~$ snap interfaces
error: no interfaces found

root@systemdlivetnew:~$ snap install core
core 16-2.27.6 from 'canonical' installed

root@systemdlivetnew:~$ snap interfaces
Slot Plug
:account-control -
:alsa -
:autopilot-introspection -
:avahi-observe -
:bluetooth-control -
:broadcom-asic-control -
:browser-support -
...

---- reboot ----

root@systemdlivetnew:~$ snap list
Name Version Rev Developer Notes
core 16-2.27.6 2898 canonical core

root@systemdlivetnew:~$ snap interfaces
Slot Plug
:account-control -
:alsa -
:autopilot-introspection -
:avahi-observe -
:bluetooth-control -
:broadcom-asic-control -
:browser-support -
...

root@systemdlivetnew:~$ snap install hello-world
hello-world 6.3 from 'canonical' installed

root@systemdlivetnew:~$ snap interfaces
Slot Plug
:account-control -
:alsa -
:autopilot-introspection -
:avahi-observe -
:bluetooth-control -
:broadcom-asic-control -
:browser-support -

---- reboot ----

root@systemdlivetnew:~$ snap list
Name Version Rev Developer Notes
core 16-2.27.6 2898 canonical core
hello-world 6.3 27 canonical -

root@systemdlivetnew:~$ snap interfaces
error: no interfaces found

root@systemdlivetnew:~$ systemctl restart snapd.service

root@systemdlivetnew:~$ snap interfaces
Slot Plug
:account-control -
:alsa -
:autopilot-introspection -
:avahi-observe -
:bluetooth-control -
:broadcom-asic-control -
:browser-support -

This also happens with other snaps like canonical-livepatch.

[Rafael David Tinoco (rafaeldtinoco)]

Looks to me that the interface inter-dependencies should be also reflected in the start order in systemd unit files that are created per every installed snap:

After installing hello-world, if I add the needed relationships:

[Unit]
Description=Mount unit for hello-world
After=snapd.service snap-core-2898.mount
Requires=snapd.service snap-core-2898.mount

[Mount]
What=/var/lib/snapd/snaps/hello-world_27.snap
Where=/snap/hello-world/27
Type=squashfs
Options=nodev,ro

[Install]
WantedBy=multi-user.target

(note After/Requires being added to the default one)

I get no problems on every reboot:

root@systemdlivetnew:system$ snap list
Name Version Rev Developer Notes
core 16-2.27.6 2898 canonical core
hello-world 6.3 27 canonical -

root@systemdlivetnew:system$ snap interfaces
Slot Plug
:account-control -
:alsa -
:autopilot-introspection -
:avahi-observe -
:bluetooth-control -
:broadcom-asic-control -
:browser-support -

I'll test adding a second snap now, and then adding the relationship.

[Rafael David Tinoco (rafaeldtinoco)]

root@systemdlivetnew:~$ snap install canonical-livepatch
canonical-livepatch 7 from 'canonical' installed

root@systemdlivetnew:~$ canonical-livepatch enable xxxxxxxx
Successfully enabled device. Using machine-token: xxxxxxxx

root@systemdlivetnew:~$ snap list
Name Version Rev Developer Notes
canonical-livepatch 7 25 canonical -
core 16-2.27.6 2898 canonical core
hello-world 6.3 27 canonical -

root@systemdlivetnew:~$ snap interfaces
Slot Plug
:account-control -
:alsa -
:autopilot-introspection -
:avahi-observe -
:bluetooth-control -
:broadcom-asic-control -
:browser-support -
:camera -
:classic-support -
:core-support core:core-support-plug
:cups-control -
:dcdbas-control -
:docker-support -
:firewall-control -
:framebuffer -
:greengrass-support -
:gsettings -
:hardware-observe canonical-livepatch
:hardware-random-control -
:hardware-random-observe -
:home -
:io-ports-control -
:joystick -
:kernel-module-control canonical-livepatch
:kubernetes-support -
:libvirt -
:locale-control -
:log-observe -
:lxd-support -
:modem-manager -
:mount-observe -
:netlink-audit -
:netlink-connector -
:network -
:network-bind canonical-livepatch
:network-control canonical-livepatch
:network-manager canonical-livepatch
:network-observe -
:network-setup-control -
:network-setup-observe -
:ofono -
:opengl -
:openvswitch -
:openvswitch-support -
:optical-drive -
:password-manager-service -
:physical-memory-control -
:physical-memory-observe -
:ppp -
:process-control -
:pulseaudio -
:raw-usb -
:removable-media -
:screen-inhibit-control -
:shutdown -
:snapd-control -
:system-observe canonical-livepatch
:system-trace -
:time-control -
:timeserver-control -
:timezone-control -
:tpm -
:uhid -
:unity7 -
:upower-observe -
:x11 -

-------- REBOOT --------

root@systemdlivetnew:~$ snap list
Name Version Rev Developer Notes
canonical-livepatch 7 25 canonical -
core 16-2.27.6 2898 canonical core
hello-world 6.3 27 canonical -

root@systemdlivetnew:~$ snap interfaces
Slot Plug
- canonical-livepatch:hardware-observe
- canonical-livepatch:kernel-module-control
- canonical-livepatch:network-bind
- canonical-livepatch:network-control
- canonical-livepatch:network-manager
- canonical-livepatch:system-observe

Systemd race condition is st...

[Rafael David Tinoco (rafaeldtinoco)]

After installing JUST canonical-livepatch snap and rebooting:

root@systemdlivetnew:~$ snap interfaces core | grep -v Slot | wc -l
0
root@systemdlivetnew:~$ snap interfaces canonical-livepatch | grep -v Slot | wc -l
6

The interfaces from core are gone. Setting dependency for Systemd:

root@systemdlivetnew:~$ snap interfaces core | grep -v Slot | wc -l
66
root@systemdlivetnew:~$ snap interfaces canonical-livepatch | grep -v Slot | wc -l
0

Interfaces from core are there but from canonical-livepatch are gone. The only way I could get it enabled again is by doing:

root@systemdlivetnew:~$ snap disable canonical-livepatch
canonical-livepatch disabled

root@systemdlivetnew:~$ snap enable canonical-livepatch
canonical-livepatch enabled

root@systemdlivetnew:~$ snap interfaces canonical-livepatch | grep -v Slot | wc -l
6
root@systemdlivetnew:~$ snap interfaces core | grep -v Slot | wc -l
66

root@systemdlivetnew:~$ systemctl status "snap-canonical\x2dlivepatch-25.mount" | grep Activ
   Active: active (mounted) since Thu 2017-10-05 13:05:41 UTC; 3min 34s ago
root@systemdlivetnew:~$ systemctl status snap.canonical-livepatch.canonical-livepatchd.service | grep Activ
   Active: active (running) since Thu 2017-10-05 13:08:20 UTC; 1min 0s ago

And then snap.canonical-livepatch.canonical-livepatchd.service could start.

-------- REBOOT --------

root@systemdlivetnew:~$ snap interfaces core | grep -v Slot | wc -l
66
root@systemdlivetnew:~$ snap interfaces canonical-livepatch | grep -v Slot | wc -l
0
root@systemdlivetnew:~$ snap disable canonical-livepatch
canonical-livepatch disabled
root@systemdlivetnew:~$ snap enable canonical-livepatch
canonical-livepatch enabled
root@systemdlivetnew:~$
root@systemdlivetnew:~$ snap interfaces core | grep -v Slot | wc -l
66
root@systemdlivetnew:~$ snap interfaces canonical-livepatch | grep -v Slot | wc -l
6
root@systemdlivetnew:~$ canonical-livepatch status
kernel: 4.4.0-96.119~14.04.1-generic
fully-patched: true
version: ""

It looks like, despite that having the right boot order, in this snap, for livepatch mount unit file, I have also to disable/enable the snap in order to get the slots re-created. Differently from hello-world which just guaranteeing the systemd order was enough for it to work (but it had no slots connected also).

[Michael Vogt (mvo)]

I just tried to reproduce this with 2.28.5~14.04 (which is in trusty-proposed) and did not manage to trigger this error yet. I will try with 2.27.6 next.

[Michael Vogt (mvo)]

I retried with 2.27.6 with the instructions and could not reproduce the failure. I am using a 14.04 autopkgtest image. What image is your systemd based on? Maybe that is the key?

[Rafael David Tinoco (rafaeldtinoco)]

Michael,

All this was done with:

$ dpkg -l | grep snapd
ii snapd 2.28.4~14.04 amd64 Daemon and tooling that enable snap packages

Inside a KVM guest. My systemd was, at that time, the -proposed fix for bug:

https://bugs.launchpad.net/snapd/+bug/1718966

Which was the latest one. Not sure what you meant by "image based on" (its a fresh installation, not a cloud image if that is what you meant). I reproduced it based on a feedback from another user, and faced the same issues as he did.

I just re-tried reproducing it (purged snapd, upgraded os, re-installed snapd):

$ dpkg -l | grep snap
ii snapd 2.28.5~14.04 amd64 Daemon and tooling that enable snap packages

With the same systemd version I had (didn't get updated)

ii systemd 204-5ubuntu20.25 amd64 system and service manager

And I wasn't able to. I went for my cache and re-installed 2.28.4~14.04 and saw that the problem is there:

$ snap interfaces
Slot Plug
- canonical-livepatch:hardware-observe
- canonical-livepatch:kernel-module-control
- canonical-livepatch:network-bind
- canonical-livepatch:network-control
- canonical-livepatch:network-manager
- canonical-livepatch:system-observe

Re-installed 2.28.5~14.04 and problem is gone again.

So by the time I reported this (2017-10-05) this fix didn't exist. Based on changelog:

snapd (2.28.5~14.04) trusty; urgency=medium

  * New upstream release, LP: #1714984
    - snap-confine: cleanup broken nvidia udev tags
    - cmd/snap-confine: update valid security tag regexp
    - overlord/ifacestate: refresh udev backend on startup
    - dbus: ensure io.snapcraft.Launcher.service is created on re-
      exec
    - snap-confine: add support for handling /dev/nvidia-modeset
    - interfaces/network-control: remove incorrect rules for tun

 -- Michael Vogt <email address hidden> Fri, 13 Oct 2017 23:25:46 +0200

It looks like this upstream version fixes this issue somehow (not documented in the changelog only) and not very clear (at least for me) in the diff http://launchpadlibrarian.net/340792949/snapd_2.28.4~14.04_2.28.5~14.04.diff.gz. It could be related to udev enumeration issue and the changes related to this in this new version (sc_maybe_fixup_udev).

I'll ask final user to verify this new version and feedback us. Tks.

[Michael Vogt (mvo)]

I now also tried this on a trusty desktop system but still couldn't reproduce the issue. All testing happend inside qemu VMs. Any hints how to reproduce this are appreciated.

[Rafael David Tinoco (rafaeldtinoco)]

Michael,

I'm afraid this is a timing thing happening because of services initialization depending on HW configuration and/or number of enabled services/devices (that is why I suspected on the udev enumeration issue, since the systemd-udev would probe HW and start udev triggers/etc).

I'm sharing my VM xml:

http://pastebin.ubuntu.com/25822005/

And my systemd units:

http://pastebin.ubuntu.com/25822014/

http://pastebin.ubuntu.com/25822018/

Please note that this was a follow on to case:

https://bugs.launchpad.net/snapd/+bug/1718966

So I have my /etc/fstab like:

## /etc/fstab

LABEL=ROOT / ext4 errors=remount-ro 0 1
LABEL=VAR /var ext4 defaults 0 1

## end of file

[Zygmunt Krynicki (zyga)]

I just managed to reproduce this on a vanilla Ubuntu 14.04 desktop VM. I was rebooting in a loop and running "snap interfaces" to see what's going on. I had disabled re-exec so that I would be testing 2.27.5 repeatedly.

I collected the following logs:

syslog: http://paste.ubuntu.com/25822015/
journalctl http://paste.ubuntu.com/25822017/
journalctl status snapd.service http://pastebin.ubuntu.com/25822002/
snap list: http://paste.ubuntu.com/25822020/

Unfortunately due to broken logging on 14.04 I only see parts of the snapd output.

Interestingly systemd on 14.04 seems very much broken:

  $ systemctl
  Failed to issue method call: No such method 'ListUnits'

I'll focus on getting some level of logging and on understanding systemd build configuration.

[Zygmunt Krynicki (zyga)]

With a few more interations and a small hack that makes logging possible I got a state where no plugs or slots were present at all (not even core). This strongly suggests that when snapd starts the various .mount units are not ready yet and none of the snap.yaml files can be read.

[Zygmunt Krynicki (zyga)]

After a few more cycles where I managed to reproduce this this is the full error output with SNAPD_DEBUG=1; not very much:

zyga@ubuntu-trusty:~$ cat /var/log/snapd.*
2017/10/26 11:11:33.609059 cmd.go:178: DEBUG: re-exec disabled by user
2017/10/26 11:11:33.658429 helpers.go:99: snap epoch cannot be empty
2017/10/26 11:11:33.659033 helpers.go:206: cannot connect plug "network-control" from snap "canonical-livepatch", no such plug
2017/10/26 11:11:33.659067 helpers.go:206: cannot connect plug "network-manager" from snap "canonical-livepatch", no such plug
2017/10/26 11:11:33.659083 helpers.go:206: cannot connect plug "system-observe" from snap "canonical-livepatch", no such plug
2017/10/26 11:11:33.659105 helpers.go:206: cannot connect plug "hardware-observe" from snap "canonical-livepatch", no such plug
2017/10/26 11:11:33.659121 helpers.go:206: cannot connect plug "kernel-module-control" from snap "canonical-livepatch", no such plug
2017/10/26 11:11:33.659133 helpers.go:206: cannot connect plug "network-bind" from snap "canonical-livepatch", no such plug
2017/10/26 11:11:35.675160 daemon.go:252: started snapd/2.27.5~14.04 (series 16; classic) ubuntu/14.04 (amd64) linux/4.4.0-97-generic.
2017/10/26 11:11:35.677056 main.go:72: DEBUG: activation done in 2.068s
2017/10/26 11:11:35.680561 snapmgr.go:516: DEBUG: Next refresh scheduled for 2017-10-26 18:24:42.258937299 +0200 CEST.
2017/10/26 11:11:54.420019 daemon.go:179: DEBUG: uid=1000;@ GET /v2/interfaces 597.999µs 200

[Zygmunt Krynicki (zyga)]

I decided to see if adding Before=snapd.service to the generated mount units will do any good.

On 3rd boot I got this:

zyga@ubuntu-trusty:~$ cat /var/log/snapd.*
2017/10/26 11:16:39.943403 cmd.go:178: DEBUG: re-exec disabled by user
2017/10/26 11:16:39.998231 helpers.go:99: snap epoch cannot be empty
2017/10/26 11:16:39.998543 helpers.go:206: cannot connect plug "core-support-plug" from snap "core", no such plug
2017/10/26 11:16:39.998564 helpers.go:206: cannot connect plug to slot "hardware-observe" from snap "core", no such slot
2017/10/26 11:16:39.998576 helpers.go:206: cannot connect plug to slot "kernel-module-control" from snap "core", no such slot
2017/10/26 11:16:39.998585 helpers.go:206: cannot connect plug to slot "network-bind" from snap "core", no such slot
2017/10/26 11:16:39.998595 helpers.go:206: cannot connect plug to slot "network-control" from snap "core", no such slot
2017/10/26 11:16:39.998605 helpers.go:206: cannot connect plug to slot "network-manager" from snap "core", no such slot
2017/10/26 11:16:39.998615 helpers.go:206: cannot connect plug to slot "system-observe" from snap "core", no such slot
2017/10/26 11:16:41.976905 daemon.go:252: started snapd/2.27.5~14.04 (series 16; classic) ubuntu/14.04 (amd64) linux/4.4.0-97-generic.
2017/10/26 11:16:41.977053 main.go:72: DEBUG: activation done in 2.034s
2017/10/26 11:16:41.979381 snapmgr.go:516: DEBUG: Next refresh scheduled for 2017-10-26 18:11:28.327523199 +0200 CEST.
zyga@ubuntu-trusty:~$ snap interfaces
2017/10/26 11:17:04.045127 cmd.go:178: DEBUG: re-exec disabled by user
Slot Plug
- canonical-livepatch:hardware-observe
- canonical-livepatch:kernel-module-control
- canonical-livepatch:network-bind
- canonical-livepatch:network-control
- canonical-livepatch:network-manager
- canonical-livepatch:system-observe

[Zygmunt Krynicki (zyga)]

During that same boot this is the status of the core mount unit:

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
snap-core-3247.mount -> '/org/freedesktop/systemd1/unit/snap_2dcore_2d3247_2emount'

snap-core-3247.mount - Mount unit for core
   Loaded: loaded (/etc/systemd/system/snap-core-3247.mount; enabled)
   Active: active (mounted) since czw 2017-10-26 11:16:39 CEST; 1min 51s ago
    Where: /snap/core/3247
     What: /dev/loop2
  Process: 664 ExecMount=/bin/mount /var/lib/snapd/snaps/core_3247.snap /snap/core/3247 -t squashfs -o nodev,ro (code=exited, status=0/SUCCESS)

[Rafael David Tinoco (rafaeldtinoco)]

Yes Zygmunt, now you're in the same path I was when I reported this. restarting units after systemd is settled doesn't necessarily work all the times. If you restart snapd AND then the snaps related units, it works.

[Zygmunt Krynicki (zyga)]

So it seems that core snap was already mounted when the snapd unit started. This seems to imply that Before=snapd.service worked correctly and something else is causing this.

11:24 < pedronis> zyga-ubuntu: does systemd own logs tells something about the order it did things?
11:25 < zyga-ubuntu> pedronis: no, things are very crippled
11:25 < zyga-ubuntu> pedronis: but let me look at one thing, it may indicate time
11:25 < zyga-ubuntu> so...
11:25 < zyga-ubuntu> core mounted on
11:25 < zyga-ubuntu> Active: active (mounted) since czw 2017-10-26 11:16:39 CEST; 1min 51s ago
11:27 < zyga-ubuntu> snapd started on
11:27 < zyga-ubuntu> Active: active (running) since czw 2017-10-26 11:16:41 CEST; 10min ago
11:27 < zyga-ubuntu> so core *was* mounted at the time snapd was started

[Zygmunt Krynicki (zyga)]

A version of snapd 2.27.5 with extra logging just turned out with a smoking gun:

I added a simple log statements to where we load interfaces from known snaps and found this:

2017/10/26 11:56:09.757571 cmd.go:178: DEBUG: re-exec disabled by user
2017/10/26 11:56:09.812427 helpers.go:105: snap epoch cannot be empty
2017/10/26 11:56:09.812813 helpers.go:105: snap epoch cannot be empty
2017/10/26 11:56:09.812924 helpers.go:105: snap epoch cannot be empty
2017/10/26 11:56:09.813078 helpers.go:213: cannot connect plug "kernel-module-control" from snap "canonical-livepatch", no such plug
2017/10/26 11:56:09.813099 helpers.go:213: cannot connect plug "network-bind" from snap "canonical-livepatch", no such plug
2017/10/26 11:56:09.813111 helpers.go:213: cannot connect plug "network-control" from snap "canonical-livepatch", no such plug
2017/10/26 11:56:09.813130 helpers.go:213: cannot connect plug "network-manager" from snap "canonical-livepatch", no such plug
2017/10/26 11:56:09.813142 helpers.go:213: cannot connect plug "system-observe" from snap "canonical-livepatch", no such plug
2017/10/26 11:56:09.813153 helpers.go:213: cannot connect plug "core-support-plug" from snap "core", no such plug
2017/10/26 11:56:09.813165 helpers.go:213: cannot connect plug "hardware-observe" from snap "canonical-livepatch", no such plug
2017/10/26 11:56:09.824138 backend.go:145: cannot create host snap-confine apparmor configuration: open /snap/core/3247/etc/apparmor.d/usr.lib.snapd.snap-confine: no such file or directory
2017/10/26 11:56:11.036680 daemon.go:252: started snapd/unknown (series 16; classic) ubuntu/14.04 (amd64) linux/4.4.0-97-generic.
2017/10/26 11:56:11.036835 main.go:72: DEBUG: activation done in 1.279s
2017/10/26 11:56:11.046239 snapmgr.go:516: DEBUG: Next refresh scheduled for 2017-10-26 21:44:00.746753961 +0200 CEST.
going to add all snaps into the interface repository
snapstate.ActiveInfos() returned: []*snap.Info{(*snap.Info)(0xc82025e000), (*snap.Info)(0xc82025e280), (*snap.Info)(0xc82025e500)}
adding snap to interface repository (canonical-livepatch)
&snap.Info{SuggestedName:"canonical-livepatch", Version:"", Type:"", Architectures:[]string(nil), Assumes:[]string(nil), OriginalTitle:"", OriginalSummary:"", OriginalDescription:"", Environment:strutil.OrderedMap{keys:[]string(nil), vals:map[string]string(nil)}, LicenseAgreement:"", LicenseVersion:"", Epoch:"", Confinement:"", Apps:map[string]*snap.AppInfo{"canonical-livepatch":(*snap.AppInfo)(0xc8202441e0), "canonical-livepatchd":(*snap.AppInfo)(0xc8202442d0)}, LegacyAliases:map[string]*snap.AppInfo(nil), Hooks:map[string]*snap.HookInfo(nil), Plugs:map[string]*snap.PlugInfo(nil), Slots:map[string]*snap.SlotInfo(nil), SideInfo:snap.SideInfo{RealName:"canonical-livepatch", SnapID:"b96UJ4vttpNhpbaCWctVzfduQcPwQ5wn", Revision:snap.Revision{N:26}, Channel:"stable", Contact:"mailto:<email address hidden>", EditedTitle:"canonical-livepatch", EditedSummary:"Canonical Livepatch Client", EditedDescription:"Canonical Livepatch Client", Private:false}, Broken:"cannot read snap \"canonical-livepatch\": cannot find installed snap \"canonical-livepatch\" at revision 26", DownloadInfo:sn...

[Michael Vogt (mvo)]

The fix might be as simple as:

diff --git a/systemd/systemd.go b/systemd/systemd.go
index 9bd323390..74db2a79e 100644
--- a/systemd/systemd.go
+++ b/systemd/systemd.go
@@ -412,6 +412,7 @@ func (s *systemd) WriteMountUnitFile(name, what, where, fstype string) (string,

  c := fmt.Sprintf(`[Unit]
 Description=Mount unit for %s
+Before=snapd.service

 [Mount]
 What=%s

[Michael Vogt (mvo)]

A first pr is on github snapcore/snapd:#4081 - we also need to rewrite the existing mount units which is a bit more work.

[Zygmunt Krynicki (zyga)]

The bug was finally traced to the lack of systemd dependency between all the generated mount units and snapd.service itself.

[Rafael David Tinoco (rafaeldtinoco)]

Thank you Michael, Zygmunt. That is definitely in sync of our discoveries. I'll wait for the merges/SRUs to inform final user. Tks again.

[Michael Vogt (mvo)]

We will make this part of the upcoming 2.29 release.

[Rafael David Tinoco (rafaeldtinoco)]

Michael,

No pressure on my side, just wanted to know if you have a release date/cycle you can share about when 2.29 is supposed to be ready. Do you ?

[Eric Desrochers (slashd)]

With regard to LP: #1726258 -- [SRU] 2.29

The latest uploads for snapd (2.29) into bionic (deve release) and affected stable release are all stuck in -proposed with autopkgtest failures.

# rmadison
 snapd | 2.29.3~14.04 | trusty-proposed/universe | source, amd64, armhf, i386
 snapd | 2.29.3 | xenial-proposed | source, amd64, arm64, armhf, i386, powerpc, ppc64el, s390x
 snapd | 2.29.3+17.04 | zesty-proposed | source, amd64, arm64, armhf, i386, ppc64el, s390x
 snapd | 2.29.3+17.10 | artful-proposed | source, amd64, arm64, armhf, i386, ppc64el, s390x
 snapd | 2.29.3+18.04 | bionic-proposed | source, amd64, arm64, armhf, i386, ppc64el, s390x

# Pending SRU
# https://people.canonical.com/~ubuntu-archive/pending-sru.html

Artful:
snapd
Regression in autopkgtest for snapcraft (armhf): test log
Regression in autopkgtest for livecd-rootfs (s390x): test log
Regression in autopkgtest for snapd (s390x): test log
Regression in autopkgtest for snapd (i386): test log

Zesty:
snapd
Regression in autopkgtest for snapcraft (amd64): test log
Regression in autopkgtest for snapcraft (arm64): test log
Regression in autopkgtest for snapcraft (i386): test log
Regression in autopkgtest for snapd (s390x): test log
Regression in autopkgtest for snapd (i386): test log
Regression in autopkgtest for ubuntu-image (armhf): test log

Xenial:
snapd
Regression in autopkgtest for snapd (s390x): test log
Regression in autopkgtest for snapd (i386): test log
Regression in autopkgtest for snapcraft (amd64): test log
Regression in autopkgtest for snapcraft (arm64): test log
Regression in autopkgtest for snapcraft (i386): test log

Trusty:
snapd
ppc64el: Dependency wait
arm64: Dependency wait
powerpc: Dependency wait

# Excuse page ...
# https://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html

snapd (2.28.5+17.10 to 2.29.3+18.04)
Maintainer: Ubuntu Developers
11 days old
autopkgtest for livecd-rootfs/2.484: amd64: Pass, arm64: Pass, armhf: Pass, i386: Pass, ppc64el: Pass, s390x: Pass
autopkgtest for snapcraft/2.34+17.10: amd64: Ignored failure, arm64: Always failed, armhf: Ignored failure, i386: Ignored failure, ppc64el: Always failed, s390x: Always failed
autopkgtest for snapd/2.29.3+18.04: amd64: Pass, arm64: Always failed, armhf: Pass, i386: Regression ♻ , ppc64el: Pass, s390x: Pass
autopkgtest for ubuntu-image/1.1+17.10ubuntu5: amd64: Pass, arm64: Pass, armhf: Pass, i386: Pass, ppc64el: Pass, s390x: Pass
Not considered

[Eric Desrochers (slashd)]

I just talked to mvo and he will push a new package today.

[David Coronel (davecore)]

Marked as Fix Released. I tested this on snapd 2.29.4.2~14.04 and works as expected.