Cannot install snaps on Ubuntu 14.04 with /var on its own partition

[David Coronel]

[Impact]

Installing snaps is not possible on Ubuntu 14.04 when having /var on its own partition, whether its LVM or not.

The issue is with the core snap being unable to mount:

The error with /var isolated and using LVM:

root@ubuntu:~# snap install canonical-livepatch
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck@dev-mapper-vg00\x2dvarvol.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck@dev-mapper-vg00\x2dvarvol.service' for details.
)

The error with /var isolated but without using LVM:

root@ubuntu:~# snap install canonical-livepatch
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck@dev-disk-by\x2duuid-7383abd2\x2d019c\x2d46c2\x2d8b36\x2d34633cc8f3ca.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck@dev-disk-by\x2duuid-7383abd2\x2d019c\x2d46c2\x2d8b36\x2d34633cc8f3ca.service' for details.
)

The same error happens if I try to install the hello-world snap (with LVM in this example):

root@ubuntu:~# snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck@dev-mapper-vg00\x2dvarvol.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck@dev-mapper-vg00\x2dvarvol.service' for details.
)

I cannot reproduce the issue in Ubuntu 16.04.

I couldn't reproduce this issue by using the Ubuntu 14.04 cloud image which doesn't isolate /var on its own partition. I tried adding a secondary disk to that cloud image VM and create a dummy VG and LV, but couldn't reproduce the issue.

Also could not reproduce using Ubuntu 14.04 (with LVM or not) but with only a / partition and swap.

[Test Case]

# Install Ubuntu 14.04 in KVM (I used the 14.04.4 server iso) and configure /, /var and swap on their own partitions (with LVM or not, the issue happens in both situations).

root@ubuntu:~# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
rootvol vg00 -wi-ao--- 3.72g
swap vg00 -wi-ao--- 3.72g
varvol vg00 -wi-ao--- 3.72g

root@ubuntu:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 484M 4.0K 484M 1% /dev
tmpfs 100M 988K 99M 1% /run
/dev/dm-0 3.7G 1.7G 1.8G 49% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 497M 0 497M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/mapper/vg00-varvol 3.7G 716M 2.8G 21% /var

# Upgrade system, install snapd and reboot

root@ubuntu:~# apt update
root@ubuntu:~# apt upgrade -y
root@ubuntu:~# apt install -y snapd
root@ubuntu:~# reboot

# After reboot, check kernel version and try to install the canonical-livepatch snap:

root@ubuntu:~# uname -a
Linux ubuntu 4.4.0-96-generic #119~14.04.1-Ubuntu SMP Wed Sep 13 08:40:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

root@ubuntu:~# snap list
No snaps are installed yet. Try "snap install hello-world".

root@ubuntu:~# snap install canonical-livepatch
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck@dev-mapper-vg00\x2dvarvol.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck@dev-mapper-vg00\x2dvarvol.service' for details.
)

[Regression Potential]

- Unit file has been added to systemd, could cause an error in some units initialization. Since systemd is not used as "init" system for Trusty, this is minor regression.
- Could break all systemd units depending (After=/Wants=) systemd-fsck@.service but they are already broken.

** A "pre-SRU" test package provided in Tinoco's PPA has been also intensively tested with success, which bring even more confidence for this change. Note that the same level/quality of testing will be performed one more time when the package will be found in trusty-proposed. **

* Manual autopkgtest againts the systemd .dsc file :
adt-run [12:31:42]: @@@@@@@@@@@@@@@@@@@@ summary
timedated PASS
hostnamed PASS
localed-locale PASS
localed-x11-keymap PASS
logind PASS
api PASS

* About the autopkgtest regressions found :
https://bugs.launchpad.net/snapd/+bug/1718966/comments/14

[Other Info]

[Original Description]

Installing snaps is not possible on Ubuntu 14.04 when having /var on its own partition, whether its LVM or not.

[David Coronel (davecore)]

Additional information:

root@ubuntu:~# systemctl status systemd-fsck@dev-mapper-vg00\x2dvarvol.service
<email address hidden>
   Loaded: error (Reason: No such file or directory)
   Active: inactive (dead)

root@ubuntu:~# systemctl status systemd-fsck@dev-mapper-vg00\varvol.service
<email address hidden>
   Loaded: error (Reason: No such file or directory)
   Active: inactive (dead)

root@ubuntu:~# systemctl status --all | grep fsck
root@ubuntu:~#

Trusty doesn't have the full implementation of systemd like xenial has.

xenial:

$ ls -l /lib/systemd/system/systemd-fsck*
-rw-r--r-- 1 root root 551 Jul 18 19:56 /lib/systemd/system/systemd-fsckd.service
-rw-r--r-- 1 root root 540 Jul 18 19:56 /lib/systemd/system/systemd-fsckd.socket
-rw-r--r-- 1 root root 674 Jul 18 19:56 /lib/systemd/system/systemd-fsck-root.service
-rw-r--r-- 1 root root 648 Jul 18 19:56 /lib/systemd/system/systemd-fsck@.service

trusty:

# ls -l /lib/systemd/system/systemd-fsck*
ls: cannot access /lib/systemd/system/systemd-fsck*: No such file or directory

The system tries to call the fsck check but fails because the method or file doesn't exist:

Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck@dev-mapper-vg00\x2dvarvol.service failed to load: No such file or directory.

On a xenial system, that file would be this:
==========================
$ cat /lib/systemd/system/systemd-fsck@.service
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.

[Unit]
Description=File System Check on %f
Documentation=man:systemd-fsck@.service(8)
DefaultDependencies=no
BindsTo=%i.device
Wants=systemd-fsckd.socket
After=%i.device systemd-fsck-root.service local-fs-pre.target systemd-fsckd.socket
Before=shutdown.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/lib/systemd/systemd-fsck %f
TimeoutSec=0
==========================

And the systemctl services and units would be there on xenial:
==========================
$ systemctl status --all | grep fsck | grep ^●
● systemd-fsck-root.service - File System Check on Root Device
● systemd-fsck@dev-disk-by\x2duuid-0f1a47d4\x2d67c5\x2d4c18\x2d9159\x2dfd1d4409f30d.service - File System Check on /dev/disk/by-uuid/0f1a47d4-67c5-4c18-9159-fd1d4409f30d
● systemd-fsck@dev-disk-by\x2duuid-D233\x2dB0AC.service - File System Check on /dev/disk/by-uuid/D233-B0AC
● systemd-fsckd.service - File System Check Daemon to report status
● system-systemd\x2dfsck.slice
● systemd-fsckd.socket - fsck to fsckd communication Socket
==========================

But not on Trusty:

root@trustysnap:~# systemctl status --all | grep fsck
root@trustysnap:~#

I don't see any relevant logs in /var/log.

[Rafael David Tinoco (rafaeldtinoco)]

I was also able to reproduce this, here are my notes for now:

## /etc/fstab

LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 1
LABEL=TESTE /teste ext4 defaults 0 1

Right after boot:

inaddy@trustylivepatch:~$ systemctl list-units --all | grep fsck
systemd-...el-LVVAR.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-LVVAR.service
systemd-...el-TESTE.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-TESTE.service

This indicates that UPSTART is the one mounting the block devices, NOT SYSTEMD using its mount units. SNAPS are mounting the SQUASH filesystems using SYSTEMD UNITS, despite UPSTART scripts.

It is likely that this wasn't noticed, on systems mounting "/" only, because the "-.mount" SYSTEMD UNIT doesn't depend on "systemd-fsck@.service" unit, it depends only on "systemd-fsck-root.service", non existent in TRUSTY's SYSTEMD version. Probably this made SYSTEMD to act like no error existed.

For SYSTEMD mount units to work, it is needed that no fsck unit error exists - like when having /var or any other mounting besides root filesystem - allowing all SYSTEMD units created by snappy to work.

Comparing default setups for TRUSTY and ZESTY:

--------
## TRUSTY

$ dpkg -L systemd | grep fsck
/lib/systemd/systemd-fsck

$ systemctl list-units --all | grep fsck
systemd-...el-LVVAR.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-LVVAR.service
systemd-...el-TESTE.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-TESTE.service

$ systemctl list-units --all | grep mount
-.mount loaded active mounted /
teste.mount loaded active mounted /teste
var.mount loaded active mounted /var
umount.target loaded inactive dead Unmount All Filesystems

## ZESTY

$ dpkg -L systemd | grep fsck
/lib/systemd/system/systemd-fsck-root.service
/lib/systemd/system/systemd-fsck@.service
/lib/systemd/system/systemd-fsckd.service
/lib/systemd/system/systemd-fsckd.socket
/lib/systemd/systemd-fsck
/lib/systemd/systemd-fsckd

$ systemctl list-unit-files | grep fsck
systemd-fsck-root.service static
systemd-fsck@.service static
systemd-fsck@dev-disk-by\x2dlabel-TESTE.service static
systemd-fsckd.service static
systemd-fsckd.socket static

$ systemctl list-unit-files | grep mount
-.mount generated
home-inaddy-work.mount generated
mnt.mount static
mountall.service masked
umountfs.service masked
umountroot.service masked
umount.target static
--------

SYSTEMD in TRUSTY was treated differently for FSCK. TRUSTY's version contains systemd-fsck but not systemd-fsckd, the daemon responsible for consolidating all fsck information for SYSTEMD journal. It is also clear that TRUSTY did not include any unit file for systemd-fsck@.service, that might still be considered for the automatically generated mount unit files.

You can reproduce this by trying to use TRUSTY SYSTEMD mount units:

--------
## TRUSTY...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rafael David Tinoco (rafaeldtinoco)]

Ok so in systemd-204, in TRUSTY, you will find the following logic to create the mount units:

- manager_loop
  - process-event (WATCH_MOUNT)
    - mount_fd_event
      - mount_load_proc_self_mountinfo
        - mount_add_one (if load_extras == true)
          - mount_add_extras
            - mount_add_device_links

The last function will add: device/mount/socket/swap/path/requires/... dependencies in the mount unit to be created. The first one "device" is handled by "mount_add_device_links" and, according to logic:

        if (p->passno > 0 &&
            UNIT(m)->manager->running_as == SYSTEMD_SYSTEM) {
                char *name;
                Unit *fsck;
                /* Let's add in the fsck service */

                /* aka SPECIAL_FSCK_SERVICE */
                name = unit_name_from_path_instance("systemd-fsck", p->what, ".service");
                if (!name)
                        return -ENOMEM;

If filesystem option passno is 0, it won't have the systemd-fsck unit as a requirement, meaning that no error will be given, allowing the mount unit, created by snappy, to work.

----

Check it out:

1)

## /etc/fstab

LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 1
LABEL=TESTE /teste ext4 defaults 0 1

$ sudo snap install hello-world

error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck@dev-disk-by\x2dlabel-LVVAR.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck@dev-disk-by\x2dlabel-LVVAR.service' for details.

2)

## /etc/fstab

LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 0
LABEL=TESTE /teste ext4 defaults 0 0

$ sudo snap install hello-world
hello-world 6.3 from 'canonical' installed

----

Now I'm checking why debian/rules in systemd-204 didn't include fsck@.service files in the final .deb for "systemd" package. Possibly this wasn't noticed because at the time systemd was running in Trusty, it wasn't imagined that mount units would be managed by systemd (like snappy is doing now). I'll check if adding the units for fsck (plus having systemd-fsck binary, already there) is enough for the mount logic to work.

For now, use the workaround above ^.

[David Coronel (davecore)]

I confirm the workaround works for me:

Before:

root@ubuntu:~# cat /etc/fstab
/dev/mapper/vg00-rootvol / ext4 errors=remount-ro 0 1
/dev/mapper/vg00-varvol /var ext4 defaults 0 2
/dev/mapper/vg00-swap none swap sw 0 0

root@ubuntu:~# snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck@dev-mapper-vg00\x2dvarvol.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck@dev-mapper-vg00\x2dvarvol.service' for details.
)

After:

root@ubuntu:~# cat /etc/fstab
/dev/mapper/vg00-rootvol / ext4 errors=remount-ro 0 1
/dev/mapper/vg00-varvol /var ext4 defaults 0 0
/dev/mapper/vg00-swap none swap sw 0 0

root@ubuntu:~# snap install hello-world
hello-world 6.3 from 'canonical' installed

root@ubuntu:~# snap install canonical-livepatch
canonical-livepatch 7 from 'canonical' installed

root@ubuntu:~# sudo canonical-livepatch enable <redacted>

root@ubuntu:~# canonical-livepatch status
kernel: 4.4.0-96.119~14.04.1-generic
fully-patched: true
version: ""

Thanks Rafael!

[Rafael David Tinoco (rafaeldtinoco)]

Okay, so another quick workaround, now keeping the fsck enabled for the volumes:

Create a file called "/lib/systemd/upstart/systemd-fsck@.service" with:

----
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.

[Unit]
Description=File System Check on %f
Documentation=man:systemd-fsck@.service(8)
DefaultDependencies=no
BindsTo=%i.device
After=systemd-readahead-collect.service systemd-readahead-replay.service %i.device
Before=shutdown.target

[Service]
Type=oneshot
RemainAfterExit=no
ExecStart=/lib/systemd/systemd-fsck %f
StandardOutput=journal+console
TimeoutSec=0
----

and then do:

$ sudo ln -s /lib/systemd/upstart/systemd-fsck@.service /lib/systemd/upstart/multi-user.target.wants/systemd-fsck@.service
$ sudo systemctl daemon-reload

This will allow the mount units to work using systemd in Trusty:

inaddy@trustylivepatch:~$ sudo systemctl start teste.mount
inaddy@trustylivepatch:~$ df -kh | grep teste
/dev/sdb2 976M 1.3M 908M 1% /teste
inaddy@trustylivepatch:~$ sudo systemctl stop teste.mount
inaddy@trustylivepatch:~$ cat /etc/fstab| grep teste
LABEL=TESTE /teste ext4 defaults 0 1

While still having fsck enabled in /etc/fstab.

I'll provide a SRU patch for systemd in Trusty and it shall solve this issue.

[Rafael David Tinoco (rafaeldtinoco)]

I've made a small change creating a unit file slightly different from the one already present in systemd-204, considering different dependencies for this usage (mounting/umounting volumes when upstart has already taken part on it).

The result is here: http://pastebin.ubuntu.com/25663762/

As you can see, after installing the new packages, being proposed in this fix, snapd can work again mounting all new squash filesystems it depends on.

- Attaching the debdiff for SRU.
- Requesting SRU sponsoring.
- Will verify once it gets uploaded to -proposed.

[Rafael David Tinoco (rafaeldtinoco)]

Meanwhile I have created a PPA containing the fix for all those affected users:

https://launchpad.net/~inaddy/+archive/ubuntu/lp1718966

$ sudo add-apt-repository ppa:inaddy/lp1718966
$ sudo apt-get update
$ sudo apt-get dist-upgrade

Is likely enough for you to have a temporary fix.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "trusty_systemd_204-5ubuntu20.25.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[David Coronel (davecore)]

I tested the fix from ppa:inaddy/lp1718966 and I am now able to install snaps on my two test systems without using the /etc/fstab workaround, one with LVM and one without. Both systems have /var isolated. Thanks!

[Eric Desrochers (slashd)]

Sponsored, the package is now in the Trusty upload queue, waiting for the SRU verification team to approve the package to start building in trusty-proposed.

- Eric

[Rafael David Tinoco (rafaeldtinoco)]

After fixing this BUG (for canonical-livepatch snap to work, but being affected by broken systemd in Trusty due to the lack of systemd-fsck@.service unit file, I'm facing the following opened BUG: https://bugs.launchpad.net/snapd/+bug/1721518 for canonical-livepatch (or any other snap) to work).

[Łukasz Zemczak (sil2100)]

Hello David, or anyone else affected,

Accepted systemd into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/204-5ubuntu20.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Eric Desrochers (slashd)]

## PENDING SRU / REGRESSION EXPLANATIONS **

All the current systemd (trusty) autopktest failure are not related to the current SRU.
Seems like they are failing for a while now and should be considered as known test failure already.

Regression in autopkgtest for network-manager (ppc64el): test log
 - Always fails the same way since at least "systemd/204-5ubuntu20.17" : http://autopkgtest.ubuntu.com/packages/n/network-manager/trusty/ppc64el

Regression in autopkgtest for linux-lts-wily (armhf): test log
 - Always timeout the same way since at least "systemd/204-5ubuntu20.17" : http://autopkgtest.ubuntu.com/packages/l/linux-lts-wily/trusty/armhf

Regression in autopkgtest for linux-lts-vivid (ppc64el): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-vivid (i386): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-vivid (amd64): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-vivid (armhf): test log
 - Always fails the same way since at least "systemd/204-5ubuntu20.14" : http://autopkgtest.ubuntu.com/packages/l/linux-lts-vivid/trusty/armhf

Regression in autopkgtest for linux-lts-utopic (ppc64el): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-utopic (i386): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-utopic (amd64): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-utopic (armhf): test log
 - Always fails the same way since at least "systemd/204-5ubuntu20.15" : http://autopkgtest.ubuntu.com/packages/l/linux-lts-utopic/trusty/armhf

Regression in autopkgtest for linux-lts-xenial (ppc64el): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-xenial (i386): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-xenial (amd64): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-xenial (armhf): test log
 - Always fails the same way since "systemd/204-5ubuntu20.19" : http://autopkgtest.ubuntu.com/packages/l/linux-lts-xenial/trusty/armhf

** ERROR: running version does not match source package **
Source Package Version: 3.16.0-77.99~14.04.1
Running Kernel Version: 3.13.0-133.182

- Eric

[Rafael David Tinoco (rafaeldtinoco)]

Verifying...

http://pastebin.ubuntu.com/25753116/

After using -proposed version:

http://paste.ubuntu.com/25753146/

Considering this fixed. Verified!

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 204-5ubuntu20.25

---------------
systemd (204-5ubuntu20.25) trusty; urgency=medium

  * rules: introduce fsck@.service for snappy (LP: #1718966)

 -- Rafael David Tinoco <email address hidden> Mon, 02 Oct 2017 21:39:38 +0000

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.