Cannot install snaps on Ubuntu 14.04 with /var on its own partition
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| snapd |
Undecided
|
Rafael David Tinoco | ||
| systemd (Ubuntu) |
Medium
|
Rafael David Tinoco | ||
| Trusty |
Medium
|
Rafael David Tinoco |
Bug Description
[Impact]
Installing snaps is not possible on Ubuntu 14.04 when having /var on its own partition, whether its LVM or not.
The issue is with the core snap being unable to mount:
The error with /var isolated and using LVM:
root@ubuntu:~# snap install canonical-livepatch
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-
)
The error with /var isolated but without using LVM:
root@ubuntu:~# snap install canonical-livepatch
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-
)
The same error happens if I try to install the hello-world snap (with LVM in this example):
root@ubuntu:~# snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-
)
I cannot reproduce the issue in Ubuntu 16.04.
I couldn't reproduce this issue by using the Ubuntu 14.04 cloud image which doesn't isolate /var on its own partition. I tried adding a secondary disk to that cloud image VM and create a dummy VG and LV, but couldn't reproduce the issue.
Also could not reproduce using Ubuntu 14.04 (with LVM or not) but with only a / partition and swap.
[Test Case]
# Install Ubuntu 14.04 in KVM (I used the 14.04.4 server iso) and configure /, /var and swap on their own partitions (with LVM or not, the issue happens in both situations).
root@ubuntu:~# lvs
LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
rootvol vg00 -wi-ao--- 3.72g
swap vg00 -wi-ao--- 3.72g
varvol vg00 -wi-ao--- 3.72g
root@ubuntu:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 484M 4.0K 484M 1% /dev
tmpfs 100M 988K 99M 1% /run
/dev/dm-0 3.7G 1.7G 1.8G 49% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 497M 0 497M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/mapper/
# Upgrade system, install snapd and reboot
root@ubuntu:~# apt update
root@ubuntu:~# apt upgrade -y
root@ubuntu:~# apt install -y snapd
root@ubuntu:~# reboot
# After reboot, check kernel version and try to install the canonical-livepatch snap:
root@ubuntu:~# uname -a
Linux ubuntu 4.4.0-96-generic #119~14.04.1-Ubuntu SMP Wed Sep 13 08:40:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:~# snap list
No snaps are installed yet. Try "snap install hello-world".
root@ubuntu:~# snap install canonical-livepatch
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-
)
[Regression Potential]
- Unit file has been added to systemd, could cause an error in some units initialization. Since systemd is not used as "init" system for Trusty, this is minor regression.
- Could break all systemd units depending (After=/Wants=) systemd-
** A "pre-SRU" test package provided in Tinoco's PPA has been also intensively tested with success, which bring even more confidence for this change. Note that the same level/quality of testing will be performed one more time when the package will be found in trusty-proposed. **
* Manual autopkgtest againts the systemd .dsc file :
adt-run [12:31:42]: @@@@@@@
timedated PASS
hostnamed PASS
localed-locale PASS
localed-x11-keymap PASS
logind PASS
api PASS
* About the autopkgtest regressions found :
https:/
[Other Info]
[Original Description]
Installing snaps is not possible on Ubuntu 14.04 when having /var on its own partition, whether its LVM or not.
David Coronel (davecore) wrote : | #1 |
I was also able to reproduce this, here are my notes for now:
## /etc/fstab
LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 1
LABEL=TESTE /teste ext4 defaults 0 1
Right after boot:
inaddy@
systemd-
systemd-
This indicates that UPSTART is the one mounting the block devices, NOT SYSTEMD using its mount units. SNAPS are mounting the SQUASH filesystems using SYSTEMD UNITS, despite UPSTART scripts.
It is likely that this wasn't noticed, on systems mounting "/" only, because the "-.mount" SYSTEMD UNIT doesn't depend on "systemd-
For SYSTEMD mount units to work, it is needed that no fsck unit error exists - like when having /var or any other mounting besides root filesystem - allowing all SYSTEMD units created by snappy to work.
Comparing default setups for TRUSTY and ZESTY:
--------
## TRUSTY
$ dpkg -L systemd | grep fsck
/lib/systemd/
$ systemctl list-units --all | grep fsck
systemd-
systemd-
$ systemctl list-units --all | grep mount
-.mount loaded active mounted /
teste.mount loaded active mounted /teste
var.mount loaded active mounted /var
umount.target loaded inactive dead Unmount All Filesystems
## ZESTY
$ dpkg -L systemd | grep fsck
/lib/systemd/
/lib/systemd/
/lib/systemd/
/lib/systemd/
/lib/systemd/
/lib/systemd/
$ systemctl list-unit-files | grep fsck
systemd-
systemd-
systemd-
systemd-
systemd-
$ systemctl list-unit-files | grep mount
-.mount generated
home-inaddy-
mnt.mount static
mountall.service masked
umountfs.service masked
umountroot.service masked
umount.target static
--------
SYSTEMD in TRUSTY was treated differently for FSCK. TRUSTY's version contains systemd-fsck but not systemd-fsckd, the daemon responsible for consolidating all fsck information for SYSTEMD journal. It is also clear that TRUSTY did not include any unit file for systemd-
You can reproduce this by trying to use TRUSTY SYSTEMD mount units:
--------
## TRUSTY...
Changed in snapd: | |
assignee: | nobody → Rafael David Tinoco (inaddy) |
status: | New → Confirmed |
Changed in systemd (Ubuntu): | |
assignee: | nobody → Rafael David Tinoco (inaddy) |
importance: | Undecided → Medium |
status: | New → In Progress |
Launchpad Janitor (janitor) wrote : | #3 |
Status changed to 'Confirmed' because the bug affects multiple users.
Changed in systemd (Ubuntu Trusty): | |
status: | New → Confirmed |
Changed in systemd (Ubuntu Trusty): | |
assignee: | nobody → Rafael David Tinoco (inaddy) |
Ok so in systemd-204, in TRUSTY, you will find the following logic to create the mount units:
- manager_loop
- process-event (WATCH_MOUNT)
- mount_fd_event
- mount_load_
- mount_add_one (if load_extras == true)
- mount_add_extras
- mount_add_
The last function will add: device/
if (p->passno > 0 &&
/* Let's add in the fsck service */
/* aka SPECIAL_
if (!name)
If filesystem option passno is 0, it won't have the systemd-fsck unit as a requirement, meaning that no error will be given, allowing the mount unit, created by snappy, to work.
----
Check it out:
1)
## /etc/fstab
LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 1
LABEL=TESTE /teste ext4 defaults 0 1
$ sudo snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-
2)
## /etc/fstab
LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 0
LABEL=TESTE /teste ext4 defaults 0 0
$ sudo snap install hello-world
hello-world 6.3 from 'canonical' installed
----
Now I'm checking why debian/rules in systemd-204 didn't include fsck@.service files in the final .deb for "systemd" package. Possibly this wasn't noticed because at the time systemd was running in Trusty, it wasn't imagined that mount units would be managed by systemd (like snappy is doing now). I'll check if adding the units for fsck (plus having systemd-fsck binary, already there) is enough for the mount logic to work.
For now, use the workaround above ^.
Changed in systemd (Ubuntu Trusty): | |
status: | Confirmed → In Progress |
Changed in snapd: | |
status: | Confirmed → In Progress |
David Coronel (davecore) wrote : | #5 |
I confirm the workaround works for me:
Before:
root@ubuntu:~# cat /etc/fstab
/dev/mapper/
/dev/mapper/
/dev/mapper/
root@ubuntu:~# snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-
)
After:
root@ubuntu:~# cat /etc/fstab
/dev/mapper/
/dev/mapper/
/dev/mapper/
root@ubuntu:~# snap install hello-world
hello-world 6.3 from 'canonical' installed
root@ubuntu:~# snap install canonical-livepatch
canonical-livepatch 7 from 'canonical' installed
root@ubuntu:~# sudo canonical-livepatch enable <redacted>
root@ubuntu:~# canonical-livepatch status
kernel: 4.4.0-96.
fully-patched: true
version: ""
Thanks Rafael!
Okay, so another quick workaround, now keeping the fsck enabled for the volumes:
Create a file called "/lib/systemd/
----
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
[Unit]
Description=File System Check on %f
Documentation=
DefaultDependen
BindsTo=%i.device
After=systemd-
Before=
[Service]
Type=oneshot
RemainAfterExit=no
ExecStart=
StandardOutput=
TimeoutSec=0
----
and then do:
$ sudo ln -s /lib/systemd/
$ sudo systemctl daemon-reload
This will allow the mount units to work using systemd in Trusty:
inaddy@
inaddy@
/dev/sdb2 976M 1.3M 908M 1% /teste
inaddy@
inaddy@
LABEL=TESTE /teste ext4 defaults 0 1
While still having fsck enabled in /etc/fstab.
I'll provide a SRU patch for systemd in Trusty and it shall solve this issue.
Changed in snapd: | |
status: | In Progress → Invalid |
I've made a small change creating a unit file slightly different from the one already present in systemd-204, considering different dependencies for this usage (mounting/umounting volumes when upstart has already taken part on it).
The result is here: http://
As you can see, after installing the new packages, being proposed in this fix, snapd can work again mounting all new squash filesystems it depends on.
- Attaching the debdiff for SRU.
- Requesting SRU sponsoring.
- Will verify once it gets uploaded to -proposed.
tags: | added: sts sts-sponsor |
Meanwhile I have created a PPA containing the fix for all those affected users:
https:/
$ sudo add-apt-repository ppa:inaddy/
$ sudo apt-get update
$ sudo apt-get dist-upgrade
Is likely enough for you to have a temporary fix.
The attachment "trusty_
[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]
tags: | added: patch |
description: | updated |
Changed in systemd (Ubuntu Trusty): | |
importance: | Undecided → Medium |
description: | updated |
David Coronel (davecore) wrote : | #10 |
I tested the fix from ppa:inaddy/
Eric Desrochers (slashd) wrote : | #11 |
Sponsored, the package is now in the Trusty upload queue, waiting for the SRU verification team to approve the package to start building in trusty-proposed.
- Eric
tags: |
added: sts-sponsor-done sts-sru-needed removed: sts-sponsor |
description: | updated |
description: | updated |
After fixing this BUG (for canonical-livepatch snap to work, but being affected by broken systemd in Trusty due to the lack of systemd-
Hello David, or anyone else affected,
Accepted systemd into trusty-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
Changed in systemd (Ubuntu): | |
status: | In Progress → Invalid |
Changed in systemd (Ubuntu Trusty): | |
status: | In Progress → Fix Committed |
tags: | added: verification-needed verification-needed-trusty |
Eric Desrochers (slashd) wrote : | #14 |
## PENDING SRU / REGRESSION EXPLANATIONS **
All the current systemd (trusty) autopktest failure are not related to the current SRU.
Seems like they are failing for a while now and should be considered as known test failure already.
Regression in autopkgtest for network-manager (ppc64el): test log
- Always fails the same way since at least "systemd/
Regression in autopkgtest for linux-lts-wily (armhf): test log
- Always timeout the same way since at least "systemd/
Regression in autopkgtest for linux-lts-vivid (ppc64el): test log
- ERROR: running version does not match source package
Regression in autopkgtest for linux-lts-vivid (i386): test log
- ERROR: running version does not match source package
Regression in autopkgtest for linux-lts-vivid (amd64): test log
- ERROR: running version does not match source package
Regression in autopkgtest for linux-lts-vivid (armhf): test log
- Always fails the same way since at least "systemd/
Regression in autopkgtest for linux-lts-utopic (ppc64el): test log
- ERROR: running version does not match source package
Regression in autopkgtest for linux-lts-utopic (i386): test log
- ERROR: running version does not match source package
Regression in autopkgtest for linux-lts-utopic (amd64): test log
- ERROR: running version does not match source package
Regression in autopkgtest for linux-lts-utopic (armhf): test log
- Always fails the same way since at least "systemd/
Regression in autopkgtest for linux-lts-xenial (ppc64el): test log
- ERROR: running version does not match source package
Regression in autopkgtest for linux-lts-xenial (i386): test log
- ERROR: running version does not match source package
Regression in autopkgtest for linux-lts-xenial (amd64): test log
- ERROR: running version does not match source package
Regression in autopkgtest for linux-lts-xenial (armhf): test log
- Always fails the same way since "systemd/
** ERROR: running version does not match source package **
Source Package Version: 3.16.0-
Running Kernel Version: 3.13.0-133.182
- Eric
description: | updated |
Verifying...
http://
After using -proposed version:
http://
Considering this fixed. Verified!
tags: |
added: verification-done verification-done-trusty removed: verification-needed verification-needed-trusty |
tags: | removed: patch |
Launchpad Janitor (janitor) wrote : | #16 |
This bug was fixed in the package systemd - 204-5ubuntu20.25
---------------
systemd (204-5ubuntu20.25) trusty; urgency=medium
* rules: introduce fsck@.service for snappy (LP: #1718966)
-- Rafael David Tinoco <email address hidden> Mon, 02 Oct 2017 21:39:38 +0000
Changed in systemd (Ubuntu Trusty): | |
status: | Fix Committed → Fix Released |
The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
Additional information:
root@ubuntu:~# systemctl status systemd- fsck@dev- mapper- vg00\x2dvarvol. service
<email address hidden>
Loaded: error (Reason: No such file or directory)
Active: inactive (dead)
root@ubuntu:~# systemctl status systemd- fsck@dev- mapper- vg00\varvol. service
<email address hidden>
Loaded: error (Reason: No such file or directory)
Active: inactive (dead)
root@ubuntu:~# systemctl status --all | grep fsck
root@ubuntu:~#
Trusty doesn't have the full implementation of systemd like xenial has.
xenial:
$ ls -l /lib/systemd/ system/ systemd- fsck* system/ systemd- fsckd.service system/ systemd- fsckd.socket system/ systemd- fsck-root. service system/ systemd- fsck@.service
-rw-r--r-- 1 root root 551 Jul 18 19:56 /lib/systemd/
-rw-r--r-- 1 root root 540 Jul 18 19:56 /lib/systemd/
-rw-r--r-- 1 root root 674 Jul 18 19:56 /lib/systemd/
-rw-r--r-- 1 root root 648 Jul 18 19:56 /lib/systemd/
trusty:
# ls -l /lib/systemd/ system/ systemd- fsck* system/ systemd- fsck*: No such file or directory
ls: cannot access /lib/systemd/
The system tries to call the fsck check but fails because the method or file doesn't exist:
Mount snap "core" (2898) ([start snap-core- 2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd- fsck@dev- mapper- vg00\x2dvarvol. service failed to load: No such file or directory.
On a xenial system, that file would be this: ======= ======= ===== system/ systemd- fsck@.service
=======
$ cat /lib/systemd/
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
[Unit] man:systemd- fsck@.service( 8) cies=no fsckd.socket fsck-root. service local-fs-pre.target systemd- fsckd.socket shutdown. target
Description=File System Check on %f
Documentation=
DefaultDependen
BindsTo=%i.device
Wants=systemd-
After=%i.device systemd-
Before=
[Service] /lib/systemd/ systemd- fsck %f ======= ======= =====
Type=oneshot
RemainAfterExit=yes
ExecStart=
TimeoutSec=0
=======
And the systemctl services and units would be there on xenial: ======= ======= ===== fsck-root. service - File System Check on Root Device fsck@dev- disk-by\ x2duuid- 0f1a47d4\ x2d67c5\ x2d4c18\ x2d9159\ x2dfd1d4409f30d .service - File System Check on /dev/disk/ by-uuid/ 0f1a47d4- 67c5-4c18- 9159-fd1d4409f3 0d fsck@dev- disk-by\ x2duuid- D233\x2dB0AC. service - File System Check on /dev/disk/ by-uuid/ D233-B0AC fsckd.service - File System Check Daemon to report status systemd\ x2dfsck. slice fsckd.socket - fsck to fsckd communication Socket ======= ======= =====
=======
$ systemctl status --all | grep fsck | grep ^●
● systemd-
● systemd-
● systemd-
● systemd-
● system-
● systemd-
=======
But not on Trusty:
root@trustysnap:~# systemctl status --all | grep fsck
root@trustysnap:~#
I don't see any relevant logs in /var/log.