snapd should depend on squashfuse (for use in containers)

Bug #1628289 reported by Stéphane Graber on 2016-09-27
80
This bug affects 11 people
Affects Status Importance Assigned to Milestone
Snappy
Undecided
Unassigned
snapd (Ubuntu)
Medium
Unassigned
squashfuse (Ubuntu)
High
Unassigned
Xenial
High
Unassigned

Bug Description

We're finally making progress on the apparmor stacking and snapd in container front. The next LXD release will include the needed support as will the kernel soon afterwards.

With that, one can finally get snaps to install inside containers, but for any of it to work, squashfuse must be present in the container so that snapd can use it to mount the filesystem.

squashfuse is in the archive and I've contributed support to snapd a while back, so all that should be needed is for the snapd package to be updated to depend or at least recommend squashfuse.

Related bugs:
 * bug 1756173: snapd [SRU] 2.32

Oliver Grawert (ogra) wrote :

i only see squashfuse in yakkety universe ...
all development focus is currently on xenial, ubuntu-core is only created for xenial and images are only built for xenial. for this dependency squashfuse needs to be SRUd into xenial...

And let's make it a dependency not a recommends, please, we want lxd and
snaps to be universally available without having to check packages first.

Mark

Stéphane Graber (stgraber) wrote :

I'll upload squashfuse to xenial as an SRU, shouldn't be too controversial since it's a new package. Then I'll let the snapd folks depend on it which will trigger an MIR as it's currently in universe.

Stéphane Graber (stgraber) wrote :

And yeah, I do have snaps working inside LXD on my laptop, running nextcloud that way now. We still need a number of pieces to land in Yakkety (kernel and LXD at least) but that should be done by early next week.

Then we can focus on getting that stuff working in Xenial. We'll need a whole bunch of SRUs but nothing seems particularly hard, just time consuming (well, kernel may be hard).

Stéphane Graber (stgraber) wrote :

Uploaded to the xenial queue.

Changed in squashfuse (Ubuntu):
status: New → In Progress
assignee: nobody → Stéphane Graber (stgraber)
Oliver Grawert (ogra) wrote :

kernel wise i was actually wondering if we shouldnt perhaps focus on the -hwe lts kernels (probably a valuable topic for the sprint), but i guess the 4.8 release from yakkety might be to late for us for GA to get enough testing.

Mark Shuttleworth (sabdfl) wrote :

We can't ask people to install a new kernel to get one part of LXD. That
might work for 17.10, but not this soon after 16.04.

Mark

Stéphane Graber (stgraber) wrote :

The two kernel features we need for snaps in LXD are both scheduled for backporting to the 4.4 kernel once we're satisfied that they are stable in Yakkety's 4.8.

Mark Shuttleworth (sabdfl) wrote :

Superstars, thank you :)

Mark

Oliver Grawert (ogra) wrote :

@mark, oh, sorry, i should have mentioned that i was talking about images here :) not classic. for classic there need to be feature backports indeed.

Hello Stéphane, or anyone else affected,

Accepted squashfuse into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/squashfuse/0.1.100-0ubuntu1~ubuntu16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in squashfuse (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed
Stéphane Graber (stgraber) wrote :

Confirmed that squashfuse works as expected on Xenial.

tags: added: verification-done
removed: verification-needed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package squashfuse - 0.1.100-0ubuntu1~ubuntu16.04.1

---------------
squashfuse (0.1.100-0ubuntu1~ubuntu16.04.1) xenial; urgency=medium

  * No-change backport to xenial (LP: #1628289)

 -- Stéphane Graber <email address hidden> Thu, 29 Sep 2016 12:08:47 -0400

Changed in squashfuse (Ubuntu Xenial):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for squashfuse has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Changed in squashfuse (Ubuntu):
status: In Progress → Fix Released
Stéphane Graber (stgraber) wrote :

Marking the squashfuse side of this bug as fix released.

Now that squashfuse is available in xenial, yakkety and zesty, can someone please change snapd to depend on it so that users of snaps inside containers don't need to manually install it?

Michael Vogt (mvo) on 2016-11-30
Changed in snappy:
status: New → Fix Committed
status: Fix Committed → Fix Released
Stéphane Graber (stgraber) wrote :

Why was this marked as Fix released? As far as I can tell, snapd still doesn't depend on squashfuse...

Michael Vogt (mvo) on 2016-12-07
Changed in snappy:
status: Fix Released → Triaged
Michael Vogt (mvo) wrote :

I looked into this today and to move forward we need:
- a MIR for squashfuse as snapd is in main
- an upload of squashfuse to trusty because we support that as well now

Stéphane Graber (stgraber) wrote :

There is no point in SRUing to trusty so we won't be doing that. The reason is that trusty containers will never be able to run snapd inside them due to a decision not to enable apparmor nesting inside trusty containers.

Trusty users are expected to run xenial containers if they care about running snapd inside containers.

Stéphane Graber (stgraber) wrote :

As requested by mvo above, this bug is now the MIR for squashfuse. The same package is in zesty, yakkety and xenial so the result of the review should apply equally to all series.

## Paperwork
Availability: in universe in all relevant series
Rationale: required for snapd inside LXD containers
Security: No bugs reported against package in Ubuntu or security related issues on Github
Quality assurance:
 - Packaging standard for a fuse filesystem
 - No debconf questions
 - No bugs
 - No bugs
 - Looked at in Ubuntu (never updated but was SRUed to all releases)
 - Not hardware dependent
 - No upstream testsuite, the plan is to have snapd tested inside LXD as part of snapd autopkgtest, which will indirectly exercise squashfuse
 - Has a watch file
UI: No UI
Dependencies: Depends on a bunch of compression libraries, all in main
Standards compliance: compliant
Maintenance:
 - Ubuntu Server team subscribed to LP bugs
Background information:
 - Needed for snaps inside LXD containers where using the kernel "squashfs" isn't possible (denied for unprivileged users).
Security:
 - No existing CVE or other security report for squashfuse
 - No suid/sgid binaries
 - Code will run as "root" in the case of LXD but doesn't have to, and "root" in the case of LXD means root inside a user namespace, so effectively an unprivileged user on the host.

Michael Terry (mterry) wrote :

Resetting bug status for MIR portion. I generally find it easier to follow MIR bugs when they are separate, but it's fine, we can do it here too.

Doko, can you take this one?

Changed in squashfuse (Ubuntu):
assignee: Stéphane Graber (stgraber) → Matthias Klose (doko)
status: Fix Released → New
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in squashfuse (Ubuntu):
status: New → Confirmed

We have to be careful with new dependencies because those cause issues
for 'apt-get update'.

Mark

Stéphane Graber (stgraber) wrote :

Yeah, a recommend would be absolutely fine here and wouldn't cause "apt upgrade" to hold snapd, if I recall correctly, that's in fact what I first requested.

Matthias Klose (doko) wrote :

MIR review:

 - this looks fine, but the package doesn't come with a testsuite.
 - would it be possible to add a simple autopkg test?

Stéphane Graber (stgraber) wrote :

Should be easy enough to have an autopkgtest create a minimal squashfs filesystem and mount it, I'll upload something along those lines to Zesty. I'm assuming you don't need me to also SRU said autopkgtest to all the stable releases until we have something more user relevant to SRU right?

Stéphane Graber (stgraber) wrote :

Autopkgtest added and passed in zesty.

Stéphane Graber (stgraber) wrote :

doko tells me that we now need snapd to depend on squashfuse so we can finish the promotion to main.

mvo: Can you please do that?

Michael Vogt (mvo) wrote :
Changed in snappy:
status: Triaged → In Progress
Matthias Klose (doko) on 2017-02-27
Changed in squashfuse (Ubuntu):
assignee: Matthias Klose (doko) → nobody
Dustin Kirkland  (kirkland) wrote :

This is high priority.

Changed in squashfuse (Ubuntu):
importance: Undecided → High
Changed in squashfuse (Ubuntu Xenial):
importance: Undecided → High
John Lenton (chipaca) wrote :

I understand this is high priority, but the PR you just wrote is a poor copy of the older one from February where we attempted the same thing, and it's just as wrong for the same reasons.

Is there really no other way to pull squashfuse onto the lxd images? I don't know how they're built, but can't whatever pulls lxd also pull squashfuse in? Or whatever pulls snapd? Are those seeded? I honestly know next to nothing about how that's accomplished.

Stéphane Graber (stgraber) wrote :

squashfuse must be installed in the container that's going to run snapd not on the host, so it can't be a dependency of LXD itself.

The image that's used for the LXD containers is an unmodified Ubuntu cloud image. If we do go with seeding squashfuse for this, it would mean that it would be included in all Ubuntu cloud images, regardless of where they're run. squashfuse isn't exactly big though so that's probably still an option. The main downside to this is that it wouldn't apply to existing systems, only to newly deployed ones.

Hi,

I am facing the same issue with Ubuntu 16.04.2 LTS.

I get the below error;
####@master:~$ sudo snap install conjure-up --classic
error: cannot perform the following tasks:
- Download snap "core" (2381) from channel "stable" (Get https://068ed04f23.site.internapcdn.net/download-snap/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_2381.snap?t=2017-07-25T23:00:00Z&h=05c3f74d7c74d299152d151954a42eeb688eca41: x509: certificate signed by unknown authority)
####@master:~$

Could anyone pls help me with this.

dbclinton (dbclin) wrote :

I've installed squashfuse on my Ubuntu LXC container (under Ubuntu 16.04.2) and then tried to install the NextCloud snap, but it's still failing with this same error:

error: cannot perform the following tasks:
- Mount snap "core" (2381) ([start snap-core-2381.mount] failed with exit status 1: Job for snap-core-2381.mount failed. See "systemctl status snap-core-2381.mount" and "journalctl -xe" for details.
)

Any ideas?
Thanks,

dbclinton (dbclin) wrote :

Just to update my previous comment: poking around Stéphane Graber's blog a bit suggests to me that I really shouldn't expect success with this using less than 16.10.

Marat Khalili (mkh-t) wrote :

> Just to update my previous comment: poking around Stéphane Graber's blog a bit suggests to me that I really shouldn't expect success with this using less than 16.10.

Can someone please confirm it is really true? Bug description contains Xenial, and this discussion -- https://github.com/lxc/lxd/issues/3421 -- talks about 16.04. But it indeed doesn't work even with squashfuse. May it depend on container configuration?

It seems 16.10 and 17.04 are still affected by this problem. Don't know about 17.10 as I haven't been able to make it work properly under LXC/LXD with Juju.

Andreas Hasenack (ahasenack) wrote :

Responding to comments #36 and #37, I can confirm that a xenial container (x1) in a xenial host can install snaps after you install squashfuse.

Without squashfuse:
root@x1:~# snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (3247) ([start snap-core-3247.mount] failed with exit status 1: Job for snap-core-3247.mount failed. See "systemctl status snap-core-3247.mount" and "journalctl -xe" for details.
)

With squashfuse:
(...)
Preparing to unpack .../squashfuse_0.1.100-0ubuntu1~ubuntu16.04.1_amd64.deb ...
Unpacking squashfuse (0.1.100-0ubuntu1~ubuntu16.04.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up squashfuse (0.1.100-0ubuntu1~ubuntu16.04.1) ...
root@x1:~# snap install hello-world
hello-world 6.3 from 'canonical' installed
root@x1:~#

Marat Khalili (mkh-t) wrote :

---
root@test:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

root@test:/# apt install squashfuse
Reading package lists... Done
Building dependency tree
Reading state information... Done
squashfuse is already the newest version (0.1.100-0ubuntu1~ubuntu16.04.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@test:/# snap install nextcloud
error: cannot perform the following tasks:
- Mount snap "core" (3440) ([start snap-core-3440.mount] failed with exit status 1: Job for snap-core-3440.mount failed. See "systemctl status snap-core-3440.mount" and "journalctl -xe" for details.
)

root@test:/# snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (3440) ([start snap-core-3440.mount] failed with exit status 1: Job for snap-core-3440.mount failed. See "systemctl status snap-core-3440.mount" and "journalctl -xe" for details.
)

root@test:/#
---

What I'm doing wrong?

Marat Khalili (mkh-t) wrote :

My problem was solved by:

1. Adding the following lines to the container config, then stopping and starting the container:
---
# Make snapd work
lxc.aa_profile = unconfined
# don't drop: mac_admin mac_override
lxc.cap.drop =
lxc.cap.drop = sys_time sys_module sys_rawio

lxc.hook.autodev = sh -c 'mknod -m 666 ${LXC_ROOTFS_MOUNT}/dev/fuse c 10 229'
---

2. Executing the following lines in the container:
---
apt install snapd squashfuse fuse # last one is necessary too!
mkdir -p /lib/modules
---

There can still be some intermittent apparmor errors during snap installs, just trying again fixes them. I still wonder how it worked for others out of the box.

Stéphane Graber (stgraber) wrote :

The problem in your case is that you're trying to get this to work with LXC rather than LXD.

LXD images ship with a /lib/modules directory, ship with fuse pre-installed, LXD sets up /dev/fuse by default for you and comes with apparmor namespacing support so the container can load apparmor profiles properly.

I'd very strongly recommend against anyone using the configuration above with LXC as the lxc.aa_profile=unconfined part, combined with retaining mac_admin and mac_override will cause snapd to overwrite apparmor profiles of the host.

Marat Khalili (mkh-t) wrote :

> The problem in your case is that you're trying to get this to work with LXC rather than LXD.

Should I understand it as snapd is not currently supported under LXC? No way to reproduce manually in LXC whatever LXD does?

> I'd very strongly recommend against anyone using the configuration above with LXC as the lxc.aa_profile=unconfined [...]

Obviously not the best solution security-wise, but doesn't snapd provide its own level of isolation? It shouldn't be worse than running snapd on host, for people choosing not to rollout LXD?

(Sorry for being increasingly off-topic, but currently Google brings people wishing to install e.g. nextcloud in a container here.)

Stéphane Graber (stgraber) wrote :

On Sat, Nov 18, 2017 at 11:25:49PM +0300, Marat Khalili wrote:
> > The problem in your case is that you're trying to get this to work with LXC rather than LXD.
>
> Should I understand it as snapd is not currently supported under LXC? No way to reproduce manually in LXC whatever LXD does?

We don't directly support it in LXC, no.

If you want to replicate what LXD does, you're going to need to:
 - Bind-mount /dev/fuse
 - Allow /dev/fuse to be written to by the container
 - Create a new apparmor namespace and load the base apparmor profile into it (lxc-container-default in lxc's case)
 - Set lxc.aa_profile to point to that namespace
 - Add some kind of hook to cleanup that namespace when the container exits

> > I'd very strongly recommend against anyone using the configuration above with LXC as the lxc.aa_profile=unconfined [...]
>
> Obviously not the best solution security-wise, but doesn't snapd provide its own level of isolation? It shouldn't be worse than running snapd on host, for people choosing not to rollout LXD?
>
> (Sorry for being increasingly off-topic, but currently Google brings people wishing to install e.g. nextcloud in a container here.)

Setting aa_profile=unconfined allows your container, to load, unload and
modify any apparmor profile on the host.

That means it can interfere with any process running on the system,
crossing container boundary. It can prevent random system processes from
functioning by loading new profiles or altering existings ones, ...

Even if all workloads are trustworthy on your system, you then have the
problem that only a single container can run snaps as running multiple
containers with snapd or having snapd run on the container and in the
host will result in apparmor profile conflicts with global policies
being modified every time snap-confine runs in one of the containers or
host.

Stéphane

Frode Nordahl (fnordahl) wrote :

Following previous discussions and proposals I understand there are some issues with getting 'squashfuse' installed automatically for container workloads.

In the meantime, for improved UX, would having "snap install" command inform operator of 'squashfuse' dependency for in-container operation be something?

Scott Moser (smoser) wrote :

This is fixed in snapd under https://github.com/snapcore/snapd/pull/4049
It will be fixed in snapd 2.31.2

Scott Moser (smoser) on 2018-03-07
Changed in snap (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
status: Confirmed → Fix Committed
status: Fix Committed → Confirmed
status: Confirmed → Triaged
Changed in squashfuse (Ubuntu):
status: Confirmed → Fix Released
John Lenton (chipaca) wrote :

I removed the "snap" package bug task, as I doubt this bug affects the gene finder.

no longer affects: snap (Ubuntu)
Scott Moser (smoser) on 2018-03-07
Changed in snapd (Ubuntu):
status: New → Confirmed
importance: Undecided → Medium
Scott Moser (smoser) wrote :

Bug 1756173 is set to put 2.32 into bionic and SRU to trusty, xenial, and artful.

So if you're interested in this fix, watch that bug.

description: updated

This MIR has been completed, but squashfuse is still not in main (and apparently not Depend-ed on by snapd).

None of the referenced bugs mentions squashfuse, what is the status of this?

Changed in snapd (Ubuntu):
status: Confirmed → Incomplete
Changed in snappy:
status: In Progress → Incomplete

My understand of the status is that snapd grew its own code for mounting snaps via fuse, which would make this MIR unnecessary.

As for host restrictions I need to use LXC too (not lxd), what I needed to get this working was just:

lxc config as:

# Mount local kernel modules dir, snap needs it for its bind-mount
lxc.mount.entry = /lib/modules lib/modules none bind,create=dir,optional 0 0

# Mounting fuse (for snap squashfs)
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional

# Mount proc and sys in rw to get udev working for snaps
lxc.mount.auto=
lxc.mount.auto=proc:rw sys:rw

Once that snapd *and* fuse packages got installed (squashfuse too?) snaps worked as expected.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.