snapd should depend on squashfuse (for use in containers)

[Stéphane Graber]

We're finally making progress on the apparmor stacking and snapd in container front. The next LXD release will include the needed support as will the kernel soon afterwards.

With that, one can finally get snaps to install inside containers, but for any of it to work, squashfuse must be present in the container so that snapd can use it to mount the filesystem.

squashfuse is in the archive and I've contributed support to snapd a while back, so all that should be needed is for the snapd package to be updated to depend or at least recommend squashfuse.

Related bugs:
 * bug 1756173: snapd [SRU] 2.32

[Oliver Grawert (ogra)]

i only see squashfuse in yakkety universe ...
all development focus is currently on xenial, ubuntu-core is only created for xenial and images are only built for xenial. for this dependency squashfuse needs to be SRUd into xenial...

[Mark Shuttleworth (sabdfl)]

And let's make it a dependency not a recommends, please, we want lxd and
snaps to be universally available without having to check packages first.

Mark

[Stéphane Graber (stgraber)]

I'll upload squashfuse to xenial as an SRU, shouldn't be too controversial since it's a new package. Then I'll let the snapd folks depend on it which will trigger an MIR as it's currently in universe.

[Stéphane Graber (stgraber)]

And yeah, I do have snaps working inside LXD on my laptop, running nextcloud that way now. We still need a number of pieces to land in Yakkety (kernel and LXD at least) but that should be done by early next week.

Then we can focus on getting that stuff working in Xenial. We'll need a whole bunch of SRUs but nothing seems particularly hard, just time consuming (well, kernel may be hard).

[Stéphane Graber (stgraber)]

Uploaded to the xenial queue.

[Oliver Grawert (ogra)]

kernel wise i was actually wondering if we shouldnt perhaps focus on the -hwe lts kernels (probably a valuable topic for the sprint), but i guess the 4.8 release from yakkety might be to late for us for GA to get enough testing.

[Mark Shuttleworth (sabdfl)]

We can't ask people to install a new kernel to get one part of LXD. That
might work for 17.10, but not this soon after 16.04.

Mark

[Stéphane Graber (stgraber)]

The two kernel features we need for snaps in LXD are both scheduled for backporting to the 4.4 kernel once we're satisfied that they are stable in Yakkety's 4.8.

[Mark Shuttleworth (sabdfl)]

Superstars, thank you :)

Mark

[Oliver Grawert (ogra)]

@mark, oh, sorry, i should have mentioned that i was talking about images here :) not classic. for classic there need to be feature backports indeed.

[Steve Langasek (vorlon)]

Hello Stéphane, or anyone else affected,

Accepted squashfuse into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/squashfuse/0.1.100-0ubuntu1~ubuntu16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Stéphane Graber (stgraber)]

Confirmed that squashfuse works as expected on Xenial.

[Launchpad Janitor (janitor)]

This bug was fixed in the package squashfuse - 0.1.100-0ubuntu1~ubuntu16.04.1

---------------
squashfuse (0.1.100-0ubuntu1~ubuntu16.04.1) xenial; urgency=medium

  * No-change backport to xenial (LP: #1628289)

 -- Stéphane Graber <email address hidden> Thu, 29 Sep 2016 12:08:47 -0400

[Stéphane Graber (stgraber)]

The verification of the Stable Release Update for squashfuse has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Stéphane Graber (stgraber)]

Marking the squashfuse side of this bug as fix released.

Now that squashfuse is available in xenial, yakkety and zesty, can someone please change snapd to depend on it so that users of snaps inside containers don't need to manually install it?

[Stéphane Graber (stgraber)]

Why was this marked as Fix released? As far as I can tell, snapd still doesn't depend on squashfuse...

[Michael Vogt (mvo)]

I looked into this today and to move forward we need:
- a MIR for squashfuse as snapd is in main
- an upload of squashfuse to trusty because we support that as well now

[Stéphane Graber (stgraber)]

There is no point in SRUing to trusty so we won't be doing that. The reason is that trusty containers will never be able to run snapd inside them due to a decision not to enable apparmor nesting inside trusty containers.

Trusty users are expected to run xenial containers if they care about running snapd inside containers.

[Stéphane Graber (stgraber)]

As requested by mvo above, this bug is now the MIR for squashfuse. The same package is in zesty, yakkety and xenial so the result of the review should apply equally to all series.

## Paperwork
Availability: in universe in all relevant series
Rationale: required for snapd inside LXD containers
Security: No bugs reported against package in Ubuntu or security related issues on Github
Quality assurance:
 - Packaging standard for a fuse filesystem
 - No debconf questions
 - No bugs
 - No bugs
 - Looked at in Ubuntu (never updated but was SRUed to all releases)
 - Not hardware dependent
 - No upstream testsuite, the plan is to have snapd tested inside LXD as part of snapd autopkgtest, which will indirectly exercise squashfuse
 - Has a watch file
UI: No UI
Dependencies: Depends on a bunch of compression libraries, all in main
Standards compliance: compliant
Maintenance:
 - Ubuntu Server team subscribed to LP bugs
Background information:
 - Needed for snaps inside LXD containers where using the kernel "squashfs" isn't possible (denied for unprivileged users).
Security:
 - No existing CVE or other security report for squashfuse
 - No suid/sgid binaries
 - Code will run as "root" in the case of LXD but doesn't have to, and "root" in the case of LXD means root inside a user namespace, so effectively an unprivileged user on the host.

[Michael Terry (mterry)]

Resetting bug status for MIR portion. I generally find it easier to follow MIR bugs when they are separate, but it's fine, we can do it here too.

Doko, can you take this one?

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mark Shuttleworth (sabdfl)]

We have to be careful with new dependencies because those cause issues
for 'apt-get update'.

Mark

[Stéphane Graber (stgraber)]

Yeah, a recommend would be absolutely fine here and wouldn't cause "apt upgrade" to hold snapd, if I recall correctly, that's in fact what I first requested.

[Matthias Klose (doko)]

MIR review:

 - this looks fine, but the package doesn't come with a testsuite.
 - would it be possible to add a simple autopkg test?

[Stéphane Graber (stgraber)]

Should be easy enough to have an autopkgtest create a minimal squashfs filesystem and mount it, I'll upload something along those lines to Zesty. I'm assuming you don't need me to also SRU said autopkgtest to all the stable releases until we have something more user relevant to SRU right?

[Stéphane Graber (stgraber)]

Autopkgtest added and passed in zesty.

[Stéphane Graber (stgraber)]

doko tells me that we now need snapd to depend on squashfuse so we can finish the promotion to main.

mvo: Can you please do that?

[Michael Vogt (mvo)]

https://github.com/snapcore/snapd/pull/2830

[Stéphane Graber (stgraber)]

You meant https://github.com/snapcore/snapd/pull/2856

[Dustin Kirkland  (kirkland)]

This is high priority.

[Dustin Kirkland  (kirkland)]

Here's a pull request:

https://github.com/snapcore/snapd/pull/3605

[John Lenton (chipaca)]

I understand this is high priority, but the PR you just wrote is a poor copy of the older one from February where we attempted the same thing, and it's just as wrong for the same reasons.

Is there really no other way to pull squashfuse onto the lxd images? I don't know how they're built, but can't whatever pulls lxd also pull squashfuse in? Or whatever pulls snapd? Are those seeded? I honestly know next to nothing about how that's accomplished.

[Stéphane Graber (stgraber)]

squashfuse must be installed in the container that's going to run snapd not on the host, so it can't be a dependency of LXD itself.

The image that's used for the LXD containers is an unmodified Ubuntu cloud image. If we do go with seeding squashfuse for this, it would mean that it would be included in all Ubuntu cloud images, regardless of where they're run. squashfuse isn't exactly big though so that's probably still an option. The main downside to this is that it wouldn't apply to existing systems, only to newly deployed ones.

[Magesh Kumar Rajamani (mageshmcc)]

Hi,

I am facing the same issue with Ubuntu 16.04.2 LTS.

I get the below error;
####@master:~$ sudo snap install conjure-up --classic
error: cannot perform the following tasks:
- Download snap "core" (2381) from channel "stable" (Get https://068ed04f23.site.internapcdn.net/download-snap/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_2381.snap?t=2017-07-25T23:00:00Z&h=05c3f74d7c74d299152d151954a42eeb688eca41: x509: certificate signed by unknown authority)
####@master:~$

Could anyone pls help me with this.

[dbclinton (dbclin)]

I've installed squashfuse on my Ubuntu LXC container (under Ubuntu 16.04.2) and then tried to install the NextCloud snap, but it's still failing with this same error:

error: cannot perform the following tasks:
- Mount snap "core" (2381) ([start snap-core-2381.mount] failed with exit status 1: Job for snap-core-2381.mount failed. See "systemctl status snap-core-2381.mount" and "journalctl -xe" for details.
)

Any ideas?
Thanks,

[dbclinton (dbclin)]

Just to update my previous comment: poking around Stéphane Graber's blog a bit suggests to me that I really shouldn't expect success with this using less than 16.10.

[Marat Khalili (mkh-t)]

> Just to update my previous comment: poking around Stéphane Graber's blog a bit suggests to me that I really shouldn't expect success with this using less than 16.10.

Can someone please confirm it is really true? Bug description contains Xenial, and this discussion -- https://github.com/lxc/lxd/issues/3421 -- talks about 16.04. But it indeed doesn't work even with squashfuse. May it depend on container configuration?

[Felipe Alfaro Solana (felipe-alfaro-gmail)]

It seems 16.10 and 17.04 are still affected by this problem. Don't know about 17.10 as I haven't been able to make it work properly under LXC/LXD with Juju.

[Andreas Hasenack (ahasenack)]

Responding to comments #36 and #37, I can confirm that a xenial container (x1) in a xenial host can install snaps after you install squashfuse.

Without squashfuse:
root@x1:~# snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (3247) ([start snap-core-3247.mount] failed with exit status 1: Job for snap-core-3247.mount failed. See "systemctl status snap-core-3247.mount" and "journalctl -xe" for details.
)

With squashfuse:
(...)
Preparing to unpack .../squashfuse_0.1.100-0ubuntu1~ubuntu16.04.1_amd64.deb ...
Unpacking squashfuse (0.1.100-0ubuntu1~ubuntu16.04.1) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up squashfuse (0.1.100-0ubuntu1~ubuntu16.04.1) ...
root@x1:~# snap install hello-world
hello-world 6.3 from 'canonical' installed
root@x1:~#

[Marat Khalili (mkh-t)]

---
root@test:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.3 LTS
Release: 16.04
Codename: xenial

root@test:/# apt install squashfuse
Reading package lists... Done
Building dependency tree
Reading state information... Done
squashfuse is already the newest version (0.1.100-0ubuntu1~ubuntu16.04.1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

root@test:/# snap install nextcloud
error: cannot perform the following tasks:
- Mount snap "core" (3440) ([start snap-core-3440.mount] failed with exit status 1: Job for snap-core-3440.mount failed. See "systemctl status snap-core-3440.mount" and "journalctl -xe" for details.
)

root@test:/# snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (3440) ([start snap-core-3440.mount] failed with exit status 1: Job for snap-core-3440.mount failed. See "systemctl status snap-core-3440.mount" and "journalctl -xe" for details.
)

root@test:/#
---

What I'm doing wrong?

[Marat Khalili (mkh-t)]

My problem was solved by:

1. Adding the following lines to the container config, then stopping and starting the container:
---
# Make snapd work
lxc.aa_profile = unconfined
# don't drop: mac_admin mac_override
lxc.cap.drop =
lxc.cap.drop = sys_time sys_module sys_rawio

lxc.hook.autodev = sh -c 'mknod -m 666 ${LXC_ROOTFS_MOUNT}/dev/fuse c 10 229'
---

2. Executing the following lines in the container:
---
apt install snapd squashfuse fuse # last one is necessary too!
mkdir -p /lib/modules
---

There can still be some intermittent apparmor errors during snap installs, just trying again fixes them. I still wonder how it worked for others out of the box.

[Stéphane Graber (stgraber)]

The problem in your case is that you're trying to get this to work with LXC rather than LXD.

LXD images ship with a /lib/modules directory, ship with fuse pre-installed, LXD sets up /dev/fuse by default for you and comes with apparmor namespacing support so the container can load apparmor profiles properly.

I'd very strongly recommend against anyone using the configuration above with LXC as the lxc.aa_profile=unconfined part, combined with retaining mac_admin and mac_override will cause snapd to overwrite apparmor profiles of the host.

[Marat Khalili (mkh-t)]

> The problem in your case is that you're trying to get this to work with LXC rather than LXD.

Should I understand it as snapd is not currently supported under LXC? No way to reproduce manually in LXC whatever LXD does?

> I'd very strongly recommend against anyone using the configuration above with LXC as the lxc.aa_profile=unconfined [...]

Obviously not the best solution security-wise, but doesn't snapd provide its own level of isolation? It shouldn't be worse than running snapd on host, for people choosing not to rollout LXD?

(Sorry for being increasingly off-topic, but currently Google brings people wishing to install e.g. nextcloud in a container here.)

[Stéphane Graber (stgraber)]

On Sat, Nov 18, 2017 at 11:25:49PM +0300, Marat Khalili wrote:
> > The problem in your case is that you're trying to get this to work with LXC rather than LXD.
>
> Should I understand it as snapd is not currently supported under LXC? No way to reproduce manually in LXC whatever LXD does?

We don't directly support it in LXC, no.

If you want to replicate what LXD does, you're going to need to:
 - Bind-mount /dev/fuse
 - Allow /dev/fuse to be written to by the container
 - Create a new apparmor namespace and load the base apparmor profile into it (lxc-container-default in lxc's case)
 - Set lxc.aa_profile to point to that namespace
 - Add some kind of hook to cleanup that namespace when the container exits

> > I'd very strongly recommend against anyone using the configuration above with LXC as the lxc.aa_profile=unconfined [...]
>
> Obviously not the best solution security-wise, but doesn't snapd provide its own level of isolation? It shouldn't be worse than running snapd on host, for people choosing not to rollout LXD?
>
> (Sorry for being increasingly off-topic, but currently Google brings people wishing to install e.g. nextcloud in a container here.)

Setting aa_profile=unconfined allows your container, to load, unload and
modify any apparmor profile on the host.

That means it can interfere with any process running on the system,
crossing container boundary. It can prevent random system processes from
functioning by loading new profiles or altering existings ones, ...

Even if all workloads are trustworthy on your system, you then have the
problem that only a single container can run snaps as running multiple
containers with snapd or having snapd run on the container and in the
host will result in apparmor profile conflicts with global policies
being modified every time snap-confine runs in one of the containers or
host.

Stéphane

[Frode Nordahl (fnordahl)]

Following previous discussions and proposals I understand there are some issues with getting 'squashfuse' installed automatically for container workloads.

In the meantime, for improved UX, would having "snap install" command inform operator of 'squashfuse' dependency for in-container operation be something?

[Scott Moser (smoser)]

This is fixed in snapd under https://github.com/snapcore/snapd/pull/4049
It will be fixed in snapd 2.31.2

[John Lenton (chipaca)]

I removed the "snap" package bug task, as I doubt this bug affects the gene finder.

[Scott Moser (smoser)]

Bug 1756173 is set to put 2.32 into bionic and SRU to trusty, xenial, and artful.

So if you're interested in this fix, watch that bug.

[Mathieu Trudel-Lapierre (cyphermox)]

This MIR has been completed, but squashfuse is still not in main (and apparently not Depend-ed on by snapd).

None of the referenced bugs mentions squashfuse, what is the status of this?

[Michael Hudson-Doyle (mwhudson)]

My understand of the status is that snapd grew its own code for mounting snaps via fuse, which would make this MIR unnecessary.

[Marco Trevisan (Treviño) (3v1n0)]

As for host restrictions I need to use LXC too (not lxd), what I needed to get this working was just:

lxc config as:

# Mount local kernel modules dir, snap needs it for its bind-mount
lxc.mount.entry = /lib/modules lib/modules none bind,create=dir,optional 0 0

# Mounting fuse (for snap squashfs)
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional

# Mount proc and sys in rw to get udev working for snaps
lxc.mount.auto=
lxc.mount.auto=proc:rw sys:rw

Once that snapd *and* fuse packages got installed (squashfuse too?) snaps worked as expected.

[Samuele Pedroni (pedronis)]

Indeed snapd carries this on its own.

[zolika84 (zolika84)]

Issue is still the same - the issue comes out when you try to run snapd on a chromebook (linux terminal), or in WSL2.

[Ian Johnson (anonymouse67)]

@zolika84 please start a new bug with details of what you expect/fail to see from snapd on those systems, we don't explicitly support WSL2 yet, there are some forum topics around that explain how you can make it work, but snapd doesn't work OOTB in WSL2. I'm not sure what the issue is with running on Chrome OS.