core snap's configuration hangs on debian | openSUSE | mainline kernel

[Michael Hudson-Doyle]

On Debian stretch (or sid), if you install snapd and then run "snap install hello", the install of the core snap hangs at the "running configure hook" step. If you run "snap install core" then run "snap install hello", things work.

When it hangs, pstree output looks like this:

systemd,1 --system --deserialize 16
[...]
  |-snapd,6659
  | |-configure,6670 -e /snap/core/1441/meta/hooks/configure
  | | `-(snapctl,6679)
  | | |-{snapctl},6680
  | | `-{snapctl},6681
  | |-{snapd},6663
  | |-{snapd},6664
  | |-{snapd},6665
  | |-{snapd},6666
  | |-{snapd},6667
  | |-{snapd},6668
  | |-{snapd},6669
  | `-{snapd},6671

stracing the configure process just shows it sitting in an unfinished "read(3," syscall. Let me know if there is any way I can debug this. I guess the unusual thing about Debian might be that the snapd on the system is a little old (2.21). Or maybe it's something else, who knows!

[Shuduo Sang (sangshuduo)]

Can reproduced on openSUSE Tumbleweed and Leap 42.2 too.

[Zygmunt Krynicki (zyga)]

I had a look at this and it's puzzling. What is going on is that dash running the configure hook hangs on a read (presumably from a pipe) from snapctl get ... which is in a curious state. There are three processes (because golang clones) one zombie and two children. The children are sleeping and I was unable to procure a backtrace of where they were (missing symbols).

[Simon Fels (morphis)]

I've investigated this a bit and found that snapctl is hanging up because it receives a seccomp denial (we build snap-confine on OpenSuse with --enable-seccomp and --disable-apparmor).

type=SECCOMP msg=audit(1490202008.823:1229): auid=1000 uid=1000 gid=100 ses=39 pid=29303 comm="snapctl.orig" exe="/usr/bin/snapctl.orig" sig=31 arch=c000003e syscall=49 compat=0 ip=0x55d171e2f294 code=0x0

Is what I see in /var/log/audit/audit.log

syscall=49 is bind and bind is correctly specified in all seccomp profiles snapd generates:

simon@linux-82h5:/var/lib/snapd/seccomp> fgrep -R bind
profiles/snap.core.hook.configure:mbind
profiles/snap.core.hook.configure:# of socket(), bind(), connect(), etc individually.
profiles/snap.core.hook.configure:bind
profiles/snap.hello-world.evil:mbind
profiles/snap.hello-world.evil:# of socket(), bind(), connect(), etc individually.
profiles/snap.hello-world.sh:mbind
profiles/snap.hello-world.sh:# of socket(), bind(), connect(), etc individually.
profiles/snap.hello-world.hello-world:mbind
profiles/snap.hello-world.hello-world:# of socket(), bind(), connect(), etc individually.
profiles/snap.hello-world.env:mbind
profiles/snap.hello-world.env:# of socket(), bind(), connect(), etc individually.

After building snap-confine with --disable-seccomp snapctl works as it should.

So in summary something with the seccomp setup is broken on other distributions.

[Jamie Strandboge (jdstrand)]

I see problems with Ubuntu Core 16 in a VM (ie, with apparmor enabled). There the core snap is trying to access /run/snapd.socket:

Mar 7 15:08:47 snappy-16-amd64 kernel: [ 343.634033] audit: type=1400 audit(1488899327.930:9): apparmor="DENIED" operation="create" profile="snap.core.hook.configure" pid=1382 comm="snapctl" family="inet" sock_type="stream" protocol=6 requested_mask="create" denied_mask="create"
Mar 7 15:08:47 snappy-16-amd64 kernel: [ 343.634141] audit: type=1400 audit(1488899327.930:10): apparmor="DENIED" operation="create" profile="snap.core.hook.configure" pid=1382 comm="snapctl" family="inet6" sock_type="stream" protocol=6 requested_mask="create" denied_mask="create"
Mar 7 15:08:47 snappy-16-amd64 kernel: [ 343.634179] audit: type=1400 audit(1488899327.930:11): apparmor="DENIED" operation="create" profile="snap.core.hook.configure" pid=1382 comm="snapctl" family="inet6" sock_type="stream" protocol=6 requested_mask="create" denied_mask="create"
Mar 7 15:08:47 snappy-16-amd64 kernel: [ 343.649750] audit: type=1400 audit(1488899327.946:12): apparmor="DENIED" operation="open" profile="snap.core.hook.configure" name="/run/snapd.socket" pid=1382 comm="snapctl" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0

This is https://bugs.launchpad.net/snapd/+bug/1674484 and https://bugs.launchpad.net/snappy/+bug/1644573.

On OpenSuse I suspect what is happening is that because apparmor is disabled, /run/snapd.socket is not denied so then the bind() is tried and there is a seccomp denial.

I think the real questions are understanding why /run/snapd.socket is being used, and why is bind() being used. The default template allows use of /usr/bin/snapctl and /run/snapd-snap.socket, but the core snap and by the looks of it, other snaps like network-manager, wpa-supplicant are using snapctl which generates a denied access to /run/snapd.socket. Is this intentional? Is snapctl attempting to open /run/snapd.socket and if it can't it falls back to /run/snapd-snap.socket (if so, I would argue this is probably the wrong behavior since it creates a lot of confusion due to the noisy denial)? Is the configure hook working correctly anywhere?

*If* the core snap has legitimate access to /run/snapd.socket, then perhaps its configure hook should also 'plugs: [ snapd-control ]'.

All that said, it looks like core r1530 now plugs 'network-bind'. I'm not sure if that was to address this bug-- if it wasn't, then the inet apparmor denials and the bind seccomp call will all be addressed by this change.

[Michael Hudson-Doyle (mwhudson)]

I couldn't reproduce this yesterday with core snap 1512 which does have the network-bind plug. So maybe that fixed it. It was added in https://github.com/snapcore/core/commit/6de4becf8a3f7af69f665b7452a93a97a50c0547 which has the commit message 'add network-bind to core-support as it appears to be required on the "Boundary Devices Nitrogen"', so if it fixed this bug I'm pretty sure it was by accident.

[Simon Fels (morphis)]

On OpenSuse the network-bind plug of the core snap was connected too but the bind problem still appeared.

[Zygmunt Krynicki (zyga)]

Let's try snap-confine with SNAP_CONFINE_DEBUG=yes and perhaps patch it with extra logging around the established seccomp profile.

[Simon Fels (morphis)]

We've release snapd 2.23.5-1 into openSUSE now which disables seccomp.

[Michael Hudson-Doyle (mwhudson)]

core snap 1512 got unreleased so this bug came back.

Where do I look for secomp denials? There doesn't seem to be anything in syslog.

[Zygmunt Krynicki (zyga)]

It was reported that this also happens on vanilla upstream kernel.

[Zygmunt Krynicki (zyga)]

You can run the configure hook (or just snapctl) under confinement of
the configure hook snap-confine (or via snap run --shell --hook)

On Thu, Mar 23, 2017 at 8:28 PM, Zygmunt Krynicki
<email address hidden> wrote:
> It was reported that this also happens on vanilla upstream kernel.
>
> --
> You received this bug notification because you are a member of Snappy
> Developers, which is subscribed to Snappy.
> Matching subscriptions: xxx-bugs-on-snapd
> https://bugs.launchpad.net/bugs/1674193
>
> Title:
> core snap's configuration hangs on debian|openSUSE
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/snappy/+bug/1674193/+subscriptions

[Kevin Friesth (kfriesth)]

It fails the way with an upline kernel 4.10.5, config is attached

[Simon Fels (morphis)]

With snapd 2.23 now building on Fedora 25 I can reproduce the same problem without specifying --disable-seccomp for the snap-confine build.

[Lukas (jemus42)]

I'm also seeing this on Ubuntu Xenial with apparmor-enabled kernel(?) (i.e. apparmor bootscript from scaleway).

It's a fresh server with very few packages installed.

`Linux <hostname omitted> 4.8.14-apparmor-2 #1 SMP Tue Jan 10 16:12:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux`

[Oliver Grawert (ogra)]

@lukas, well, this is also a non-ubuntu kernel ... same as mainline builds i'd guess

[Jamie Strandboge (jdstrand)]

This doesn't seem to have anything to do with upstream vs Ubuntu kernels. For example, on Ubuntu 16.04 LTS classic I see:

$ cat /proc/version_signature
Ubuntu 4.4.0-66.87-generic 4.4.44

$ aa-enabled
Yes

$ snap list
Name Version Rev Developer Notes
core 16-2 1441 canonical -

# With AppArmor in strict mode
$ sudo snap run --shell --hook=configure core # NO HANG
...
$

with the following denials:
Mar 24 09:46:37 sec-xenial-amd64 kernel: [ 4103.232237] audit: type=1400 audit(1490366797.640:77): apparmor="DENIED" operation="create" profile="snap.core.hook.configure" pid=2407 comm="snapctl" family="inet" sock_type="stream" protocol=6 requested_mask="create" denied_mask="create"
Mar 24 09:46:37 sec-xenial-amd64 kernel: [ 4103.232471] audit: type=1400 audit(1490366797.640:78): apparmor="DENIED" operation="create" profile="snap.core.hook.configure" pid=2407 comm="snapctl" family="inet6" sock_type="stream" protocol=6 requested_mask="create" denied_mask="create"
Mar 24 09:46:37 sec-xenial-amd64 kernel: [ 4103.232579] audit: type=1400 audit(1490366797.640:79): apparmor="DENIED" operation="create" profile="snap.core.hook.configure" pid=2407 comm="snapctl" family="inet6" sock_type="stream" protocol=6 requested_mask="create" denied_mask="create"
Mar 24 09:46:37 sec-xenial-amd64 kernel: [ 4103.237185] audit: type=1400 audit(1490366797.644:80): apparmor="DENIED" operation="open" profile="snap.core.hook.configure" name="/run/snapd.socket" pid=2407 comm="snapctl" requested_mask="wr" denied_mask="wr" fsuid=0 ouid=0

If I add the following to the apparmor profile (or put the profile in complain mode with "sudo sed -i 's/attach_disconnected/attach_disconnected,complain'/ /var/lib/snapd/apparmor/profiles/snap.core.hook.configure"):
network inet,
network inet6,
/run/snapd.socket rw,

and reload:
$ sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.core.hook.configure

then run hook:
$ sudo snap run --shell --hook=configure core # HANG

I see:
Mar 24 09:47:56 sec-xenial-amd64 kernel: [ 4181.864114] audit: type=1326 audit(1490366876.272:82): auid=1000 uid=0 gid=0 ses=1 pid=2440 comm="snapctl" exe="/usr/bin/snapctl" sig=31 arch=c000003e syscall=49 compat=0 ip=0x563064039294 code=0x0

$ scmp_sys_resolver 49
bind

I can then add 'bind' to /var/lib/snapd/seccomp/profiles/snap.core.hook.configure and try again it runs without hanging:
$ sudo snap run --shell --hook=configure core # NO HANG
...
$

What the above clearly shows is that if apparmor policy is adjusted to not block the accesses snapctl is attempting, then snapctl hits a seccomp denial for bind. This explains why systems with apparmor disabled but seccomp enabled see this problem.

Digging into this more, if the apparmor policy is adjusted to have (and bind is removed from the seccomp policy):

#network inet,
network inet6,
#/run/snapd.socket rw,

I can reproduce the hang:
$ sudo apparmor_parser -r /var/lib/snapd/apparmor/profiles/snap.core.hook.configure && sudo snap run --shell --hook=configure core # HANG

Ie, the '/run/snapd.socket rw' rule has no effect and is a red herring. I maintain my point in comment #4 that snapctl attempting to use /run/snapd.socket is causing confusio...

[Jamie Strandboge (jdstrand)]

I went ahead and filed https://bugs.launchpad.net/snapd/+bug/1675812 for the noisy /run/snapd.socket denial.

[Jamie Strandboge (jdstrand)]

For completeness I booted into http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.4.44/ (ie, the corresponding upstream kernel for the Ubuntu release I tested on):

$ cat /proc/version
Linux version 4.4.44-040444-generic (kernel@tangerine) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #201701200532 SMP Fri Jan 20 10:33:55 UTC 2017

$ sudo snap run --shell --hook=configure core # HANG

With the following denial:
Mar 24 11:06:35 sec-xenial-amd64 kernel: [ 2078.560962] audit: type=1326 audit(1490371595.703:29): auid=1000 uid=0 gid=0 ses=2 pid=1664 comm="snapctl" exe="/usr/bin/snapctl" sig=31 arch=c000003e syscall=49 compat=0 ip=0x564e89b2a294 code=0x0

Adding 'bind' to /var/lib/snapd/seccomp/profiles/snap.core.hook.configure allows it to work:
$ sudo snap run --shell --hook=configure core # NO HANG
...
$

Note: I didn't have to fiddle with apparmor for this because the upstream kernel does not have the network compat patches, therefore apparmor doesn't mediate the problematic 'network inet6' rule described in comment #16, so the access is allowed and snapctl proceeds to use bind().

[Lukas (jemus42)]

So… in the meantime, is there a fix for users like me who just set up an Ubuntu VPS to use a specific snap?

Specifically, how exactly would I modify /var/lib/snapd/seccomp/profiles/snap.core.hook.configure, or should I, as a person how is really new to snap, just leave that alone altogether and just wait for an updated release?

[Simon Fels (morphis)]

This will be fixed for everyone with https://github.com/snapcore/snapd/pull/3101 which will be part of 2.23.6

[Michael Hudson-Doyle (mwhudson)]

Just to confirm, I won't need to patch the version of the snapd package in Debian to get that fix, just wait for a version of the core snap with 2.23.6+ in it?

[Simon Fels (morphis)]

@Michael: That is my guess but would good if we can check this quickly. Do you have a Debian system up and running you can quickly check on if there are any problems with updating to a newer core snap from candidate/beta/edge?

[Michael Hudson-Doyle (mwhudson)]

"snap install --edge core" works fine in my VM, but then "snap install
core" has always worked for me, it's only implicit installations of the
core snap that hang.

On 31 March 2017 at 18:18, Simon Fels <email address hidden> wrote:

> @Michael: That is my guess but would good if we can check this quickly.
> Do you have a Debian system up and running you can quickly check on if
> there are any problems with updating to a newer core snap from
> candidate/beta/edge?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1674193
>
> Title:
> core snap's configuration hangs on debian | openSUSE | mainline kernel
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/snappy/+bug/1674193/+subscriptions
>

[Simon Fels (morphis)]

Confirmed that debian is affected by this change too.

[Simon Fels (morphis)]

Using a core snap from beta/candidate (snapd 2.23.6) fixes the problem on debian.

[Petr (benapetr)]

root@dev1:/home/petanb# snap install --edge core
error: cannot perform the following tasks:
- Setup snap "core" (2422) security profiles (cannot setup seccomp for snap "core": fork/exec /usr/lib/snapd/snap-seccomp: no such file or directory)
- Setup snap "core" (2422) security profiles (fork/exec /usr/lib/snapd/snap-seccomp: no such file or directory)

Still doesn't work in debian

[Chris Wall-Grimes (cwgardua)]

Additional confirm on Debian 9:

$ sudo snap install --edge core
error: cannot perform the following tasks:
- Setup snap "core" (2488) security profiles (cannot setup seccomp for snap "core": fork/exec /usr/lib/snapd/snap-seccomp: no such file or directory)
- Setup snap "core" (2488) security profiles (fork/exec /usr/lib/snapd/snap-seccomp: no such file or directory)

[Michael Vogt (mvo)]

I think the error you are seeing is caused by the following:
- snap install/refresh core
- new core gets installed, re-execs into 2.26.{9,14}
- install/refresh fails (most likely in the configure hook)
- the install/refresh in undone
- *however* the snapd is not restarted into the old version
- next time snapd runs its the 2.26.x version that assumes things like the snap-seccomp compiler are available which are not anymore

[Zygmunt Krynicki (zyga)]

Note that this is *still* occurring if you install core and any snap at the same time. For example "snap install hello" on a system with no other snaps installed will still reproduce this issue.

[Chris Overcash (covercash2)]

i worked around this issue by purging and reinstalling snapd, then running `snap install --edge core` before installing any snaps.

[Zygmunt Krynicki (zyga)]

I just tested this on Debian sid on with 2.27.2-2 and the bug is now fixed, installing a core snap alongside a random snap (hello snap in my case) works correctly and no longer hangs.

[Petr (benapetr)]

sorry, but I just installed a fresh new clean debian 9.1, installed snap:

sudo apt-get install snapd

then when I tried sudo snap install core I get this very error. So I don't think anything is fixed.

[SaSHa SHokovic (sasha-shokovic)]

Had the same on debian 9.1 but solution below somehow fixed the issue, differences from the first time install are: 1.apt instead of apt-get 2.first of all install core and then apps

#apt purge snapd
#apt install snapd
#snap install --edge core
#snap install YourFavoriteApp

[Ricardo N Feliciano (felicianotech)]

Bumping this since it's been over a year.

A fix still hasn't been published for Debian.

A fresh install of Debian 9.5 still exibits this issue. `snap install core` has to be manually run before installing other snaps.

[Zygmunt Krynicki (zyga)]

The Debian testing package has been updated to 2.37.4 as seen on https://tracker.debian.org/pkg/snapd
The version in Debian stable is 2.21-2 and we have no plans to update it at this time.