cannot change mount namespace

[Andreas Schultz]

Multiple snaps are either broken or "only" display permission denied messages.

slack snap is not starting at all with:

> update.go:85: cannot change mount namespace according to change mount (/run/user/1000/doc/by-app/snap.slack /run/user/1000/doc none bind,rw,x-snapd.ignore-missing 0 0): cannot inspect "/run/user/1000/doc": lstat /run/user/1000/doc: permission denied

firefox snap does start, but also logs errors:

update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/doc /usr/share/doc none bind,ro 0 0): cannot inspect "/var/lib/snapd/hostfs/usr/share/doc": lstat /var/lib/snapd/hostfs/usr/share/doc: permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/fonts /usr/share/fonts none bind,ro 0 0): cannot inspect "/var/lib/snapd/hostfs/usr/share/fonts": lstat /var/lib/snapd/hostfs/usr/share/fonts: permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/local/share/fonts /usr/local/share/fonts none bind,ro 0 0): cannot inspect "/usr/local/share/fonts": lstat /usr/local/share/fonts: permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/cups/doc-root /usr/share/cups/doc-root none bind,ro 0 0): cannot create directory "/usr/share/cups/doc-root": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gimp/2.0/help /usr/share/gimp/2.0/help none bind,ro 0 0): cannot create directory "/usr/share/gimp/2.0": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gtk-doc /usr/share/gtk-doc none bind,ro 0 0): cannot inspect "/var/lib/snapd/hostfs/usr/share/gtk-doc": lstat /var/lib/snapd/hostfs/usr/share/gtk-doc: permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/libreoffice/help /usr/share/libreoffice/help none bind,ro 0 0): cannot create directory "/usr/share/libreoffice/help": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/xubuntu-docs /usr/share/xubuntu-docs none bind,ro 0 0): cannot inspect "/var/lib/snapd/hostfs/usr/share/xubuntu-docs": lstat /var/lib/snapd/hostfs/usr/share/xubuntu-docs: permission denied
update.go:85: cannot change mount namespace according to change mount (/run/user/1000/doc/by-app/snap.firefox /run/user/1000/doc none bind,rw,x-snapd.ignore-missing 0 0): cannot inspect "/run/user/1000/doc": lstat /run/user/1000/doc: permission denied

ProblemType: Bug
DistroRelease: Ubuntu 22.10
Package: snap (not installed)
ProcVersionSignature: Ubuntu 5.19.0-19.19-generic 5.19.7
Uname: Linux 5.19.0-19-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.23.0-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: XFCE
Date: Tue Oct 4 17:29:01 2022
InstallationDate: Installed on 2017-09-26 (1834 days ago)
InstallationMedia: Ubuntu-Server 17.10 "Artful Aardvark" - Alpha amd64 (20170924)
SourcePackage: snap
UpgradeStatus: Upgraded to kinetic on 2022-05-22 (134 days ago)

[Andreas Schultz (aschultz)]

sorry, wrong package :-(

[Alberto Mardegan (mardy)]

Hi Andreas, thanks for reporting this bug! Which snapd version are you using? How did this bug happened, was it after an upgrade or have you had this for a longer time?

Edit: a few commands that can help us understand the situation better:

======
snap version
snap list
sudo journalctl -b -u snapd
sudo journalctl -b -t audit
======

(make sure you have attempted to run at least a couple of snaps before running the last two commands)

[Matthew Darwin (bugs-mdarwin)]

Same problem. Ubuntu 22.10
$ snap version
snap 2.57.2
snapd 2.57.2
series 16
ubuntu 22.10
kernel 5.19.0-19-generic

$ snap list
Name Version Rev Tracking Publisher Notes
bare 1.0 5 latest/stable canonical✓ base
core18 20220831 2566 latest/stable canonical✓ base
core20 20220826 1623 latest/stable canonical✓ base
core22 20220902 310 latest/stable canonical✓ base
discord 0.0.20 143 latest/stable snapcrafters -
gnome-3-34-1804 0+git.3556cb3 77 latest/stable canonical✓ -
gnome-42-2204 0+git.d874303 29 latest/stable canonical✓ -
gtk-common-themes 0.1-81-g442e511 1535 latest/stable canonical✓ -
slack 4.28.182 66 latest/stable slack✓ -
snapd 2.57.2 17029 latest/stable canonical✓ snapd

[Alberto Mardegan (mardy)]

Thanks Matthew for the information. Can you please also attach the output from the other two commands mentioned above?

sudo journalctl -b -u snapd
sudo journalctl -b -t audit

[Matthias Harter (hartimcwildfly)]

Workaround is to use kernel 5.19.0-18-generic.

[Alberto Mardegan (mardy)]

In light of the latest comment, I'm adding the kernel team to this bug.

[Benni (benedikt-mas)]

Okt 10 12:58:41 mas-XPS-13-9300 systemd[1]: Starting Snap Daemon...
Okt 10 12:58:41 mas-XPS-13-9300 snapd[2825]: AppArmor status: apparmor is enabled and all features are available
Okt 10 12:58:42 mas-XPS-13-9300 snapd[2825]: AppArmor status: apparmor is enabled and all features are available
Okt 10 12:58:42 mas-XPS-13-9300 snapd[2825]: overlord.go:263: Acquiring state lock file
Okt 10 12:58:42 mas-XPS-13-9300 snapd[2825]: overlord.go:268: Acquired state lock file
Okt 10 12:58:42 mas-XPS-13-9300 snapd[2825]: daemon.go:247: started snapd/2.57.2 (series 16; classic) ubuntu/22.10 (amd64) linux/5.19.0-19-generic.
Okt 10 12:58:42 mas-XPS-13-9300 snapd[2825]: daemon.go:340: adjusting startup timeout by 2m50s (pessimistic estimate of 30s plus 5s per snap)
Okt 10 12:58:43 mas-XPS-13-9300 systemd[1]: Started Snap Daemon.
Okt 10 12:59:10 mas-XPS-13-9300 snapd[2825]: storehelpers.go:748: cannot refresh: snap has no updates available: "1password", "altair", "authy", "bare", "bitwarden", "chromium", "core", "core18", "core20", "cup>
Okt 10 12:59:12 mas-XPS-13-9300 snapd[2825]: storehelpers.go:748: cannot refresh: snap has no updates available: "1password", "altair", "authy", "bare", "bitwarden", "chromium", "core", "core18", "core20", "cup>
Okt 10 13:03:45 mas-XPS-13-9300 snapd[2825]: storehelpers.go:748: cannot refresh: snap has no updates available: "1password", "altair", "authy", "bare", "bitwarden", "chromium", "core", "core18", "core20", "cup>
Okt 10 13:03:45 mas-XPS-13-9300 snapd[2825]: autorefresh.go:540: Automatische Aktualisierung: Alle Snaps sind aktuell

[Andy Postnikov (apostnikov)]

Confirm workaround to use kernel 5.19.0-18-generic.

having exactly the same issue with slack's snap and very similar in logs

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[John Johansen (jjohansen)]

There is an apparmor userspace update in flight as well can you confirm your apparmor version by adding the output of

dpkg -l apparmor

[Richard Baka (bakarichard91)]

I am not the original author but it affects me too:
apparmor 3.0.7-1ubuntu1 amd64 user-space parser utility for AppArmor

[Alberto Mardegan (mardy)]

Thanks Alex and John for jumping in -- I did some investigation and I'm more and more persuaded that this is indeed a kernel (AppArmor bug).

The good thing is that this is 100% reproducible by just installing the latest 22.10 daily images: firefox starts with warnings, and slack does not start at all. It's also true, as first suggested by Mathias, that booting with the kernel 5.19.0-18-generic makes the problem go away.

Even with that kernel there are still error messages left, related to mkdir failing, but that is due to bug 1951210 which has been fixed with https://github.com/snapcore/snapd/pull/12127 (but the fix has not been released yet, hence we still see these errors).

The errors which turns out to be fatal (for slack) are those mentioned by Andreas as he submitted the bug:

> update.go:85: cannot change mount namespace according to change mount (/run/user/1000/doc/by-app/snap.slack /run/user/1000/doc none bind,rw,x-snapd.ignore-missing 0 0): cannot inspect "/run/user/1000/doc": lstat /run/user/1000/doc: permission denied

The failure is on "lstat", which triggers the AppArmor's getattr permission. The audit logs with the latest kernel show a flood of denials on getattr, which disappear with the previous kernel version. Could it be that the latest kernel has changed something in the way that getattr is handled?
I just found https://gitlab.com/apparmor/apparmor/-/issues/132 and I wonder if that code path has finally been enabled.

[John Johansen (jjohansen)]

So re: issue/132 that code path has always been enabled. How we have worked around it is by implicitly adding the GETATTR perm to the mapping.

Their were significant changes around permission lookup and mapping but not around how/where the check is done, so I assume it is in the mapping code though at first glance it appears to be right. I am still digging.

[Andreas Schultz (aschultz)]

Sorry for not responding earlier. The logs already posted by others mirror what I have seen.
The work-around for me is also to 5.19.0-18-generic (or rather 6.0.0-060000.202210022231 from the mainline PPA).

It feels like the problem might be related to the kernel change discussed in
https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1990064

[John Johansen (jjohansen)]

This is not related to the change in lp1990064. If it was you would see log messages similar to

apparmor="DENIED" operation="userns_create" class="namespace" info="User namespace creation restricted" error=-13 profile="unconfined" pid=21323 comm="steamwebhelper" requested="userns_create" denied="userns_create"

which we are not seeing. We are seeing a huge jump in getattr log messages and that will be related to a different change.

[John Johansen (jjohansen)]

The following patch fixes the issue for me.

[John Johansen (jjohansen)]

Note: this bug report has two parts to it.

1. Snap issue: mkdir failing covered by bug 1951210 and fixed in https://github.com/snapcore/snapd/pull/12127

2. apparmor module issue in the kernel, covered by patch in #18

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "kernel patch to apparmor" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Frode Nordahl (fnordahl)]

fwiw; I can confirm that the proposed patch also fixes issues with LXD virtual machines and block devices that was present on 5.19.0-19 and 5.19.0-20 ref bug 1992564.

[Ken VanDine (ken-vandine)]

I can confirm that 5.19.0-21-generic in kinetic-proposed does indeed fix this issue for me.

[Luis Alberto Pabón (copong)]

Same here, packages from proposed fixed it for me as well.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.19.0-21.21

---------------
linux (5.19.0-21.21) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-21.21 -proposed tracker (LP: #1992639)

  * cannot change mount namespace (LP: #1991691)
    - SAUCE: apparmor: Fix getaatr mediation causing snap failures

  * Kernel regresses openjdk on riscv64 (LP: #1992484)
    - SAUCE: Revert "riscv: mmap with PROT_WRITE but no PROT_READ is invalid"

 -- Andrea Righi <email address hidden> Wed, 12 Oct 2022 19:53:36 +0200

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-hwe-5.19/5.19.0-24.25~22.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Brian Rogers (brian-rogers)]

Is the kernel patch considered a workaround? It's not upstream, so if you install for example a kernel ppa mainline kernel, this issue comes back.

Is there supposed to be a snapd fix coming?

[Andi Chandler (bing)]

As per #26, I too am still seeing this with the Mainline PPA kernel on Ubuntu 22.10

> andi@hotblack:~$ uname -a
> Linux hotblack 6.2.0-060200rc4-generic #202301151633 SMP PREEMPT_DYNAMIC Sun Jan 15 16:40:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Don't recall it being an issue in 6.1.x

No seeing similar errors with snaps for Authy, Bitwarden and Azure Storage Explorer.

[John Johansen (jjohansen)]

The apparmor patch in this bug is not in the upstream kernel because the userns mediation code it is patching is not in the upstream kernel. If the mainline kernel ppa it is failing it will be for a different reason.

[Philip Meulengracht (the-meulengracht)]

I've been trying to reproduce this on a few different kernels now, I can not reproduce this with

The default 5.19 kernel that 22.10 comes with (official images, after updating).
The latest mainline 6.1.10 kernel release.

I have 2.58 snapd installed, and on both I can start slack.

I can however reproduce this on the latest release candidate, the 6.2-rc7 mainline kernel. The only difference being the kernel version, otherwise system being identical.

On 6.2-rc7 I get:

update.go:85: cannot change mount namespace according to change mount (/run/user/1000/doc/by-app/snap.slack /run/user/1000/doc none bind,rw,x-snapd.ignore-missing 0 0): cannot inspect "/run/user/1000/doc": lstat /run/user/1000/doc: permission denied

[John Johansen (jjohansen)]

Is there a message in the kernel ring buffer (dmesg) or kernel audit log?

[Philip Meulengracht (the-meulengracht)]

Yes, sorry.

https://paste.ubuntu.com/p/5w4f6w5CpG/

[John Johansen (jjohansen)]

we do have several apparmor denials in there but none of them are directly related to namespace creation. I have pasted then below just to make sure they don't disappear when the pastebin is reaped. It is possible that one of these denials is blocking the creation of a namespace if its calling a function that setups the namespace to fail before doing the actual namespace creation but I think this unlikely just because the paths don't line up with /run/user/

More concerning is
[ 58.869512] kauditd_printk_skb: 66 callbacks suppressed

which means we are missing some messages. Generally setting /proc/sys/kernel/printk_ratelimit to 0 will fix this and let us get most if not all of the missing messages if the test is rerun.
ie.
  echo 0 > /proc/sys/kernel/printk_ratelimit
  rerun test
  grab log

[ 58.869517] audit: type=1400 audit(1675757852.408:120): apparmor="DENIED" operation="capable" class="cap" profile="/usr/lib/snapd/snap-confine" pid=1986 comm="snap-confine" capability=12 capname="net_admin"
[ 58.869556] audit: type=1400 audit(1675757852.408:121): apparmor="DENIED" operation="capable" class="cap" profile="/usr/lib/snapd/snap-confine" pid=1986 comm="snap-confine" capability=38 capname="perfmon"
[ 58.891561] audit: type=1400 audit(1675757852.428:122): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/meta/snap.yaml" pid=2003 comm="5" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 58.893320] audit: type=1400 audit(1675757852.432:123): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/etc/apparmor.d/cache/" pid=2003 comm="5" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 58.923054] audit: type=1400 audit(1675757852.460:124): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/usr/local/share/fonts/" pid=2003 comm="5" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 58.923069] audit: type=1400 audit(1675757852.460:125): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/usr/local/share/" pid=2003 comm="5" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 58.925563] audit: type=1400 audit(1675757852.464:126): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/var/lib/snapd/hostfs/usr/share/fonts/" pid=2003 comm="5" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 58.972193] audit: type=1400 audit(1675757852.508:127): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/usr/local/share/fonts/" pid=2003 comm="5" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[ 59.020734] audit: type=1400 audit(1675757852.561:128): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/meta/snap.yaml" pid=2009 comm="5" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 59.021624] audit: type=1400 audit(1675757852.561:129): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/etc/apparmor.d/cache/" pid=2009 comm="5" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0

[Philip Meulengracht (the-meulengracht)]

I reran the test with printk_ratelimit set to 0

https://paste.ubuntu.com/p/cSWg8vJHjB/

It seems there are denials related to the /run/user after changing the ratelimit

[ 414.009909] audit: type=1400 audit(1675760471.797:304): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/run/user/1000/doc/" pid=3064 comm="5" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 414.009917] audit: type=1400 audit(1675760471.797:305): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/run/user/1000/" pid=3064 comm="5" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[ 414.009921] audit: type=1400 audit(1675760471.797:306): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/run/user/" pid=3064 comm="5" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[ 414.009971] audit: type=1400 audit(1675760471.797:307): apparmor="DENIED" operation="getattr" class="file" profile="snap-update-ns.slack" name="/run/user/1000/doc/" pid=3064 comm="5" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000

[John Johansen (jjohansen)]

So yes those look to be the culprit.

To snap-update.ns.slack profile you will need to add the rule

  r @{run}/user/@{uid}/doc/,

you can do this to the generated profile (it will get thrown away when it gets regenerated but should be sufficient to test). The profiles are stored in

  /var/lib/snapd/apparmor/profiles/

and then you can reload it with

  sudo apparmor_parser -r /path/to/profile/file

and rerun the test

[Philip Meulengracht (the-meulengracht)]

Hi John!

After adding the missing rule for /run/user/1000/doc/, those namespace issues are now gone. However slack still fails to start, with the following dmesg output:

https://paste.ubuntu.com/p/bbcWZG6qQP/

I just want to hear your opinion whether you believe this is related to missing snapd rules (which seems weird to me looking at the sudden sheer number, but I'm probably ignorant here), or this could be an issue another place?

[John Johansen (jjohansen)]

Philip so possibly snapd will need to add some new rules. This isn't a case of missing on older kernels but the new kernel requiring something more/new. I need to investigate the why more. There are three potential options I see

1. this is a regression in apparmor, around the handling of getattr. This is possible as there were changes in how permissions where handled. With that said apparmor does have regression tests around getattr that are passing so if this is the case that would indicate something is wrong in the tests.

2. The kernel could have added a new check, that is being surfaced by apparmor. This would mean adding new snapd rules.

3. Userspace libwrappers have some checks conditional on some kernel feature and the new kernel triggers this check leading to the new permission request.

[Philip Meulengracht (the-meulengracht)]

Hi John,

Thank you for sharing your thoughts on this. I'll try to look into experimenting with adding getattr in the seccomp profiles and investigating the paths it accesses. I'll share if I figure something out as well.

[Philip Meulengracht (the-meulengracht)]

Hi again John,

I managed to fix most of the denials now, and slack successfully starts up (still quite a few denies, but most can be explained). Took quite a few new rules. Thank you for your help and insight on this.

I'll post updates as soon as I have them. I need to find the proper interfaces for the new rules first.

[John Johansen (jjohansen)]

This is popping up more and looks to be a regression in apparmor. I don't have a fix yet

[John Johansen (jjohansen)]

The fix for the getattr issue in comment #26-#39 has now landed in upstream 6.2 and be part of the final release.

[Laurent Bonnaud (laurent-bonnaud)]

The fix a now upstream according to last comment.
Thanks!

[octogone (octogone-dev)]

Hello,
This information, might be usefull. I've encounter the same problem with a snap and found the problem. It's quite odd.
It s linked to AppArmor, it secure the snaps.

In my Home directory in Linux I've replaced the default folder (images, music, videos, etc... ) with symlink with the same name pointing to other drives. Looks like that was causing security issue with AppArmor, I've removed the symlink and put back the folders, rebooted and the snap is now launching now. You can apply that solution to other snaps.

[John Johansen (jjohansen)]

AppArmor does mediation post symlink resolution. Using symlinks to move a file or directories location means the profile for the application needs to be updated. That is why you see the failure when using symlinks to move those folders, those applications have not been give access to the location you have moved the files to.