snapd

Issue with system-packages-doc auto connection (snap-update-ns confinement issue)

Bug #2044335 reported by Kian Parvin on 2023-11-23

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	snapd	Fix Committed	Undecided	Zygmunt Krynicki	snapd 2.62

Bug Description

Hi,

I'm looking to migrate an existing Snap to the bare base from core22 (i.e. the "build-base" is still core22 but the "base" is bare). I'm running into an issue only on the new bare base snap where the system-packages-doc interface doesn't seem to autoconnect properly until I manually disconnect and reconnect but I don't observe this problem on the core22 snap.

The snap in question is the canonical-livepatch client snap - https://snapcraft.io/canonical-livepatch

Working:
To see the Snap working properly you can do the following
`sudo snap install canonical-livepatch --channel=core22/edge`
`snap run --shell canonical-livepatch`
`stat /usr/share/doc/linux-image-$(uname -r)/changelog.Debian.gz`

We do this in order to try and determine the build date of the currently running kernel and it works fine.
This snap is built from the following snapcraft.yaml - https://github.com/canonical/livepatch-client/blob/develop/snaps/core22/snapcraft.yaml

Not working:
To observe the problem with the bare base snap run the following,
`sudo snap remove canonical-livepatch`
`sudo snap install canonical-livepatch --channel=bare/edge`
`canonical-livepatch.bash` - because this is a bare base snap we don't have the regular bash shell, instead I have added a statically built shell as a part for debugging.
`stat /usr/share/doc/linux-image-$(uname -r)/changelog.Debian.gz`
The above should fail with "No such file or directory"
`exit`
`sudo snap disconnect canonical-livepatch:system-packages-doc`
`sudo snap connect canonical-livepatch:system-packages-doc`
`canonical-livepatch.bash`
`stat /usr/share/doc/linux-image-$(uname -r)/changelog.Debian.gz`
This now works.

The snapcraft.yaml for this Snap can be found here - https://github.com/canonical/livepatch-client/blob/develop/snaps/bare/snapcraft.yaml

Sergio Schvezov (sergiusens) on 2023-11-23

affects:

snapcraft → snapd

Sergio Cazzolato (sergio-j-cazzolato) on 2023-11-24

Changed in snapd:
status:	New → Confirmed

Revision history for this message

Zygmunt Krynicki (zyga) wrote on 2024-01-15:

I've reproduced this issue as instructed. What is interesting is that the issue re-appears if one discards the snap mount namespace with:

```
sudo /usr/lib/snapd/snap-discard-ns canonical-livepatch
```

This shows the following failures during snap-update-ns called from snap-confine:

```
zyga@novigrad:/run/snapd/ns$ canonical-livepatch.bash
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/boot /boot none bind,ro 0 0): cannot create directory "/boot": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/local/share/doc /usr/local/share/doc none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/cups/doc-root /usr/share/cups/doc-root none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/doc /usr/share/doc none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gimp/2.0/help /usr/share/gimp/2.0/help none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gtk-doc /usr/share/gtk-doc none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/libreoffice/help /usr/share/libreoffice/help none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/xubuntu-docs /usr/share/xubuntu-docs none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
```

I think this works because when snap-update-ns is called from snapd, it is doing so without apparmor profile, so the changes (whatever those are) are applied. When the changes are happening from snap-confine, snap-update-ns is confined, thus hitting the problem.

I've reproduced this issue as instructed. What is interesting is that the issue re-appears if one discards the snap mount namespace with:

```
sudo /usr/lib/snapd/snap-discard-ns canonical-livepatch
```

This shows the following failures during snap-update-ns called from snap-confine:

```
zyga@novigrad:/run/snapd/ns$ canonical-livepatch.bash 
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/boot /boot none bind,ro 0 0): cannot create directory "/boot": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/local/share/doc /usr/local/share/doc none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/cups/doc-root /usr/share/cups/doc-root none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/doc /usr/share/doc none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gimp/2.0/help /usr/share/gimp/2.0/help none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gtk-doc /usr/share/gtk-doc none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/libreoffice/help /usr/share/libreoffice/help none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/xubuntu-docs /usr/share/xubuntu-docs none bind,ro 0 0): cannot create writable mimic over "/usr": permission denied
```

Zygmunt Krynicki (zyga) on 2024-01-15

Changed in snapd:
assignee:	nobody → Zygmunt Krynicki (zyga)

Revision history for this message

Zygmunt Krynicki (zyga) wrote on 2024-01-16:

Download full text (3.8 KiB)

So I've managed to get it to work, as seen below:

```
zyga@novigrad:/var/lib/snapd/apparmor/profiles$ sudo /usr/lib/snapd/snap-discard-ns canonical-livepatch
zyga@novigrad:/var/lib/snapd/apparmor/profiles$ canonical-livepatch.bash
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/local/share/doc /usr/local/share/doc none bind,ro 0 0): cannot write to "/var/lib/snapd/hostfs/usr/local/share/doc" because it would affect the host in "/var/lib/snapd"
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gimp/2.0/help /usr/share/gimp/2.0/help none bind,ro 0 0): cannot write to "/var/lib/snapd/hostfs/usr/share/gimp/2.0/help" because it would affect the host in "/var/lib/snapd"
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gtk-doc /usr/share/gtk-doc none bind,ro 0 0): cannot write to "/var/lib/snapd/hostfs/usr/share/gtk-doc" because it would affect the host in "/var/lib/snapd"
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/libreoffice/help /usr/share/libreoffice/help none bind,ro 0 0): cannot write to "/var/lib/snapd/hostfs/usr/share/libreoffice/help" because it would affect the host in "/var/lib/snapd"
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/xubuntu-docs /usr/share/xubuntu-docs none bind,ro 0 0): cannot write to "/var/lib/snapd/hostfs/usr/share/xubuntu-docs" because it would affect the host in "/var/lib/snapd"

BusyBox v1.30.1 (Ubuntu 1:1.30.1-7ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/var/lib/snapd/apparmor/profiles $ stat /usr/share/doc/linux-image-$(uname -r)/changelog.Debian.gz
File: /usr/share/doc/linux-image-6.5.0-14-generic/changelog.Debian.gz
Size: 6918 Blocks: 16 IO Block: 4096 regular file
Device: 10302h/66306d Inode: 23994178 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2023-12-10 10:17:53.000000000
Modify: 2023-11-14 13:47:15.000000000
Change: 2023-12-10 10:17:54.000000000
```

The "would affect host" messages are there bacause snap-update-ns is refusing to create directories that are missing on the host (there's no xubuntu-docs in my system, for example) but IMO this is the correct mode of operation.

There's only one problem now: the reason we didn't do this originally, is that the apparmor permissions are somewhat strong (to say the least). Having said that, snap-update-ns is also called without the sandbox, so perhaps the whole premise was too optimistic. The sandbox is really tricky with path-based rules and chroot/pivot root/bind mounts, to the extent that one could argue for inode level permissions instead.

The diff is:

So I've managed to get it to work, as seen below:

```
zyga@novigrad:/var/lib/snapd/apparmor/profiles$ sudo /usr/lib/snapd/snap-discard-ns canonical-livepatch
zyga@novigrad:/var/lib/snapd/apparmor/profiles$ canonical-livepatch.bash 
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/local/share/doc /usr/local/share/doc none bind,ro 0 0): cannot write to "/var/lib/snapd/hostfs/usr/local/share/doc" because it would affect the host in "/var/lib/snapd"
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gimp/2.0/help /usr/share/gimp/2.0/help none bind,ro 0 0): cannot write to "/var/lib/snapd/hostfs/usr/share/gimp/2.0/help" because it would affect the host in "/var/lib/snapd"
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/gtk-doc /usr/share/gtk-doc none bind,ro 0 0): cannot write to "/var/lib/snapd/hostfs/usr/share/gtk-doc" because it would affect the host in "/var/lib/snapd"
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/libreoffice/help /usr/share/libreoffice/help none bind,ro 0 0): cannot write to "/var/lib/snapd/hostfs/usr/share/libreoffice/help" because it would affect the host in "/var/lib/snapd"
update.go:85: cannot change mount namespace according to change mount (/var/lib/snapd/hostfs/usr/share/xubuntu-docs /usr/share/xubuntu-docs none bind,ro 0 0): cannot write to "/var/lib/snapd/hostfs/usr/share/xubuntu-docs" because it would affect the host in "/var/lib/snapd"

BusyBox v1.30.1 (Ubuntu 1:1.30.1-7ubuntu3) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/var/lib/snapd/apparmor/profiles $ stat /usr/share/doc/linux-image-$(uname -r)/changelog.Debian.gz
  File: /usr/share/doc/linux-image-6.5.0-14-generic/changelog.Debian.gz
  Size: 6918          Blocks: 16         IO Block: 4096   regular file
Device: 10302h/66306d    Inode: 23994178    Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2023-12-10 10:17:53.000000000
Modify: 2023-11-14 13:47:15.000000000
Change: 2023-12-10 10:17:54.000000000
```

The diff is:

zyga@novigrad:/var/lib/snapd/apparmor/profiles$ diff -u snap-update-ns.canonical-livepatch{.orig,}
--- snap-update-ns.canonical-livepatch.orig	2024-01-16 10:54:33.491167418 +0100
+++ snap-update-ns.canonical-livepatch	2024-01-16 10:58:17.115343785 +0100
@@ -430,4 +430,19 @@
   mount options=(bind,remount,ro) -> /boot/,
   umount /boot/,
 
+  # Writable mimic over / - extra permissions generalized
+  "/**" rw,
+  "/tmp/.snap/usr/**" rw,
+  mount options=(rbind, rw) "/**" -> "/tmp/.snap/**",
+  mount fstype=tmpfs options=(rw) tmpfs -> "/**",
+  mount options=(rbind, rw) "/tmp/.snap/**" -> "/**",
+  mount options=(bind, rw) "/tmp/.snap/**" -> "/**",
+  mount options=(rprivate) -> "/tmp/.snap/**",
+  umount "/tmp/.snap/**",
+  mount options=(rprivate) -> "/**",
+  umount "/**",
 }

Not terrible, not great.

I would say that if we detect that mimic over / is needed, we could emit something like that into snap-update-ns profile for the snap.

Alex, do you think this is an acceptable solution from the security point of view?

Revision history for this message

Alex Murray (alexmurray) wrote on 2024-01-19:

Zyga - I think given the nature of this issue, your proposed solution seems reasonable - snap-update-ns is trusted and the apparmor profile is there to limit possible damage if it were to somehow be compromised - but it should also not block it from accomplishing its intended functionality. So this seems like the correct solution.

Zygmunt Krynicki (zyga) on 2024-01-22

Changed in snapd:
status:	Confirmed → In Progress

Revision history for this message

Zygmunt Krynicki (zyga) wrote on 2024-01-23:

Alex, one challenge I found while implementing the solution is that we don't know ahead of time that /usr needs to be created because we don't know the contents base snap of the snap that we are configuring at the time.

The only solution I have right now is to unconditionally weaken the sandbox for s-u-n as explained above when the specific interface system-packages-doc is connected.

I'm not done looking, just posting an update of the current state.

Zygmunt Krynicki (zyga) on 2024-02-02

summary:

- Issue with system-packages-doc auto connection
+ Issue with system-packages-doc auto connection (snap-update-ns
+ confinement issue)

Revision history for this message

Zygmunt Krynicki (zyga) wrote on 2024-02-05:

I've posted https://github.com/snapcore/snapd/pull/13543 with a very scoped solution.

Revision history for this message

Kian Parvin (kian-parvin) wrote on 2024-02-05:

Huge thanks for all the investigation and work looking into this. I assume after that PR lands and a new Snapd release is rolled out then this bare base snap will begin to work "out-the-box"? And my more important question, does Snapd get updated on old Ubuntu releases? The canonical-livepatch snap is required to work on machines that are running within ESM, so that would require us to support only 16.04 and up after April rolls around. Do you know if Ubuntu 16.04 would receive this Snapd update?

Revision history for this message

Zygmunt Krynicki (zyga) wrote on 2024-02-05:

Yes, your understanding is correct. I cannot promise which snapd release carries this patch but all the versions, going all the way back to Ubuntu 14.04 would receive it as a part of snapd re-execution system.

Revision history for this message

Kian Parvin (kian-parvin) wrote on 2024-02-05:

Awesome, thanks for that info and all the work on this!

Revision history for this message

Ernest Lotter (ernestl) wrote on 2024-03-18:

This will be available in snapd 2.62

Changed in snapd:
status:	In Progress → Fix Committed

Revision history for this message

Ernest Lotter (ernestl) wrote on 2024-03-29:

#10

Test feedback on snapd 2.62 (currently in beta) would be appreciated.

Changed in snapd:
milestone:	none → 2.62

Revision history for this message

Kian Parvin (kian-parvin) wrote on 2024-04-02:

#11

I can confirm this has fixed the issue with the canonical-livepatch bare base snap. Steps I ran to test:
On SnapD 2.61:
$ sudo snap install canonical-livepatch --channel=latest/stable
$ sudo pro enable livepatch
$ sudo snap refresh canonical-livepatch --channel=bare/edge
$ sudo canonical-livepatch refresh
$ sudo canonical-livepatch status
# Here we can see the error that the snap cannot read from /usr/share/doc
# Revert back to the previous working version
$ sudo snap refresh canonical-livepatch --channel=latest/stable
$ snap refresh snapd --beta
$ sudo snap refresh canonical-livepatch --channel=bare/edge
$ sudo canonical-livepatch refresh
$ sudo canonical-livepatch status
# The error is no longer present.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.