snap with layout cannot be updated multiple times

Bug #1808821 reported by Lucy Llewellyn
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
snapd
Fix Released
High
Zygmunt Krynicki

Bug Description

With the following YAML built with "snapcraft, version 3.0.1" tested against Ubuntu 18.10 (snap version below) the resultant snap can be installed twice (executing the configured app to ensure the mount namespace is set-up) but fails on the third attempt. Executing `/usr/lib/snapd/snap-discard-ns` allows two more successful subsequent attemps and then the third fails as before. This seems to be specific to non-native triplet folders in `/usr/lib` - I tested this on x86_64, so the native `/usr/lib` triplet is `x86_64-linux-gnu`. Testing with the native triplet instead of `i386-linux-gnu` does not fail however many attempts I make.

```plain
$ snap version
snap 2.36.2
snapd 2.36.2
series 16
ubuntu 18.10
kernel 4.18.0-12-generic
```

```plain
# attempt 1
$ sudo snap install layout-test_1_amd64.snap --dangerous
layout-test 1 installed
$ layout-test
[OK] bind mount works
[OK] symlink works
done.

# attempt 2
$ sudo snap install layout-test_1_amd64.snap --dangerous
layout-test 1 installed
$ layout-test
[OK] bind mount works
[OK] symlink works
done.

# attempt 3
$ sudo snap install layout-test_1_amd64.snap --dangerous
error: cannot perform the following tasks:
- Setup snap "layout-test" (unset) security profiles (cannot setup mount for snap "layout-test": cannot update mount namespace of snap "layout-test": cannot update preserved namespace of snap "layout-test": cannot update snap namespace: cannot write to "/usr/lib/i386-linux-gnu/test.txt" because it would affect the host in "/usr/lib/i386-linux-gnu")
- Setup snap "layout-test" (unset) security profiles (cannot update mount namespace of snap "layout-test": cannot update preserved namespace of snap "layout-test": cannot update snap namespace: cannot write to "/usr/lib/i386-linux-gnu/test.txt" because it would affect the host in "/usr/lib/i386-linux-gnu")
```

```yaml
name: layout-test
version: '1'
summary: layout-test
description: layout-test

grade: devel
confinement: strict
base: core18

layout:
  /usr/lib/i386-linux-gnu/test:
    bind: $SNAP
  /usr/lib/i386-linux-gnu/test.txt:
    symlink: $SNAP/test.txt

parts:
  layout-test:
    plugin: dump
    source: .
    override-pull: |
      cat > test.txt <<EOF
      This is a test file
      EOF
      cat > test.sh <<EOF
      #!/bin/bash
      if [ -f "/usr/lib/i386-linux-gnu/test/test.txt" ]; then
        echo '[OK] bind mount works'
      else
        echo '[FAIL] bind mount is broken'
      fi
      if [ -f "/usr/lib/i386-linux-gnu/test.txt" ]; then
        echo '[OK] symlink works'
      else
        echo '[FAIL] symlink is broken'
      fi
      echo 'done.'
      EOF
      chmod +x test.sh

apps:
  layout-test:
    command: test.sh
```

Zygmunt Krynicki (zyga)
Changed in snapd:
status: New → Triaged
importance: Undecided → High
Zygmunt Krynicki (zyga)
Changed in snapd:
assignee: nobody → Zygmunt Krynicki (zyga)
Revision history for this message
Zygmunt Krynicki (zyga) wrote :

I spent some time looking at this and I cannot reproduce it on 2.36.3 on openSUSE Tumbleweed. I will add a test that examines the mount changes across the layout mount profile evolution to see if we are doing something wrong there.

My gut feeling is that there is some other factor at play that is missing from the test.

Revision history for this message
Zygmunt Krynicki (zyga) wrote :

The reporter cannot reproduce this issue either. According to snap changes there was a core refresh in last night so whatever (perhaps broken) state was present in the mount profiles or mount namespace was discarded.

I am still checking if the set of mount changes executed with the pair of layouts items and the incrementing revision would do something unexpected.

Zygmunt Krynicki (zyga)
Changed in snapd:
status: Triaged → In Progress
Revision history for this message
Zygmunt Krynicki (zyga) wrote :

I'm resuming investigation, perhaps merging this issue with https://forum.snapcraft.io/t/snapd-2-37-1-refresh-broke-layouts-snap-remove-install/9803

Revision history for this message
Lucy Llewellyn (lucyllewy) wrote :

This isn't specific to the arch triplet as I surmised in my original post. I've reproduced it with a layout of:

```
layout:
  /usr/share/alsa/alsa.conf:
    bind-file: $SNAP/usr/share/alsa/alsa.conf
```

My reproduction repoistory is at https://github.com/diddlesnaps/layout-fail

If you attempt to install the resultant snap three times, executing the `layout-fail` command between each, the third attempt will fail with:

```
- Setup snap "layout-fail" (unset) security profiles (cannot setup mount for snap "layout-fail": cannot update mount namespace of snap "layout-fail": cannot update preserved namespace of snap "layout-fail": cannot update snap namespace: cannot write to "/usr/share/alsa/alsa.conf" because it would affect the host in "/usr/share/alsa")
- Setup snap "layout-fail" (unset) security profiles (cannot update mount namespace of snap "layout-fail": cannot update preserved namespace of snap "layout-fail": cannot update snap namespace: cannot write to "/usr/share/alsa/alsa.conf" because it would affect the host in "/usr/share/alsa")
```

A subsequent attempt to install without discarding the namespace fails differently with:

```
error: cannot perform the following tasks:
- Setup snap "layout-fail" (unset) security profiles (cannot setup mount for snap "layout-fail": cannot update mount namespace of snap "layout-fail": cannot update preserved namespace of snap "layout-fail": cannot update snap namespace: no such file or directory)
- Setup snap "layout-fail" (unset) security profiles (cannot update mount namespace of snap "layout-fail": cannot update preserved namespace of snap "layout-fail": cannot update snap namespace: no such file or directory)
```

If you discard the namespace then two more installations, separated by running the `layout-fail` command, will be accepted by snapd. If you do not run the `layout-fail` command between each installation you may install as many times as you like without failure.

Revision history for this message
Zygmunt Krynicki (zyga) wrote :

I've debugged the issue now. For the sake of leaving a trace:

2019/08/13 14:54:36.626895 change.go:340: DEBUG: umount "/tmp/.snap/usr/share" (error: <nil>)
2019/08/13 14:54:36.626999 trespassing.go:183: DEBUG: trespassing violated "/usr/share/alsa" while striving to "/usr/share/alsa/alsa.conf"
2019/08/13 14:54:36.627021 trespassing.go:184: DEBUG: restricted mode: true
2019/08/13 14:54:36.627029 trespassing.go:185: DEBUG: unrestricted paths: ["/tmp" "/var/snap" "/snap/test-snapd-lp-1808821"]

The problem is directly caused by snap-update-ns keeping track of allowed locations to write to. When /usr/share/ tmpfs is unmounted the corresponding permission to use it goes with it. I need to check if this is a bug in the change tracking code or change reuse code.

Revision history for this message
Zygmunt Krynicki (zyga) wrote :

I've fixed this locally. I need some extra time for unit tests but it should be going into master today.

Revision history for this message
Zygmunt Krynicki (zyga) wrote :

This is now fixed in the following pull request https://github.com/snapcore/snapd/pull/7254

Changed in snapd:
milestone: none → 2.41
Zygmunt Krynicki (zyga)
Changed in snapd:
status: In Progress → Fix Committed
Zygmunt Krynicki (zyga)
Changed in snapd:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.