“Device or resource busy” error during snap refresh when using layout with variable

[glancr team]

Refreshing a snap that contains certain (valid) layouts aborts with this error:
```
error: cannot perform the following tasks:
- Setup snap "snapd-layouts-test" (unset) security profiles (cannot setup mount for snap "snapd-layouts-test": cannot update mount namespace of snap "snapd-layouts-test": cannot update preserved namespace of snap "snapd-layouts-test": cannot update snap namespace: device or resource busy)
- Setup snap "snapd-layouts-test" (unset) security profiles (cannot update mount namespace of snap "snapd-layouts-test": cannot update preserved namespace of snap "snapd-layouts-test": cannot update snap namespace: device or resource busy)
```

The layout which caused the error in my tests was this one:
```
layout:
  /usr/lib/$SNAPCRAFT_ARCH_TRIPLET/wpe-webkit-1.0:
    bind: $SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/wpe-webkit-1.0
```
The resulting snap.yaml has $SNAPCRAFT_ARCH_TRIPLET replaced by e.g. `x86_64-linux-gnu` if built on an amd64 machine. The layout itself works fine once the snap is installed/refreshed.

After discarding the namespace file with `sudo /usr/lib/snapd/snap-discard-ns <snap-name>`, refreshes run fine again.

Reference: https://forum.snapcraft.io/t/snap-does-not-start-after-reboot/5750

[Zygmunt Krynicki (zyga)]

I can no longer reproduce this on 2.42~pre1.1 using a simplified test. Perhaps the key to triggering this bug is to also keep something open (e.g. application actually using libraries from that spot). Can you please re-check with core/snapd from the edge channel to see if there is any improvement.

If you don't find any I will attempt to build a more comprehensive test that will trigger this.

[Zygmunt Krynicki (zyga)]

I've opened a draft pull request with the regression test I wrote https://github.com/snapcore/snapd/pull/7486

[glancr team (glancr)]

I refreshed the core snap to 16-2.42~pre1.1 (rev 7831) from the edge channel. Afterwards, I ran the same test as described in the forum post, but it still results in the same error.

My test snap does not contain any actual code, so I guess it's not related to application usage of the linked mount point – but my knowledge of this area is fairly limited :-)

You mentioned in your forum reply that you're testing on Ubuntu 19.10 with kernel 5.3.0-10-generic, I'm running Ubuntu 19.04 with kernel 5.0.0-29-generic. So I set up a multipass VM with 19.10 daily, refreshed core from edge channel – but still get the same error.

[Zygmunt Krynicki (zyga)]

This is now reproduced by the test case in https://github.com/snapcore/snapd/pull/7486

[Zygmunt Krynicki (zyga)]

I'm just stating for the record that this is still on my plate. I will attempt to spend some time on this issue after the next sprint. I'll update the state to Confirmed and flip back to In Progress once I have the time to fix it.

[Zygmunt Krynicki (zyga)]

This issue is fixed by the following pull request: https://github.com/snapcore/snapd/pull/7773

[Zygmunt Krynicki (zyga)]

This was fixed in master and will be available in 2.43 by default (most likely) and in 2.42.3 under a feature flag. To use it please set the following option on command line.

snap set core experimental.robust-mount-namespace-updates=true

[glancr team (glancr)]

If I understand https://github.com/snapcore/snapd/blob/release/2.44/features/features.go correctly, this has not been enabled by default in 2.43 nor in the upcoming 2.44 release. Is there a blocker, more testing required …?

Asking because wpe-webkit-mir-kiosk users regularly hit this problem; they either get an aborted snap refresh or a defunct revision which fails to start up.

[Zygmunt Krynicki (zyga)]

This is blocked by an update to the SElinux policy

[glancr team (glancr)]

Thanks for the update; seems I overlooked https://github.com/snapcore/snapd/pull/8089 when searching for this :-) Your last comment there sounds like this may land in 2.44?

[Zygmunt Krynicki (zyga)]

This was picked up by 2.45.1 which was released as a part of the snapd snap but not yet as a part of the core snap, where it is still in the candidate channel. Marking as fix released.