Allow bitbake to create user namespace

[Hains]

Occurs since an update around March 2 Ubuntu 24.04.

Bitbake is broken due to file permission problem.

Traceback (most recent call last):
  File "/home/hains/openpli-oe-core/bitbake/bin/bitbake-worker", line 268, in child
    bb.utils.disable_network(uid, gid)
  File "/home/hains/openpli-oe-core/bitbake/lib/bb/utils.py", line 1653, in disable_network
    with open("/proc/self/uid_map", "w") as f:
PermissionError: [Errno 1] Operation not permitted

Test code

with open("/proc/self/uid_map", "w") as f:
 f.write("%s %s 1" % (1000, 1000))

ProblemType: Bug
DistroRelease: Ubuntu 24.04
Package: dash 0.5.12-6ubuntu4
ProcVersionSignature: Ubuntu 6.8.0-11.11-generic 6.8.0-rc4
Uname: Linux 6.8.0-11-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.28.0-0ubuntu1
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Fri Mar 8 14:34:08 2024
InstallationDate: Installed on 2023-03-24 (350 days ago)
InstallationMedia: Ubuntu 22.10 "Kinetic Kudu" - Release amd64 (20221020)
SourcePackage: dash
UpgradeStatus: Upgraded to noble on 2024-01-10 (58 days ago)

[Hains (hainsvdbosch)]

History log.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Imai Tomoki (tomo832)]

This issue seems to be caused by App Armor's profile "unprivileged_userns".

dmesg has the following report.

-----
[ 2229.188009] audit: type=1400 audit(1714466038.573:555): apparmor="DENIED" operation="capable" class="cap" profile="unprivileged_userns" pid=20223 comm="texinfo-dummy-n" capability=21 capname="sys_admin"
-----

And can be temporally resolved by unloading the profile.

-----
$ sudo apparmor_parser -R /etc/apparmor.d/unprivileged_userns
-----

[oh (jigglywiggly)]

Same issue here, Imai's workaround seems to fix Yocto building for me. 24.04 is rather frustrating. They also removed libtinfo5 from the default sources and that's required for Vivado.

EDIT: Also Balena Etcher doesn't work due to apparmor. https://bugs.launchpad.net/ubuntu/+source/wike/+bug/2046844/comments/4

[Ferry Toth (ftoth)]

chatgpt in this case suggests to Create a Custom AppArmor Profile:

sudo nano /etc/apparmor.d/local/bitbake
containing:

home/**/bitbake/bin/bitbake-worker {
    # Include the basic AppArmor abstractions
    #include <abstractions/base>

    /proc/self/uid_map rw,
    capability sys_admin,
}

and reload the profile:

sudo apparmor_parser -r /etc/apparmor.d/local/bitbake

What do you think, is this a valid solution?

[Ferry Toth (ftoth)]

Forget that, it doesn't work, don't know why.

[Ferry Toth (ftoth)]

This seems to work (based on a note here: https://discourse.ubuntu.com/t/ubuntu-24-04-lts-noble-numbat-release-notes/39890):

sudo vi /etc/apparmor.d/bitbake

containing:
abi <abi/4.0>,

include <tunables/global>

/home/**/bitbake/bin/bitbake flags=(unconfined) {
        userns,
}

and reload the profile:

sudo apparmor_parser -r /etc/apparmor.d/bitbake

[Ferry Toth (ftoth)]

I believe the issue is not in dash but in apparmor lacking a profile for bitbake

[Karsten S. Opdal (karsten-s-opdal)]

Tried some the above without success, but ended up with this profile that works.

file: /etc/apparmor.d/bitbake

Content:
abi <abi/4.0>,

include <tunables/global>

/home/**/bitbake/bin/bitbake-worker flags=(unconfined) {
        userns,
}

[Changqing Li (sandy-lcq)]

The following profile works for me, using ** because user of bitbake might put the code anywhere, it is not a hardcode path, so this might match any path entry.

$ cat /etc/apparmor.d/bitbake
abi <abi/4.0>,
include <tunables/global>
profile bitbake /**/bitbake/bin/bitbake flags=(unconfined) {
        userns,
}

$ sudo apparmor_parser -r /etc/apparmor.d/bitbake

[Ross Burton (ross)]

I'll add some context and explanation. Bitbake can use user namespaces to disconnect the networking for specific worker processes, which is an important isolation feature that would be unfortunate to have to disable on ubuntu hosts.

This is a very similar situation to buildah, which already has an apparmour profile to allow userns. The catch is that bitbake is not installed as part of the system, so you can't hardcode a path like /usr/bin/bitbake.

[Alexander Kanavin (kanavin)]

Something doesn't feel right in the overall design here.

Bitbake wants to take away the ability to connect to network from its child processes (which generally makes things more secure), and is not allowed to do so by the system. Isn't this... backwards?

[Philip Milev (milev-philip)]

I get the same error when building an image even in a Docker container (crops) when ran on 24.04.

```
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/builds/yocto/shared/sources/core/bitbake/bin/bitbake-worker", line 278, in child
    bb.utils.disable_network(uid, gid)
  File "/builds/yocto/shared/sources/core/bitbake/lib/bb/utils.py", line 1696, in disable_network
    with open("/proc/self/uid_map", "w") as f:
PermissionError: [Errno 1] Operation not permitted
ERROR: Task (/builds/yocto/shared/sources/core/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_prepare_recipe_sysroot) failed with exit code '1'
```

Dmesg:
```
[30666.666710] audit: type=1400 audit(1720649742.614:261): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=63501 comm="texinfo-dummy-n" requested="userns_create" target="unprivileged_userns"
[30666.673910] audit: type=1400 audit(1720649742.619:262): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=63502 comm="texinfo-dummy-n" requested="userns_create" target="unprivileged_userns"
[30666.684075] audit: type=1400 audit(1720649742.631:263): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=63507 comm="gettext-minimal" requested="userns_create" target="unprivileged_userns"
[30666.684576] audit: type=1400 audit(1720649742.632:264): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=63503 comm="quilt-native:un" requested="userns_create" target="unprivileged_userns"
[30666.685330] audit: type=1400 audit(1720649742.633:265): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=63506 comm="quilt-native:pr" requested="userns_create" target="unprivileged_userns"
[30666.686682] audit: type=1400 audit(1720649742.634:266): apparmor="DENIED" operation="capable" class="cap" profile="unprivileged_userns" pid=63507 comm="gettext-minimal" capability=21 capname="sys_admin"
[30666.687580] audit: type=1400 audit(1720649742.635:267): apparmor="DENIED" operation="capable" class="cap" profile="unprivileged_userns" pid=63506 comm="quilt-native:pr" capability=21 capname="sys_admin"
[30666.699009] audit: type=1400 audit(1720649742.646:268): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=63508 comm="m4-native:unpac" requested="userns_create" target="unprivileged_userns"
[30666.699682] audit: type=1400 audit(1720649742.647:269): apparmor="DENIED" operation="capable" class="cap" profile="unprivileged_userns" pid=63508 comm="m4-native:unpac" capability=21 capname="sys_admin"
[30666.700401] audit: type=1400 audit(1720649742.648:270): apparmor="AUDIT" operation="userns_create" class="namespace" info="Userns create - transitioning profile" profile="unconfined" pid=63511 comm="gettext-minimal" re...

[Alexander Kanavin (kanavin)]

For reference, the specific implementation in bitbake that disables networking is here:

https://git.openembedded.org/bitbake/tree/lib/bb/utils.py?h=2.8.2#n1673

[John Johansen (jjohansen)]

It does seem that way. The problem is the design of unprivileged user namespaces, it gives unprivileged applications access to a lot of kernel surface that they usually don't have access to. This has been used to elevate kernel bugs from root exploitable to being exploitable by unprivileged users.

So when used responsibly they can be used to reduce privileges, which we want to allow. But exploit code can leverage them to gain full kernel privileges. Unfortunately its impossible to distinguish between the two cases except to go through and identify the known good users.

That leaves us in the situation where we can
1. just ignore the very real problem, that has been used by dozens of exploits
2. Disable unprivileged user namespaces completely
3. Try to selectively enable/disable unprivileged user namespaces.

From a security pov allow listing (selective enablement) is the correct approach, because you can never deny list all combinations of exploits. This does unfortunately also mean that enabling unprivileged user namespaces for an application must be a privileged action, other wise an exploit only has to take an extra step to by-pass the restriction. Hence why we won't by default enable a profile for an application in a location that is run from a user writable location. A user doing said enablement local is at much lower risk than a distro doing so.

Note that Ubuntu won't be the only ones restriting unprivileged user namespaces. The ability to control them is coming to capabilities and selinux as well.

[John Johansen (jjohansen)]

@milev-philip:

containers are a difficult case. Unfortunately containers share the same kernel as the host. An application running in the container (docker image) can use unprivileged user namespaces to compromise not just the container but the host as well.

There is the ability to turn the restriction off at the host. See the 24.04 release notes https://discourse.ubuntu.com/t/ubuntu-24-04-lts-noble-numbat-release-notes/39890#unprivileged-user-namespace-restrictions

Container managers can also be modified to understand and disable the restriction for the container (lxd is doing this). But as noted above when this is done the container can be used to compromise the host, via a kernel exploit.

[John Johansen (jjohansen)]

@kanavin:

Thanks, we don't have an issue with bitbake, the issue comes down to running code out of a user writable location.

1. The location of bitbake will vary by user. Making any profile we could ship only functional for a subset of bitbak users. For the others it would require a privileged action to enable.

2. Enabling unprivileged user namespaces in a user writable location (an unprivileged action) allows an exploit to by-pass the restriction by writing that locations, as part of its setup. Doing this at a distro level advertises that location is available to all users, making it easy for exploits to be able to detect and adapt to this. When a user chooses to do it locally, it greatly reduces the risk compared to the distro level enablement.

Unfortunately atm this forces the user to understand what is going on and manually enable a profile for the application. We are working on a GUI method that users will be able to use, making this task easier. Unfortunately this also comes with the risk of users just click yes/enable without understanding the risk, but there is no way around that problem.

[Alexander Kanavin (kanavin)]

We'll probably have to wrap that code in an exception handler that would instruct the user to install an apparmor profile as root, yes.

[John Johansen (jjohansen)]

@kanavin:

Bitbake could indeed do that, it will depend on if it is considered worthwhile to carry said exception code. As I mentioned above both capabilities and SELinux are working towards limiting of unprivileged user namespaces, and the solutions needed to handle there restrictions will be different.

For Ubuntu the current plan is to SRU the apparmor GUI interface to 24.04. While I would like it to land in time for the 24.04.1 at this point I think it is unlikely as the apparmor 4.0.1 SRU took over 6 weeks.

[Alexander Kanavin (kanavin)]

What we're after is disabling the network for any child processes. It doesn't matter specifically how that gets done.

If this can be done some other way with some other facility that doesn't clash with efforts to harden the host distributions, I'm all ears.

[Ferry Toth (ftoth)]

Yes, it seems what we need is an app in a non-user writable area (i.e. as part of the system) that disables the network for a child process and an apparmor profile that allows it to do so. Then call that app from bitbake.

That sounds a lot better than asking user's 100's permissions they don't understand through a GUI (as android does), as they will just click yes else the app won't work.

[Ferry Toth (ftoth)]

Like unshare(1)?

[Ross Burton (ross)]

If I'm reading the comments in apparmor correctly (I don't have a 2404 machine myself) then unshare doesn't work out of the box either:

# This profile allows almost everything and only exists to allow
# unshare to work on a system with user namespace restrictions
# being enforced.
# unshare is allowed access to user namespaces and capabilities
# within the user namespace, but its children do not have
# capabilities, blocking unshare from being able to be used to
# arbitrarily by-pass the user namespace restrictions.
# We restrict x mapping of any code that is unknown while unshare
# has privilige within the namespace. To help ensure unshare can't
# be used to attack the kernel.
#
# disabled by default as it can break some use cases on a system that
# doesn't have or has disable user namespace restrictions for unconfined
# use aa-enforce to enable it

[Alexander Kanavin (kanavin)]

I like the idea of a trusted app. We can add support for that easily.

I agree that asking users with popups is not the right direction. It won't work in non-graphical sessions (and bitbake in particular is used that way a lot), and will only condition users to click 'Allow' without reading the text above. Same as they do for 'cookie options' or changes in privacy policies.

[John Johansen (jjohansen)]

@ross:

atm, correct unshare does Not work as it does not have a profile enabled by default. However this will be partially fixed via SRU. The SRU for apparmor 4.0.1 includes an example profile for unshare*, that will allow unshare to create user namespaces and even have capabilities within the user namespace, but any child it execs whether in the user namespace or outside of it will not have those privileges.

This will enable unshare to be used for some use cases but not all. Basically it will NOT work for the
 use case where the executed child needs privileges within the user namespace. This use case has to be privileged as other wise it allows the unprivileged user to by-pass the restriction.

* Note: the 4.0.1 SRU does not enable the unshare profile by default, as there needs to be further testing that we are not regressing current unshare users like LXD. The plan is to enable with a targeted follow-on SRU that does only does 1 thing, enable the profile by default.

[Ferry Toth (ftoth)]

So let's say the trusted app is just a shell script with it's own profile enabled. And the script calls unshare with a fixed set of CLI parameters that only allow safe operations or even just disconnect networking for its childs.

Then we decouple the unsafe operations from unshare that are disallowed for its children from the safe operations that need to be inherited.

Would that work?