attach_disconnected not sufficient for overlayfs

Bug #1408106 reported by Jamie Strandboge
40
This bug affects 7 people
Affects Status Importance Assigned to Milestone
AppArmor
Invalid
Critical
John Johansen
MAAS
Invalid
Undecided
Unassigned
apparmor (Ubuntu)
Invalid
Critical
Steve Beattie
linux (Ubuntu)
Invalid
Critical
John Johansen

Bug Description

With the following use of overlayfs, we get a disconnected path:

$ cat ./profile
#include <tunables/global>
profile foo {
  #include <abstractions/base>

  capability sys_admin,
  capability sys_chroot,
  mount,
  pivot_root,
}

$ cat ./overlay.c
#include <alloca.h>
#include <linux/sched.h>
#include <stdio.h>
#include <string.h>
#include <sys/mount.h>
#include <fcntl.h>
#include <unistd.h>

int main(int argc, char* argv[]) {
    int i = 0;
    int len = 0;
    int ret = 0;
    char* options;

    if (geteuid())
        unshare(CLONE_NEWUSER);
    unshare(CLONE_NEWNS);

    for (i = 1; i < argc; i++) {
        if (i == 1) {
            len = strlen(argv[i]) + strlen("upperdir=,lowerdir=/") + 2;
            options = alloca(len);
            ret = snprintf(options, len, "upperdir=%s,lowerdir=/", argv[i]);
        }
        else {
            len = strlen(argv[i]) + strlen("upperdir=,lowerdir=/mnt") + 2;
            options = alloca(len);
            ret = snprintf(options, len, "upperdir=%s,lowerdir=/mnt", argv[i]);
        }

        mount("overlayfs", "/mnt", "overlayfs", MS_MGC_VAL, options);
    }

    chdir("/mnt");
    pivot_root(".", ".");
    chroot(".");

    chdir("/");
    execl("/bin/bash", "/bin/bash", NULL);
}

$ sudo apparmor_parser -r ./profile && aa-exec -p foo -- ./a.out /tmp
[255]
...
Dec 12 14:31:38 localhost kernel: [57278.040216] audit: type=1400 audit(1418387498.613:712): apparmor="DENIED" operation="exec" info="Failed name lookup - disconnected path" error=-13 profile="foo" name="/bin/bash" pid=18255 comm="a.out" requested_mask="x" denied_mask="x" fsuid=1000 ouid=0

With the above, the expectation was for the denial to be /mnt/bin/bash. There are three ways forward:
1. the correct solution is to patch overlayfs to properly track the loopback, but this will take a while, may ultimately be unachievable. UPDATE: upstream is currently working on this and Ubuntu will engage with them
2. we could rely on the fact that overlayfs creates a private unshared submount, and provide a way to not mediate the path when that is present, and tagged. This would take a bit of time, and might be the preferred method over 1 longer term
3. we could extend attach_disconnected so that we can define the attach root. Eg, we can use profile foo (attach_disconnected=/mnt) {} such that '/bin/bash' maps to '/mnt/bin/bash'. UPDATE: THIS IS NOT VIABLE

description: updated
Changed in apparmor:
status: New → In Progress
importance: Undecided → Critical
assignee: nobody → John Johansen (jjohansen)
Changed in apparmor (Ubuntu):
status: In Progress → Confirmed
assignee: John Johansen (jjohansen) → Steve Beattie (sbeattie)
Changed in linux (Ubuntu):
assignee: nobody → John Johansen (jjohansen)
importance: Undecided → Critical
status: New → Confirmed
tags: added: kernel-key
summary: - allow defining the attach root for attach_disconnected
+ attach_disconnected not sufficient for overlayfs
tags: added: kernel-da-key
removed: kernel-key
description: updated
description: updated
penalvch (penalvch)
Changed in linux (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
intrigeri (intrigeri) wrote :

Hi! What kind of (realistic) timeline can we expect here? (With the move to ZFS for containers, I wonder :)

E.g. is this part of your goals for 16.10? (I mean: for the AppArmor/Ubuntu-specific parts, as I've learnt to be patient wrt. the upstreaming to Linux mainline.)

Thanks for your work on AppArmor!

Revision history for this message
LaMont Jones (lamont) wrote :

This bug causes maas testing to fail (at least the ntp test, because of overlayfs and apparmor and ntp having a profile.) See https://bugs.launchpad.net/maas/+bug/1677336

Hardware testing is a requirement for MAAS 2.2.

tags: removed: kernel-da-key
Revision history for this message
Brendan Donegan (brendan-donegan) wrote :

@lamont does this need to have a MAAS task? Are we going to address it somehow in MAAS?

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Ok, I spent quite a bit of time evaluating this and believe this bug can be closed, but other bugs open.

In looking at this I created https://code.launchpad.net/~jdstrand/+git/test-overlay (to build simply git clone, run 'snapcraft', install the snap and then run 'test-overlay' for instructions on how to test different things).

For this bug, the test code was broken and it didn't pivot_root. I'm not sure if it did pivot_root back when this was filed (I didn't check). The use of attach_disconnected is required because upperdir (man 8 mount, look for overlay) is disconnected. Once attach_disconnected is present, all file paths are mediatable:

- when using just an overlay, the paths show up where you expect them to be in the filesystem
- when using overlay plus chroot paths are mediatable but an alias rule is really needed to have worthwhile policy (otherwise you need to keep the inner-chroot policy and outer-system policy in sync). Also logged denials have the overlay mountpoint prefixed. This is consistent with how apparmor works with chroots
- when using overlay plus private mount namespace plus pivot_root, no alias rule is required and logged path denials look like the system paths (ie, the overlay mountpoint is not prefixed)

In all, closing this bug as Invalid. I'll be filing new bugs for various issues I found in my investigation.

Changed in linux (Ubuntu):
status: Triaged → Won't Fix
Changed in apparmor (Ubuntu):
status: Confirmed → Won't Fix
Changed in apparmor:
status: In Progress → Invalid
Changed in apparmor (Ubuntu):
status: Won't Fix → Invalid
Changed in linux (Ubuntu):
status: Won't Fix → Invalid
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Closing the MAAS task as it the referenced bug is marked Fix Release. If there are issues there still, please see my previous comment and look at the code in that snap-- there are viable ways to use overlayfs with chroot and an apparmor alias rule, or overlayfs with private mount, chroot and pivot_root.

Changed in maas:
status: New → Incomplete
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Actually, I marked the MAAS task as incomplete in case people want to give feedback.

Revision history for this message
Frode Nordahl (fnordahl) wrote :

This problem has surfaced again with recent MAAS Ubuntu images. One report in bug 1701297. I have information about at least two other end users hit by the problem.

Adding a workaround by setting apparmor=0 kernel parameter in MAAS 2.2 will not help users that are running previous versions.

@jdstrand Would you care to reference the other bugs you created from your investigation?

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

@Frode, I can yes, when I file them. I need to do a bit of work for simple reproducers/etc/etc to file them. I've added an item to add a comment to this bug when I do.

Revision history for this message
Andres Rodriguez (andreserl) wrote :

@Frode,

Users running 2.2 *already* have the apparmor=0 work around for *ephemeral* environments only.

For users running previous versions, we recommend you upgrade immediately, provided that 2.0 and 2.1 are out of support. If you decide not to upgrade, your options are:

1. Use a HWE kernel (such as hwe-16.04). https://bugs.launchpad.net/cloud-init/+bug/1701297/comments/21
2. Use apparmor=0 as a global/machine kernel param (this will result in the deployed machine with apparmor disabled)

Marking this invalid for MAAS provided the bug is overlayfs.

Changed in maas:
status: Incomplete → Invalid
Revision history for this message
Frode Nordahl (fnordahl) wrote :

@andreserl

There are severe security implications of doing 2) from now until all future, and unfortunately I have seen that this is being done in the wild.

I would be much more comfortable by actually finding the root cause of the issue at hand and fixing that.

This is what I am currently pursuing, so please share any information about the actual upstream kernel problem you may have.

Revision history for this message
Tyler Hicks (tyhicks) wrote :

@fnordahl Hi! Let's keep the discussion about bug 1701297 in that bug since it is focused on the change in behavior between the Xenial release kernel and the HWE kernel. That's not what this bug is about. John is investigating the change in behavior issue. Jamie's previous investigations of overlayfs/apparmor were solely focused on the Xenial release kernel so we should not pull him in at the moment as it would divert his focus from other high priority workk.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.