Per-process user controllable Apport socket file

Bug #1839420 reported by Alex Murray
262
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Apport
Fix Released
High
Unassigned
apport (Ubuntu)
Fix Released
High
Unassigned

Bug Description

Author: Sander Bos, <https://www.sbosnet.nl/>

Date: 2019-07-30

As defined in data/apport, when Apport thinks a crash
originated in a container it will forward the crash handling to a
/proc/<pid>/root/run/apport.socket file, using /proc/ information from
the crashed process:

    424 if not is_same_ns(host_pid, "pid") and not is_same_ns(host_pid, "mnt"):
    425 # If the crash came from a container, don't attempt to handle
    426 # locally as that would just result in wrong system information.
    427
    428 # Instead, attempt to find apport inside the container and
    429 # forward the process information there.
    ...
    436 try:
    437 sock.connect('/proc/%d/root/run/apport.socket' % host_pid)

Normally, a user can't change the root directory of a process since the
chroot(2) system call can only be used by root / privileged processes.
This also means only processes set up by the super user may have a
different root directory, e.g., legitimate LXC containers, and that root
can trust the value of the root directory.

However, a user can create a user namespace using unshare(2), define
itself root in it, and then in fact be able to use chroot(2). Then,
/proc/<pid>/root/ for that process can lead anywhere. Thus, the root
directory should not be trusted by Apport, since it is user controllable,
meaning the /root/run/apport.socket path can't be trusted either to
be in fact an actual, legitimate Apport socket file within an actual,
legitimate container.

The same applies to the /run/ directory within the root directory of the
process, even when not using chroot(2): the user can define a /run mount
point within a user namespace when combined with unsharing the mount
namespace (actually equal to what is the case with an actual container
OS), and Apport on the host OS would access that in-namespace mount
point via /proc/<namespace_pid>/root/run/.

Thus, both /proc/<pid>/root/ and /proc/<pid>/root/run/ are
user-controllable, and neither can be trusted by Apport.

Example of manipulating the root directory via a user namespace:

   user@ubuntu$ chroot /bin/ ./busybox sleep 100 # normal situation, chroot(2) not permitted
   chroot: cannot change root directory to '/bin/': Operation not permitted

   user@ubuntu$ unshare -Ufmpr chroot /bin/ ./busybox sleep 100 # this works

   root@ubuntu# readlink /proc/$(pgrep busybox)/root
   /bin

Example of manipulating /run/ via a user namespace (the two command
lines are started simultaneously):

   user@ubuntu$ echo 'sleep 5; mount -t tmpfs tmpfs /run; touch /run/apport.socket; sleep 5' | unshare -Ufmpr sh

   root@ubuntu# sleep 2; ls /proc/$(pgrep unshare)/root/run/apport.socket; sleep 5; ls /proc/$(pgrep unshare)/root/run/apport.socket
   ls: cannot access '/proc/6118/root/run/apport.socket': No such file or directory
   /proc/6118/root/run/apport.socket

Thus, per-process, the user controls the socket file (via two separate
path locations, as shown above).

This can for example be abused to "catch" a core dump of a "tainted"
process: for such process a "core" core dump file is normally not
written, and the crash report file in /var/crash/ is created as root.
This makes a user unable to read such core dump, which is intended for
security since it might contain privileged contents. However, using
the methods above the apport.socket file is user controllable and may
for example be an (altered) Apport systemd socket file registered to a
"systemd --user" user initiated systemd process, so that the host Apport
instance can still communicate properly with the socket. Such altered
and user controlled apport.socket file may then for example save the
core dump in a file owned by the user effectively making a tainted,
"privileged" core dump user readable to the user (instead of just to
root) and thus defeating the intention of "fs.suid_dumpable=2" dumping a
"tainted" core dump non-readable to the user, as root. Due to things
happening in a root user namespace this might be difficult to exploit
for a setuid process, but it it easily doable for non-readable binaries,
as those also fall under the category of "tainted" core dumps.

As a different exploit example, the /proc/<pid>/root/run/apport.socket
file may be created as a symbolic link pointing to an arbitrary file
(which could even point to a destination outside the unshared namespace),
for example an actual socket file, enabling other damage / exploitation
scenarios. Apport in this case will follow such symbolic link, and
communicate with the destination (socket) file. This can be abused
into leading to several different consequences. (Side note: even
though /proc/ is used which enables the kernel to "see" the unshared
mount namespace, which seems to be intended behavior, the kernel still
considers the destination of the link relative to its own namespace,
not to the namespace's mount namespace; if this is not intended behavior
but the kernel should see the destination of the link relative to the
mount namespace instead, for example by using the root directory of the
mount namespaces's process, then this might actually be a kernel bug.)

One abuse consequence of the above symlink attack scenario is that a
user is able to both start specific new as well as influence specific
already running specific processes, including root processes and actual
container Apport processes. This on itself could be abused to run as many
processes as root as wished for possibly issuing a system DoS (e.g., due
to "RLIMIT_NPROC" not applying to root), or potentially DoS or shutdown
the sytem via files like or similar in nature to /proc/sysrq-trigger.
As a side effect, this can also make the dumping procedure of the crashed
process never-ending, e.g., in case the process started by the receiving
socket keeps running indefinitely (maliously intended or not), or is a
"normal" persistent system process.

As an example of starting a process as root, the following will start
the LXD service (at least in case it was not started already) creating
an LXD network bridge, populating /var/lib/lxd/, et cetera:

   user@ubuntu$ echo 'mount -t tmpfs tmpfs /run; ln -s /var/lib/lxd/unix.socket /run/apport.socket; timeout -s 11 1 sleep 100' | unshare -Ufmpr sh

As another example, linking to /run/lvm/lvmpolld.socket will run
lvmpolld(8) (as can be seen for example by the appearance of
"Started LVM2 poll daemon." in /var/log/syslog).

Starting processes as a different user, specifically root, is already
harmful on itself; however, actually influnencing (newly created or
already running) (root) processes may lead to more advanced and severe
exploitation methods.

In the above case of LXD for example, interacting with the LXD socket
in its actual API protocol by embedding legitimate LXD commands into
the core dump contents (being arbitrary data defined by the attacker)
and / or ancillary / ucred data (which can partially be defined by the
attacker), all which are sent over the socket by the sending Apport
instance to the destination socket, might be possible. This could lead
to arbitrary commands being issues to the LXD service including creating,
starting, and stopping containers, or even stopping the LXD service as
a whole. The same aspect of sending commands to processes holds true for
other socket files in case of linking to them, e.g., snapd socket files,
which may then also start root processes and / or be able to set up more
complex communication over such socket.

The most notable, and certainly potentially extremely harmful,
example case of communicating with (root) processes is symlinking to
an _actual_ container Apport socket file. This will start an Apport
process (as root) in said container, and communicate the socket data
(which is already "Apport-valid" data) to the receiving Apport instance.

More interestingly, the above can be done in a host-to-container,
container-to-itself, and container-to-arbitrary-other-container sense,
leading to for example host-to-container and cross-container dumping.

Exploit scenarios of the above include container escaping, (root)
privilege escalation in (or: into) a container OS, and DoS via system
resource exhaustion or storage resource exhaustion on a container OS.

Note that starting an Apport instance on a (different) container OS via
the above Apport socket-as-asymlink scenario, as well as the above LXD
manipulation scenario, can be considered remote exploitation scenarios:
container OSes (in most cases) are in fact (virtually) distinct systems,
i.e., "remote" to the system on which the attacker operates.

Changed in apport (Ubuntu):
importance: Undecided → High
Changed in apport:
importance: Undecided → High
Revision history for this message
Stéphane Graber (stgraber) wrote :

Sorry for the delay, I'm on vacation this week and so have very limited time to catch up on bug reports.

For the symlink issue, we should have apport do:
 - chdir(/proc/PID/cwd)
 - chroot(/proc/PID/root)

This will avoid any potential symlink attacks by preventing leaving the root of the container.

The other part of the issue is a tad trickier. Thinking about it, the real issue is if you have a world-executable setuid binary as that would then allow dumping the process. If the binary is owned by a user mapped in the user namespace, then it doesn't matter as you don't actually need its setuid bit and can just trace it whichever way you want.

My solution for this issue would be to check whether the executable which spawned the crashed process is owned by a uid/gid that's part of the container's map. If it is, then there is no security concern, if it isn't, then we shouldn't forward.

So to recap, the changes to apport would be:
 - Check that uid of TASK_EXE is in TASK_SUBUID, if not, log and exit
 - Check that gid of TASK_EXE is in TASK_SUBGID, if not, log and exit
 - chdir(TASK_CWD)
 - chroot(TASK_ROOT)
 - Attempt connect, if not preset/broken symlink, log and exit

It's worth noting that we are specifically NOT attaching to the namespaces of the process here as we don't want to make the host apport visible inside the container as this would pose a further security risk.

Is there an angle of this that I missed which the above changes wouldn't protect against?

If you think those changes would take care of this, I can write a patch for apport tonight, do basic regression testing for the cases that I care about and then let you do more review and landing of this.

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Thanks Stephan, can you describe a bit more how the TASK_SUBUID and TASK_SUBGID checks work?

Thanks

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Sigh, sorry for misspelling your name, Stéphane. I'll stick to copy-paste from now on. :)

Revision history for this message
Seth Arnold (seth-arnold) wrote :

CVE-2019-11483 for the userns socket misuse. I'm not entirely sure which bit is incorrect, but we can use this number to describe it.

Thanks

Revision history for this message
Stéphane Graber (stgraber) wrote :

/proc/PID/uid_map and /proc/PID/gid_map would both be parsed.

They are made of mappings that look like:
 - container id
 - host id
 - count

By parsing each one, we can effectively build a list of host id ranges that are mapped in the container. The fact that it's mapped into the container means that uid 0 in that container has control over the id and it's therefore fine to forward a crash for it.

We'd then check that the task's executable is owned by both uid and gid in those ranges and only forward in such cases.

So now if you're a random user on the system, you can't just unshare a user namespace and map your own uid/gid to 0 in it, followed by setting up a unix socket and executing a setuid binary.
When that setuid binary crashes, apport will notice that the crashed task's uid/gid map only included a single id, the user's own and that the binary which crashed isn't owned by that id, therefore skipping forwarding.

Revision history for this message
Stéphane Graber (stgraber) wrote :

If nobody has an issue with the above plan, I'll try and prepare a patch tonight.

Revision history for this message
Stéphane Graber (stgraber) wrote :

So obviously this didn't work out :)
I'll keep this issue open throughout the rest of my vacation so hopefully I can still put something together or worst case scenario, I will on the 3rd when I'm back to work.

Revision history for this message
Stéphane Graber (stgraber) wrote :
Revision history for this message
Stéphane Graber (stgraber) wrote :

The patch above was tested with:
 - Normal crash on host (processed as normal host binary crash)
 - Crash in a proper container without apport forwarding (dropped due to lack of forwarder)
 - Crash in a proper container with apport forwarding (processed by container apport)
 - Crash in a user namespace (processed as normal host binary crash)
 - Crash in a user namespace with a mount namespace and apport forwarding (rejected due to lack of pid namespace)
 - Crash in a user namespace with a pid namespace and a mount namespace and apport forwarding (rejected by new uid/gid check)

I didn't try the symlink side of the report as it's a much more obvious fix and as socket connections have worked with this patch, the mitigation must be working too.

Note that to prevent accidents due to the chroot() call, I've added a WARNING stating that we should not import anything past that point AND I've added logic to disable python module import entirely too.

Changed in apport (Ubuntu):
assignee: nobody → Canonical Security Team (canonical-security)
Tyler Hicks (tyhicks)
Changed in apport (Ubuntu):
assignee: Canonical Security Team (canonical-security) → Ubuntu Security Team (ubuntu-security)
Revision history for this message
Alex Murray (alexmurray) wrote :

Thanks for the detailed patch Stéphane - from a security point of view I wonder if there is a possibility to race on the process ID like in #1839413 - since this does a lot of operations on /proc/$PID/xxx at various times so if another process claims $PID could this cause issues? Can you please comment?

Changed in apport (Ubuntu):
assignee: Ubuntu Security Team (ubuntu-security) → Stéphane Graber (stgraber)
Revision history for this message
Stéphane Graber (stgraber) wrote : Re: [Bug 1839420] Re: Per-process user controllable Apport socket file

The crashed process keeps existing until the core dump handler (apport)
exits, so there's no risk of the pid getting recycled.

Stéphane

On Sun., Sep. 29, 2019, 4:50 p.m. Alex Murray, <email address hidden>
wrote:

> Thanks for the detailed patch Stéphane - from a security point of view I
> wonder if there is a possibility to race on the process ID like in
> #1839413 - since this does a lot of operations on /proc/$PID/xxx at
> various times so if another process claims $PID could this cause issues?
> Can you please comment?
>
> ** Changed in: apport (Ubuntu)
> Assignee: Ubuntu Security Team (ubuntu-security) => Stéphane Graber
> (stgraber)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1839420
>
> Title:
> Per-process user controllable Apport socket file
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/apport/+bug/1839420/+subscriptions
>
> Launchpad-Notification-Type: bug
> Launchpad-Bug: product=apport; status=New; importance=High; assignee=None;
> Launchpad-Bug: distribution=ubuntu; sourcepackage=apport; component=main;
> status=New; importance=High; <email address hidden>;
> Launchpad-Bug-Information-Type: Private Security
> Launchpad-Bug-Private: yes
> Launchpad-Bug-Security-Vulnerability: yes
> Launchpad-Bug-Commenters: alexmurray seth-arnold stgraber
> Launchpad-Bug-Reporter: Alex Murray (alexmurray)
> Launchpad-Bug-Modifier: Alex Murray (alexmurray)
> Launchpad-Message-Rationale: Subscriber
> Launchpad-Message-For: stgraber
>

Revision history for this message
Alex Murray (alexmurray) wrote :

Great - thanks again Stéphane - this LGTM then :)

Changed in apport (Ubuntu):
assignee: Stéphane Graber (stgraber) → nobody
Revision history for this message
Stéphane Graber (stgraber) wrote :

# Normal crash on host
Impact: regression test

stgraber@castiana:~$ sleep 1m &
[1] 15516
stgraber@castiana:~$ kill -SIGSEGV $!

Then make sure the crash was detected in /var/log/apport.log and shows up in /var/crash

# Crash in a proper container without apport
Impact: regression test

lxc launch ubuntu:18.04 c1
lxc exec c1 bash
    systemctl stop apport-forward.socket
    rm /run/apport.socket
    sleep 1m &
    kill -SIGSEGV $!

Confirm that /var/log/apport.log shows "crashed in a container without apport support"

# Crash in a proper container with apport
Impact: regression test

lxc launch ubuntu:18.04 c2
lxc exec c1 bash
    sleep 1m &
    kill -SIGSEGV $!

Confirm that /var/log/apport.log didn't log anything on the host and that /var/log/apport.log in the container shows a crash and there's a matching /var/crash file in the container

# Crash in a user namespace
Impact: regression test

unshare -U -r -f
    sleep 1m &
    kill -SIGSEGV $!

Confirm that this is processed as a normal crash in /var/log/apport.log and matching crash file

# Crash in a user namespace with a mount namespace and apport forwarding
Impact: regression test

unshare -U -r -m -f
    mount -t tmpfs tmpfs /run
    nc -l -U /run/apport.socket &
    NCPID=$!
    sleep 1m &
    kill -SIGSEGV $!
    kill $NCPID

Confirm that "/var/log/apport.log" shows "crashed in a separate mount namespace, ignoring"

# Crash in a user namespace with a pid namespace and a mount namespace and apport forwarding
Impact: security fix

unshare -U -r -m -p -f
    mount -t tmpfs tmpfs /run
    nc -l -U /run/apport.socket &
    NCPID=$!
    sleep 1m &
    kill -SIGSEGV $!
    kill $NCPID

Confirm that "/var/log/apport.log" shows "crashed in a container with no access to the binary"

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package apport - 2.20.1-0ubuntu2.20

---------------
apport (2.20.1-0ubuntu2.20) xenial-security; urgency=medium

  * SECURITY UPDATE: apport reads arbitrary files if ~/.config/apport/settings
    is a symlink (LP: #1830862)
    - apport/fileutils.py: drop permissions before reading user settings file.
    - CVE-2019-11481
  * SECURITY UPDATE: TOCTTOU race conditions and following symbolic
    links when creating a core file (LP: #1839413)
    - data/apport: use file descriptor to reference to cwd instead
      of strings.
    - CVE-2019-11482
  * SECURITY UPDATE: fully user controllable lock file due to lock file
    being located in world-writable directory (LP: #1839415)
    - data/apport: create and use lock file from /var/lock/apport.
    - CVE-2019-11485
  * SECURITY UPDATE: per-process user controllable Apport socket file
    (LP: #1839420)
    - data/apport: forward crashes only under a valid uid and gid,
      thanks Stéphane Graber for the patch.
    - CVE-2019-11483
  * SECURITY UPDATE: PID recycling enables an unprivileged user to
    generate and read a crash report for a privileged process (LP: #1839795)
    - data/apport: drop permissions before adding proc info (special thanks
      to Kevin Backhouse for the patch)
    - data/apport, apport/report.py, apport/ui.py: only access or open
      /proc/[pid] through a file descriptor for that directory.
    - CVE-2019-15790

 -- Tiago Stürmer Daitx <email address hidden> Tue, 29 Oct 2019 05:23:08 +0000

Changed in apport (Ubuntu):
status: New → Fix Released
Alex Murray (alexmurray)
information type: Private Security → Public Security
Alex Murray (alexmurray)
description: updated
tags: added: id-5d9e45ccd0f15c2eef59e1b0
tags: added: id-5db7d7dd9955e4200bf58f02
Benjamin Drung (bdrung)
Changed in apport:
milestone: none → 2.21.0
status: New → Fix Released
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.