juju-run unit root escalation vulnerability

[Ryan Beisner]

If a non-root and non-sudoer user has access to a console or otherwise has the ability remotely execute commands on unit, they can trivially become root with `juju-run <self-unit-name> 'whoami'`.

One specific example - this is the jenkins (non-privileged) user gaining root:

jenkins@juju-3ee1f0-temp1-0:~$ whoami
jenkins
jenkins@juju-3ee1f0-temp1-0:~$ sudo ls
[sudo] password for jenkins:
Sorry, try again.
[sudo] password for jenkins:
sudo: 1 incorrect password attempt
jenkins@juju-3ee1f0-temp1-0:~$ juju-run osci-master/0 "whoami"
root

This is Juju 2.1.2 stable.

[John A Meinel (jameinel)]

This isn't on arbitrary machines, but it is true on machines that were deployed with Juju.

"juju-run" runs inside of a hook context, which is inherently defined as 'run as root'.

We could potentially filter out access to juju-run via a group. The 'ubuntu' user would obviously be in that group, and charms could potentially add other users if they wanted to.

The actual goal is to allow running applications to report back information (for example juju-run status-set "blocked" "something is wrong").

We could try to limit the context of 'juju-run' to a non-root context, but then it inherently doesn't have full charm context. That would also affect people using:
 juju run --all myscript

since that would change the privileges that you have for myscript.

Its a bit hard to say "apps have access to juju-run if they want to run something in a charm context" but then say "but you don't have root access, which is what charm context is".

Needs some thought, to sort out what is really useful for an application to have access to (setting live status would be very useful), vs what vulnerabilities we raise. (apt install apache defaults to not letting the apache user get access to root.)

[Ryan Beisner (1chb1n)]

PS Many thanks to @thedac for bringing this up.

[Anastasia (anastasia-macmood)]

I've added 2.3 as a target as there will be separate solutions for 2.2 and 2.3 series. Short term solution will limit permission on a socket. Long term solution will carefully consider use cases and potentially replace global juju-run with dedicated and permission-limited commands like set-status, set-config, etc.

[Seth Arnold (seth-arnold)]

Hello; is it yet known which releases of Juju this applies to?

- 14.04's 1.25.6?
- 16.04's 2.0.2?
- 16.10's 2.0.2?
- 17.04's 2.0.02?

Are patches available yet for review?

Thanks

[John A Meinel (jameinel)]

All versions of Juju that have 'juju-run', which is probably everything
since 1.18 or so.

John
=:->

On Fri, Apr 28, 2017 at 2:06 AM, Seth Arnold <email address hidden>
wrote:

> Hello; is it yet known which releases of Juju this applies to?
>
> - 14.04's 1.25.6?
> - 16.04's 2.0.2?
> - 16.10's 2.0.2?
> - 17.04's 2.0.02?
>
> Are patches available yet for review?
>
> Thanks
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1682411
>
> Title:
> juju-run unit root escalation vulnerability
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1682411/+subscriptions
>

[Seth Arnold (seth-arnold)]

Are patches available yet for review? Thanks

[Tyler Hicks (tyhicks)]

I spoke with Anastasia today and she said that the juju team hasn't yet definitively determined the affected releases. I asked that she schedule that triage for next week and that the juju team provide minimal debdiffs for all affected releases that are in the Ubuntu archive. The Ubuntu security team can then build the security updates in the private security PPA and provide those builds to the juju team for QA. If everything looks good, the security team will publish the updates and publish an Ubuntu security notice.

[John A Meinel (jameinel)]

Attached should be a sufficient patch. I need to find all the possible versions that would be affected.

Note that the socket is automatically deleted whenever the agent restarts, so 'chmod' is only valid for the lifetime of the agent, as it won't be preserved when the agent bounces.

[John A Meinel (jameinel)]

this is the very old source package of pyjuju

[John A Meinel (jameinel)]

Looking at the links, this probably wants a 1.25 and a 2.0 and a 2.1 (+2.2) patches.

[John A Meinel (jameinel)]

I'm planning on merging whatever lands into 2.1 into develop directly from the 2.1 branch instead of landing a separate 2.2 branch.

[Joel Sing (jsing)]

FWIW this fix still leaves a race that is potentially exploitable - if you can either force the agent to restart (or know that it will restart), you just have to continuously open the socket and if you land between the net.Listen and os.Chmod calls, you still have root. This may be acceptable, but it does not fully resolve the issue.

A more correct fix is to restrict access by changing/restricting permissions on the directory containing the socket so that the socket is never accessible. The other (more complex) alternative is to open the socket manually with the correct mode.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[John A Meinel (jameinel)]

Here is an alternative that sets umask before creating the file. I would have thought that since we aren't accepting anything on the socket it would have been ok, but this should still prevent us from creating the file with the wrong mode, correct?

I wasn't able to find a trivial way to 'mksock' with the right parameters and get that passed into a net.Listener (I believe you can open them from a file descriptor, but I didn't see a mksock function).

[John A Meinel (jameinel)]

previous one was a patch against previous, this is just the patch against unmodified juju-1.25

[John A Meinel (jameinel)]

Per request, there are no PRs up for these patches. If we find that this is a valid patch, I can commit it directly to the github repository instead of going via the bot, etc.

[Andrew Wilkins (axwalk)]

umask applies to the process, so doing that may inadvertently affect other code in the same process. Is that OK? I'm not sure there's an alternative for abstract domain sockets though.

Also, what about Windows?

[Seth Arnold (seth-arnold)]

If it's possible that unknown other operations may be executing concurrently in the same process then probably a new directory should be created with restricted permissions that could be used for creating the socket, which would then be inaccessible to all processes without CAP_DAC_OVERRIDE and/or CAP_DAC_READ_SEARCH. Once the permissions are set correctly then the directory permissions could be widened.

I didn't find any reference to AF_UNIX on what appears to be the Windows socket(2) analog: https://msdn.microsoft.com/en-us/library/windows/desktop/ms740506(v=vs.85).aspx

Thanks

[John A Meinel (jameinel)]

So would creating a subdirectory, setting that to restricted permissions, then creating the socket in that directory with restricted permissions, then renaming the socket to the right directory be ok?

I really don't know what to do for the Windows socket.

[Seth Arnold (seth-arnold)]

I believe that should work:

- I don't see any caveats about renaming Unix domain sockets in rename(2)
- A quick test with socat UNIX-LISTEN:foo - paired with socat - UNIX-CONNECT:foo wasn't interrupted when renaming the socket 'foo'.

The source directory and the target directory should be on the same filesystem.

Thanks

[Nicholas Skaggs (nskaggs)]

Do you agree releasing the fix can be done by simply releasing new agents into simplestreams?

[Andrew Wilkins (axwalk)]

It seems my mention of abstract domain sockets was missed, so I'll restate in more detail. This doesn't apply to juju-run, but it does apply to jujuc (the hook tools: relation-set, relation-get, etc.). The jujuc socket is an abstract domain socket, so it has no relationship to the filesystem; abstract domain sockets aren't affected by filesystem permissions. I think you have to use SCM_CREDENTIALS to do any kind of auth on those. This is technically a different bug I guess, but it would be good to at least consider how to fix that at the same time.

Seth: on Windows we use named pipes. We'll need to look at whether it is similarly affected, but that shouldn't hold up a fix for Linux. Looking at the gopkg.in/natefinch/npipe.v2 package, and the CreateNamedPipe docs, it appears that the default security descriptor is being used, which will restrict full access to administrators and the creator.

[John A Meinel (jameinel)]

Right. We do need answers for abstract domain sockets (though from what
you've mentioned, it sounds like we're mostly there already).

This is more about the CVE that we need to get a set of releases out across
the board.

On Sun, May 7, 2017 at 8:26 AM, Andrew Wilkins <<email address hidden>
> wrote:

> It seems my mention of abstract domain sockets was missed, so I'll
> restate in more detail. This doesn't apply to juju-run, but it does
> apply to jujuc (the hook tools: relation-set, relation-get, etc.). The
> jujuc socket is an abstract domain socket, so it has no relationship to
> the filesystem; abstract domain sockets aren't affected by filesystem
> permissions. I think you have to use SCM_CREDENTIALS to do any kind of
> auth on those. This is technically a different bug I guess, but it would
> be good to at least consider how to fix that at the same time.
>
> Seth: on Windows we use named pipes. We'll need to look at whether it is
> similarly affected, but that shouldn't hold up a fix for Linux. Looking
> at the gopkg.in/natefinch/npipe.v2 package, and the CreateNamedPipe
> docs, it appears that the default security descriptor is being used,
> which will restrict full access to administrators and the creator.
>
> --
> You received this bug notification because you are a bug assignee.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1682411
>
> Title:
> juju-run unit root escalation vulnerability
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1682411/+subscriptions
>

[Seth Arnold (seth-arnold)]

Re-reading the comments here, there's a few loose ends:

- First, if we decide to go the route of creating a new directory for the socket, the directory should be created with restricted permissions (see mkdir(2)) rather than creating the directory and then setting the directory's permissions to be restrictive. I'm sorry I missed this point earlier.

- Second, I did entirely overlook the abstract (unix domain address?) sockets; the patches here were all for unix domain sockets with apparently filesystem names. Are there more sockets that need further constraints on their use?

Thanks

[Seth Arnold (seth-arnold)]

There's a small cosmetic issue in an error message:

+ listener, err := net.Listen("unix", tempSocketPath)
+ if err != nil {
+ logger.Errorf("failed to listen on unix:%s: %v", socketPath, err)

Perhaps tempSocketPath should be logged instead of socketPath; that more directly reflects the potentially failed operation.

Thanks

[John A Meinel (jameinel)]

That should, indeed be fixed. Rick Harding also had a good idea that we should port the 'controller_version' string to being reported to charmstore so we can track how many people are switching over to updated versions. We already have similar code in the 2.0 series, so we only really need this patch to the 1.25 series.

[Ian Booth (wallyworld)]

I've patched all relevant Juju branches: 1.25, 2.0, 2.1, develop
The patch includes the suggestion from comment #38 to log tempSocketPath

[Seth Arnold (seth-arnold)]

CVE from MITRE:

Use CVE-2017-9232.

Thanks

[John A Meinel (jameinel)]

Final patch applied to 2.2 and dealing with a few bits of fallout in the test suite, etc. Had missed originally that we were doing an explicit chmod 777 as part of setting up the test, and that sockets don't cleanup on listener.Close() if you've renamed the socket in the mean time. Much thanks to Ian and Andrew for polishing the original idea.

[John A Meinel (jameinel)]

Turns out 2.0.4 had slightly different tweaks necessary vs develop.

[John A Meinel (jameinel)]

Changing perms on /var/lib/.../juju-run meant that in 1.25 it broke "juju run blah". Because 'juju run blah' is user initiated, and they would have permissions to do so via 'juju ssh' anyway, this added 'sudo' to how jujud executed 'juju run' requests.

This is the current rollup of all the changes we made to 1.25 in order to still operate. Note that 1.25 will probably need at least 1 more patch to make the test suite run reliably under Go 1.8 where a lot of the error strings, etc, have been changed. Andrew has a patch for that, but it probably doesn't need to be included in the debdiff, if the Ubuntu golang builders are not using Go 1.8.

[Launchpad Janitor (janitor)]

This bug was fixed in the package juju-core - 2.0.2-0ubuntu2.1

---------------
juju-core (2.0.2-0ubuntu2.1) zesty-security; urgency=medium

  * SECURITY UPDATE: Privilege escalation via juju-run (LP: #1682411)
    - debian/patches/CVE-2017-9232.patch: create a unix domain socket with
      restricted permissions to limit juju-run to only similarly privileged
      processes.
    - CVE-2017-9232

 -- Seth Arnold <email address hidden> Thu, 25 May 2017 17:32:58 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package juju-core - 2.0.2-0ubuntu0.16.10.2

---------------
juju-core (2.0.2-0ubuntu0.16.10.2) yakkety-security; urgency=medium

  * SECURITY UPDATE: Privilege escalation via juju-run (LP: #1682411)
    - debian/patches/CVE-2017-9232.patch: create a unix domain socket with
      restricted permissions to limit juju-run to only similarly privileged
      processes.
    - CVE-2017-9232

 -- Seth Arnold <email address hidden> Thu, 25 May 2017 17:28:34 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package juju-core - 1.25.6-0ubuntu1.14.04.2

---------------
juju-core (1.25.6-0ubuntu1.14.04.2) trusty-security; urgency=medium

  * SECURITY UPDATE: Privilege escalation via juju-run (LP: #1682411)
    - debian/patches/CVE-2017-9232.patch: create a unix domain socket with
      restricted permissions to limit juju-run to only similarly privileged
      processes.
    - CVE-2017-9232

 -- Seth Arnold <email address hidden> Thu, 25 May 2017 16:37:56 -0700

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "chmod after creating the socket" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Launchpad Janitor (janitor)]

This bug was fixed in the package juju-core-1 - 1.25.6-0ubuntu2.16.10.2

---------------
juju-core-1 (1.25.6-0ubuntu2.16.10.2) yakkety-security; urgency=medium

  * SECURITY UPDATE: Privilege escalation via juju-run (LP: #1682411)
    - debian/patches/CVE-2017-9232.patch: create a unix domain socket with
      restricted permissions to limit juju-run to only similarly privileged
      processes.
    - CVE-2017-9232

 -- Seth Arnold <email address hidden> Fri, 26 May 2017 15:30:40 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package juju-core-1 - 1.25.6-0ubuntu1.16.04.2

---------------
juju-core-1 (1.25.6-0ubuntu1.16.04.2) xenial-security; urgency=medium

  * SECURITY UPDATE: Privilege escalation via juju-run (LP: #1682411)
    - debian/patches/CVE-2017-9232.patch: create a unix domain socket with
      restricted permissions to limit juju-run to only similarly privileged
      processes.
    - CVE-2017-9232

 -- Seth Arnold <email address hidden> Fri, 26 May 2017 15:29:18 -0700

[Seth Arnold (seth-arnold)]

Thanks to everyone.