cfntools command injection

Bug #1312246 reported by Jason Hullinger
268
This bug affects 1 person
Affects Status Importance Assigned to Milestone
heat-cfntools
Fix Released
Medium
Anant Patil

Bug Description

The package heat-cfntools, which are Cloud Formation tools installed on instance images, contains a command line injection vulnerability in the way it launches a subprocess. The class CommandRunner, in the file cfntools/cfn_helper.py, contains the function run that executes the command string given in the CommandRunner constructor. It does so by creating the following command arguments:

cmd = ['su', user, '-c', self._command]

Since `su {user} -c` creates another subprocess it will evaluate and execute the self._command string, making is susceptible to command line injection. Additionally, su {user} -c must be ran as root in this context otherwise the program will wait for the user to input a password.

An example of this can be found in the utility in bin/cfn-signal, which signals when an application is ready, by sending a curl request containing external input:

cmd_str = "curl -X PUT -H \'Content-Type:\' --data-binary \'%s\' \"%s\"" % \
    (cfn_helper.json.dumps(body), args.url)

Although the HTTP Request body is json encoded, the escape sequence \' does not escape a single quote around a command line string. Additionally, if the second argument, the URL, contains a double quote it will also escape from the curl string.

Examples and steps to reproduce:

As root, run the following commands:

cfn-signal -d "some content' http://www.example.com; echo hello>>/tmp/hello; #" http://www.foobar.com
cfn-signal 'http://www.example.com/?"; echo hello>/tmp/hello; #'

Where in each example, `echo hello>>/tmp/hello;` will be executed.

Depending upon how the Heat template was built, if an attacker can have input on any arguments in the JSON request body it would be possible to inject arbitrary commands ran as root on the instance. This, however, is just one example of many calls to CommandRunner using input from external resources.

It is recommended that CommandRunner try to elevate or demote itself to the desired user id and then call Popen with an argument list containing the command and arguments instead of an inline command and argument string such as:

cmd = ["ls", "-la", "/tmp/"];
try:
    os.setuid(0);
    subprocess.Popen(cmd);
except Exception as ex:
    print "Error: %s" % ex

instead of:

cmd = ['su', user, '-c', self.command]
subprocess.Popen(cmd)

This will require that all CommandRunner calls be changed to a list instead of a string throughout the cfntools package.

Environment:

Ubuntu 12.04
OS X Mavericks

Tags: cfntools heat
Changed in ossa:
status: New → Incomplete
Revision history for this message
Thierry Carrez (ttx) wrote :

heat-cfntools is currently not covered by OSSAs (see https://wiki.openstack.org/wiki/Security_supported_projects). We are considering its addition though, but a thorough security audit needs to be conducted and various issues be fixed first.

My inclination here would be to open this bug publicly and get it fixed asap. It's a bit borderline anyway, since the attack scenario is likely to need social engineering to succeed.

Revision history for this message
Jason Hullinger (jason-hullinger) wrote :

The attack scenario is just really one example due to how the function run in CommandRunner executes a subprocess. I do agree that it may require the attacker to conduct some social engineering, but it really depends upon how a user implements cfntools such as in a Heat template. For example, upon failure cfn-signal responds with a reason and if the error response contains input from an external resource it could contain malicious data. That said, I would consider this a 'negative day' vulnerability, meaning there is a known way to exploit the tool but not a known active exploit. I also agree with your assessment on opening the bug publicly.

Revision history for this message
Robert Clark (robert-clark) wrote :

I understand that the OSSA process doesn't cover heat-cfntools at this time - so in my opinion this issue should be kept private until either Jason or the appropriate person from heat-cfntools decides to make it public.

I have no objection to this bug being kept private and being closed - with Jason following up with the heat-cfntools guys directly.

Revision history for this message
Jason Hullinger (jason-hullinger) wrote :

I've added Steve Baker and Heat Drivers to the bug subscribers. I'll send a follow up email to Steve as well.

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

I wonder about the impact of this, given that cfn-init is executed by a user_data script which already runs as root, imperative commands can already be run in a compromised environment without having to subvert anything in cfntools.

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

Having said that, I'm happy for this to be fixed.

Revision history for this message
Zane Bitter (zaneb) wrote :

I initially thought the same, but this is used not only for cfn-init. As pointed out above, e.g. cfn-signal passes unescaped data to a shell to interpret. And while it's difficult to imagine a plausible vector for deliberately exploiting this, it should probably still be fixed.

Revision history for this message
Steven Dake (sdake) wrote :

heat-cfntools are typically run with CAP_SYS_ADMIN capabilities, so I am missing how a user injecting something locally into a VM they already have CAP_SYS_ADMIN capabilities on matters when the vm owner can already escalate their privileges via something simple like sudo -i and run wild. In the heat model, the instance's security model is protected exclusively by nova's registered SSH keys. Maybe that should change and is worthy of analysis but imo this particular case only becomes exploitable if the security model of instances is altered (eg: if we remove the ability for VM owners to operate with the full power of the admin context.

Regards
-steve

Thierry Carrez (ttx)
no longer affects: ossa
Revision history for this message
Zane Bitter (zaneb) wrote :

The point is that you can put in your user_data script:

  cfn-signal -d $(get_untrusted_data) <WaitCondition URL>

and the untrusted data is passed, unescaped, to a root shell that can potentially execute it.

For example, the get_untrusted_data command could return, say, "'; rm -rf / '" and instead of posting the string "'; rm -rf / '" to the Heat server it would instead delete all of your files, which is really not what the user would expect.

Revision history for this message
Jason Hullinger (jason-hullinger) wrote :

To add to Zane's point, despite it being a potential security issue, if there were a single quote in the JSON request, such as in the reason for an error response, the command would fail.

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

I have thought of an exploit scenario which may become an issue in the future.

Imagine a trusted template which can be launched by an untrusted user via some web interface. The user can only specify template parameters. They could craft a parameter value which injects an arbitrary command with this exploit.

I don't believe heat is being used in this way today, but it may in the future, especially when a template catalog exists.

Anant Patil (ananta)
Changed in heat-cfntools:
assignee: nobody → Anant Patil (ananta)
Steven Hardy (shardy)
Changed in heat-cfntools:
status: New → Triaged
importance: Undecided → Medium
Zane Bitter (zaneb)
information type: Private Security → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat-cfntools (master)

Fix proposed to branch: master
Review: https://review.openstack.org/211435

Changed in heat-cfntools:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat-cfntools (master)

Reviewed: https://review.openstack.org/211435
Committed: https://git.openstack.org/cgit/openstack/heat-cfntools/commit/?id=e424af2236ed1d6e6d0e11768f873ffe4e696221
Submitter: Jenkins
Branch: master

commit e424af2236ed1d6e6d0e11768f873ffe4e696221
Author: Anant Patil <email address hidden>
Date: Tue Aug 11 11:15:18 2015 +0530

    Fix cfntools command injection

    The CommandRunner used to run commands using su command and passing the
    actual command to be run as argument to it.

    su USER -c <cmd>

    This is susceptible to command line injection as noted in the bug.

    The fix required to do two things:

    1. Pass the command to be run as list instead of a string. This is to
    ensure that the actual arguments are passed as arguments to the program
    ought to be executed. And by doing so, avoids running any commands
    passed in the argument. On the contrary, if the command were passed as a
    string to the shell, the arguments could be formed in a way to execute
    malicious commands.

    2. The CommandRunner runs the command directly and uses setuid to lower
    the privileges if needed. If the 'runas' user is other than root, then
    its UID is obtained and setuid is invoked to set the real user-id and
    effective user-id to the given user.

    Change-Id: I654117e994fd38411508dbe9b85d06c28dc0e411
    Closes-Bug: #1312246

Changed in heat-cfntools:
status: In Progress → Fix Committed
Changed in heat-cfntools:
milestone: none → v1.4.0
Changed in heat-cfntools:
status: Fix Committed → Fix Released
Revision history for this message
Anant Patil (ananta) wrote :

I did a bit of goof up, as noted in the comment at
https://review.openstack.org/#/c/211435/

I did some analysis and I feel that the command supplied to the command
runner should be left as a string for two reasons:

1. The cfn-hup allows external command to be passed as "action" via the
hooks.conf file. This action can be any arbitrary command and there is
no way to impose any restriction on type. The commands are written in
hooks.conf, as they would be on shell, and cfn-hup has to just read the
command and execute it as a user.

2. It is probably easier to write the commands as string than as list,
like typing on a shell.

To fix the issue, the command runner needs a way to properly convert the
string into a list. I used a simple split() which is not good enough, as
it would simply split based on space without honouring the
space-separated arguments passed to command. I have found that the shlex
library solves our problem by properly splitting the command string so
that it can be run with python subprocess. For example,

>>> data = { "auth": {"identity": {"methods": ["password"],
          "password": {
           "user": {
            "name": "admin",
            "domain": {"id": "default"},
            "password": "nomoresecrete"
            }
           }
          }
    }
  }
>>> url='http://localhost:5000/v3/auth/tokens'

>>> include="--include"
>>> cmd_str=("curl %s -H \'Content-Type:\' -d \'%s\' \"%s\"" %
  (include, json.dumps(data), url))
>>> cmd_str
'curl --include -H \'Content-Type:\' -d \'{"auth": {"identity": {"password": {"user": {"domain": {"id": "default"}, "password": "nomoresecrete", "name": "admin"}}, "methods": ["password"]}}}\' "http://localhost:5000/v3/auth/tokens"'
>>> cmd=shlex.split(cmd_str)
>>> cmd
['curl', '--include', '-H', 'Content-Type:', '-d', '{"auth": {"identity": {"password": {"user": {"domain": {"id": "default"}, "password": "nomoresecrete", "name": "admin"}}, "methods": ["password"]}}}', 'http://localhost:5000/v3/auth/tokens']
>>> for i in cmd:
 print i

curl
--include
-H
Content-Type:
-d
{"auth": {"identity": {"password": {"user": {"domain": {"id": "default"}, "password": "nomoresecrete", "name": "admin"}}, "methods": ["password"]}}}
http://localhost:5000/v3/auth/tokens
>>> sub = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> o=sub.communicate()
>>> print o[0]
HTTP/1.1 201 Created

Date: Fri, 04 Sep 2015 11:31:58 GMT

Server: Apache/2.4.7 (Ubuntu)

X-Subject-Token: 57adc05d872d46878559b117706e917d

Vary: X-Auth-Token

x-openstack-request-id: req-041e6ae0-7162-4e2c-967e-4afff9931d63

Content-Length: 297

Content-Type: application/json

{"token": {"methods": ["password"], "expires_at": "2015-09-04T12:31:58.207663Z", "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "36be4c8e3bc844d785820467aa1ca744", "name": "admin"}, "audit_ids": ["HRWi0tOSQAKcHwnZikAlng"], "issued_at": "2015-09-04T11:31:58.207875Z"}}
>>>

I am planning to put a patch where we use shlex.split() to split the
command into a list and pass it to subprocess.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat-cfntools (master)

Fix proposed to branch: master
Review: https://review.openstack.org/220483

Revision history for this message
Zane Bitter (zaneb) wrote :

There's also a bug in _handle_apt_packages, where we're trying to set an environment variable:

        cmd_str = 'DEBIAN_FRONTEND=noninteractive apt-get -y install %s' % \
                  pkg_list
        CommandRunner(cmd_str).run()

This won't work now that the command is not being interpreted by the shell.

Revision history for this message
Zane Bitter (zaneb) wrote :

I just noticed that this already made it into a release, so I reported some new bugs:

bug 1492341
bug 1492356
bug 1492362
bug 1492367
bug 1492371
bug 1492381

3 of these are Critical showstoppers and 3 High priority regressions, so I think we need to get them fixed ASAP and cut a new release. As I noted on the review https://review.openstack.org/220483 the shlex approach will not work. We need two different solutions to two different use cases (user-supplied command strings, and internal or use-supplied command lists).

Revision history for this message
Steve Baker (steve-stevebaker) wrote :

It looks like this should be reverted and a 1.4.1 released.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/222030

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on heat-cfntools (master)

Change abandoned by Anant Patil (<email address hidden>) on branch: master
Review: https://review.openstack.org/220483
Reason: Start from scratch https://review.openstack.org/#/c/222030/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat-cfntools (master)

Reviewed: https://review.openstack.org/222030
Committed: https://git.openstack.org/cgit/openstack/heat-cfntools/commit/?id=f427a69443d9f50c50ecc9dadaee7e393e21d166
Submitter: Jenkins
Branch: master

commit f427a69443d9f50c50ecc9dadaee7e393e21d166
Author: Anant Patil <email address hidden>
Date: Thu Sep 10 09:46:22 2015 +0530

    Use seteuid instead of su to control privileges

    Control the privileges by setting the effective UID before running the
    command. Earlier we used to run command using su -c "USER".

    Original EUID is restored after running the command. This is required to
    run multiple commands in succession with different run-as users.

    Change-Id: I414fc6a802f11deb320b43c6d011f802a42c40c9
    Partial-Bug: #1312246

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/222554
Committed: https://git.openstack.org/cgit/openstack/heat-cfntools/commit/?id=2710bba2cb3b2cfb52488fe7aa84d5b385d61c30
Submitter: Jenkins
Branch: master

commit 2710bba2cb3b2cfb52488fe7aa84d5b385d61c30
Author: Anant Patil <email address hidden>
Date: Thu Sep 10 11:27:25 2015 +0530

    Convert all internal commands to list

    Make all internal commands as list to avoid any possibility of command
    line injection. Commands supplied as string are susceptible to
    substitution.

    All the internal commands are supplied as list to CommandRunner. As a
    convention, all the commands must be given as list to subprocess except
    the commands read from file, like in case of cfn hooks and commands
    section in metadata.

    Few internal commands require shell redirects and they will be
    implemented in another patch.

    Change-Id: Ifabaf44e341144bc85508dc05c76b1d83e41ae44
    Partial-Bug: #1312246

Revision history for this message
Anant Patil (ananta) wrote :

I think we can close this bug now with the above two patches and the other bugs raised. I just owned those bugs to make sure if the fix for this bug doesn't again cause those issues.

There are two more bugs which need to be fixed, but they are not required for this issue:
bug 1498298
bug 1498300

Changed in horizon:
status: New → Fix Committed
status: Fix Committed → New
Matthias Runge (mrunge)
Changed in horizon:
status: New → Invalid
no longer affects: horizon
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.