[OSSA 2016-013] Heat: template source URL allows network port scan (CVE-2016-9185)

[Tom Patzig]

Launching a new Heat stack and giving the template from an URL like http://localhost:22

Results in an error message like:

ERROR: Could not retrieve template: Failed to retrieve template: ('Connection aborted.', BadStatusLine('SSH-2.0-OpenSSH_6.6.1\r\n',))

This is a security issue as it allows users to scan the network for listening ports.

heat CLI does not allow that:

heat stack-create -u http://localhost:22 test
[Errno 104] Connection reset by peer

[Morgan Fainberg (mdrnstm)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Akihiro Motoki (amotoki)]

Could you share more detail on how to reproduce the issue reported?

I tried to reproduce it but I failed so far. What I did is as follows:
- Visit Horizon stack table page
- Click 'Launch Stack' button
- Select 'URL' as 'Template Source' and specify 'http://localhost:22' as 'Template URL'
- After clicking 'Next', I got '[Errno 104] Connection reset by peer' (as CLI returns)

Note that I used the master branch (newton) of Horizon code and python-heatclient (1.3.0).

[Morgan Fainberg (mdrnstm)]

The "Connection Reset by Peer" is an error I'd expect in a number of cases that could still
indicate the port scan could be completed.

It might be the case that the CLI error is because it is trying to connect to port 22 on the end-user's machine which it is not running. It is important to verify "ssh" is running on whatever machine (or something on port 22) on the machine executing the heat template get (aka the machine running horizon or the user's local machine if it is the CLI).

[Tom Patzig (tom-patzig)]

I reproduced that issue with stable Liberty.

Horizon shows me the SSH banner and on the same host the heat CLI returns 'Connection reset by peer'.

[Morgan Fainberg (mdrnstm)]

@Tom,

Thanks for clarifying!

[Tristan Cacqueray (tristan-cacqueray)]

Would it be acceptable if the BadStatusLine exception was replaced by a more generic exception such as HTTPException before it is returned to client ?

[Tom Patzig (tom-patzig)]

Yes, horizon should behave like the CLI and not return remote content to the User, if it recognizes that there is no valid template to load from remote.

[Tristan Cacqueray (tristan-cacqueray)]

@horizon-coresec, any progress ?

[Matthias Runge (mrunge)]

This is a general issue. You probably don't want your horizon host to spoof your network.
In any case, you probably don't want to expose the information of running services in the internal network.

If I'm not mistaken, Horizon strictly uses the heat api, and does not connect to the resource itself.

[Rob Cresswell (robcresswell-deactivatedaccount)]

We don't do anything other than wrap the client, so I'd be very surprised if there is an issue here that doesn't also exist in the Heat API itself, even if the CLI command has some data cleaning going on.

[Rob Cresswell (robcresswell-deactivatedaccount)]

Okay, I tested this again on Horizon, and I can't create anything different from what the CLI returns. Is there any more detail on how this might be recreated?

[Tom Patzig (tom-patzig)]

As I said, the issue described occurs with Liberty.

[Daniel Gonzalez Nothnagel (dgonzalez)]

This problem originates in the template validate method of the heat API: http://developer.openstack.org/api-ref/orchestration/v1/?expanded=validate-template-detail

If the method is called with '{"template_url": "http://localhost:22"}' the same issue as raised by Tom appears, as you can see in the following curl command:

curl -H "Content-Type: application/json" -H "X-Auth-Token: 591b637a85af47c0bd406cc7db9c3cb1" -X POST -d '{"template_url": "http://localhost:22"}' http://127.0.0.1:8004/v1/3659e5effe5a473f9dd579dd8bbeca9f/validate
{"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": {"message": "Could not retrieve template: Failed to retrieve template: ('Connection aborted.', BadStatusLine('SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1\\r\\n',))", "traceback": " File \"/opt/stack/heat/heat/common/wsgi.py\", line 842, in __call__\n request, **action_args)\n File \"/opt/stack/heat/heat/common/wsgi.py\", line 916, in dispatch\n return method(*args, **kwargs)\n File \"/opt/stack/heat/heat/api/openstack/v1/util.py\", line 38, in handle_stack_method\n return handler(controller, req, **kwargs)\n File \"/opt/stack/heat/heat/api/openstack/v1/stacks.py\", line 610, in validate_template\n data.template(),\n File \"/opt/stack/heat/heat/api/openstack/v1/stacks.py\", line 122, in template\n raise exc.HTTPBadRequest(err_reason)\n", "type": "HTTPBadRequest"}, "title": "Bad Request"}

This does not only happen with liberty, but also with newer releases and the current master version.
You can't see this issue in newer version of horizon, because they changed the behaviour from calling the validate method with 'template_url' to fetching the template themselves from the given url and then calling the validate method with 'template'.

So "ERROR: Could not retrieve template: Failed to retrieve template: ('Connection aborted.', BadStatusLine('SSH-2.0-OpenSSH_6.6.1\r\n',))" is the error message issued by the heat-api, whereas "[Errno 104] Connection reset by peer" is the error message issued by the client and newer versions of horizon.

As I see it, the easiest way to fix this would be to change the error message in https://github.com/openstack/heat/blob/master/heat/common/urlfetch.py#L54 to not include the error message from the URLError. So instead of raising URLFetchError(_('Failed to retrieve template: %s') % sex) it should just raise URLFetchError(_('Failed to retrieve template')).

TL;DR The problem lies in the heat API, not in horizon. Easiest way to fix this would be to suppress the exact error message in heat.

[Daniel Gonzalez Nothnagel (dgonzalez)]

Here is a patch that should fix this issue in heat.

[Daniel Gonzalez Nothnagel (dgonzalez)]

Sorry, my previous patch contains a pep8 warning.
This patch solves that problem.

[Zane Bitter (zaneb)]

Patch looks good to me. It'd be even better if we could log the exception so the operator can see what's happening.

Long-term I'd love to get rid of the ability to specify a template_url altogether, but that wouldn't be backwards-compatible so we're effectively forced to wait for a new API version.

[Daniel Gonzalez Nothnagel (dgonzalez)]

Zane, thanks for the suggestion to log the incident.

Here is a new patch which logs the exception.
I chose info as log level and used the same message the exception had before.
I hope that's ok? If not I can update the patch with a new log level/message.

[Zane Bitter (zaneb)]

Perfect, +2 from me.

[Tristan Cacqueray (tristan-cacqueray)]

Proposed impact description:

Title: Network information disclosure through Heat template source URL
Reporter: Tom Patzig (SAP)
Products: Heat
Affects: >=5.0.0 <=5.0.3, >=6.0.0 <=6.1.0 and ==7.0.0

Description:
Tom Patzig from SAP reported a vulnerability in Heat. By launching a new Heat stack with a local URL an authenticated user may conduct network discovery revealing internal network configuration. All Heat setup are affected.

[Daniel Gonzalez Nothnagel (dgonzalez)]

The impact description lgtm.

When can we disclose this bug? Or is there anything missing we need to do before disclosing the bug and uploading the fix to gerrit?

[Tristan Cacqueray (tristan-cacqueray)]

If we do an embargoed disclosure, then we could submit the patch around next week or the week after (most openstacker are still recovering from the summit I guess).

Though this doesn't seems like a critical bug and it could probably skip the embargoed disclosure part. In that case we should at least wait for the CVE assignment before making it public so that we can fast-track the advisory.

Though?

[Zane Bitter (zaneb)]

I agree that we can probably skip the embargoed disclosure for this one.

[Jeremy Stanley (fungi)]

Tristan's impact description in comment #10 looks good to me, unless versions prior to 5.0.0 are also vulnerable (in which case we should omit the <=5.0.0 from the Affects line).

As for "skipping" advance notification but still doing everything else under embargo, I don't think there's a viable middle ground. Either the bug is not severe enough to need an embargo in which case we should go ahead and open it now even if patches aren't ready and CVE assignment hasn't happened, or it's severe enough we need to continue to follow our pre-OSSA downstream notification process with its usual coordination timeline. Waiting to disclose the bug until we have fixes figured out and tracking assigned in private but not giving our downstream stakeholders a heads up about it sends mixed signals to the community at large.

[Jeremy Stanley (fungi)]

Sorry, I meant Tristan's impact description in comment #19.

[Morgan Fainberg (mdrnstm)]

++ to everything Jeremy said.

#19 looks good to me.

I prefer to err on the side of caution and since this has been maintained "private" this should be pushed as embargoed.

We should not skip/change the process. It is important to maintain consistency in reporting/disclosing security bugs.

[Zane Bitter (zaneb)]

I'm pretty sure that this bug has been around substantially forever, so we should probably put Affects: <=7.0.0

Now that I understand the process a little better from comment #23, I think I'm saying that this is IMHO not severe enough to require an embargo. If waiting for a CVE is not appropriate then I'd be fine with flipping this to public now and just committing the patch (which is ready to go).

[Morgan Fainberg (mdrnstm)]

@Zane,

Works for me then.

--Morgan

[Tristan Cacqueray (tristan-cacqueray)]

CVE has been requested with this affect line: <=5.0.3, >=6.0.0 <=6.1.0 and ==7.0.0

@Daniel, the bug is now public, feel free to submit patches to gerrit for master (Ocata), Newton, Mikata and Liberty.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/393146

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/393147

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/393148

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/393149

[Matthew Thode (prometheanfire)]

has the fix been merged into master or mitaka?

[Matthew Thode (prometheanfire)]

or newton...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/393146
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=eab9a33ce760c55695a5beb2e541487588b08c98
Submitter: Jenkins
Branch: master

commit eab9a33ce760c55695a5beb2e541487588b08c98
Author: Daniel Gonzalez <email address hidden>
Date: Mon Oct 17 10:22:42 2016 +0200

    Prevent template validate from scanning ports

    The template validation method in the heat API allows to specify the
    template to validate using a URL with the 'template_url' parameter.

    By entering invalid http URLs, like 'http://localhost:22' it is
    possible to scan ports by evaluating the error message of the request.

    For example, the request

    curl -H "Content-Type: application/json" -H "X-Auth-Token: <TOKEN>" \
    -X POST -d '{"template_url": "http://localhost:22"}' \
    http://127.0.0.1:8004/v1/<TENANT_ID>/validate

    causes the following error message to be returned to the user:

    "Could not retrieve template: Failed to retrieve template:
    ('Connection aborted.',
    BadStatusLine('SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1\\r\\n',))"

    This could be misused by tenants to gain knowledge about the internal
    network the heat API runs in.

    To prevent this information leak, this patch alters the error message
    to not include such details when the url scheme is not 'file'.

    SecurityImpact

    Closes-Bug: #1606500

    Change-Id: Id1f86f41c1e6c028d889eca7ccbb9cde67631950

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/393147
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=02dfb1a64f8a545a6dfed15245ac54c8ea835b81
Submitter: Jenkins
Branch: stable/newton

commit 02dfb1a64f8a545a6dfed15245ac54c8ea835b81
Author: Daniel Gonzalez <email address hidden>
Date: Mon Oct 17 10:22:42 2016 +0200

    Prevent template validate from scanning ports

    The template validation method in the heat API allows to specify the
    template to validate using a URL with the 'template_url' parameter.

    By entering invalid http URLs, like 'http://localhost:22' it is
    possible to scan ports by evaluating the error message of the request.

    For example, the request

    curl -H "Content-Type: application/json" -H "X-Auth-Token: <TOKEN>" \
    -X POST -d '{"template_url": "http://localhost:22"}' \
    http://127.0.0.1:8004/v1/<TENANT_ID>/validate

    causes the following error message to be returned to the user:

    "Could not retrieve template: Failed to retrieve template:
    ('Connection aborted.',
    BadStatusLine('SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1\\r\\n',))"

    This could be misused by tenants to gain knowledge about the internal
    network the heat API runs in.

    To prevent this information leak, this patch alters the error message
    to not include such details when the url scheme is not 'file'.

    SecurityImpact

    Closes-Bug: #1606500

    Change-Id: Id1f86f41c1e6c028d889eca7ccbb9cde67631950
    (cherry picked from commit eab9a33ce760c55695a5beb2e541487588b08c98)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/393148
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=8c681f2641ab81410a8fb99bd76ec735ba3add1e
Submitter: Jenkins
Branch: stable/mitaka

commit 8c681f2641ab81410a8fb99bd76ec735ba3add1e
Author: Daniel Gonzalez <email address hidden>
Date: Mon Oct 17 10:22:42 2016 +0200

    Prevent template validate from scanning ports

    The template validation method in the heat API allows to specify the
    template to validate using a URL with the 'template_url' parameter.

    By entering invalid http URLs, like 'http://localhost:22' it is
    possible to scan ports by evaluating the error message of the request.

    For example, the request

    curl -H "Content-Type: application/json" -H "X-Auth-Token: <TOKEN>" \
    -X POST -d '{"template_url": "http://localhost:22"}' \
    http://127.0.0.1:8004/v1/<TENANT_ID>/validate

    causes the following error message to be returned to the user:

    "Could not retrieve template: Failed to retrieve template:
    ('Connection aborted.',
    BadStatusLine('SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1\\r\\n',))"

    This could be misused by tenants to gain knowledge about the internal
    network the heat API runs in.

    To prevent this information leak, this patch alters the error message
    to not include such details when the url scheme is not 'file'.

    SecurityImpact

    Closes-Bug: #1606500

    Change-Id: Id1f86f41c1e6c028d889eca7ccbb9cde67631950
    (cherry picked from commit eab9a33ce760c55695a5beb2e541487588b08c98)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/393149
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=abfe2370edf7eda54fb5d7fc022d1e79974c8dfd
Submitter: Jenkins
Branch: stable/liberty

commit abfe2370edf7eda54fb5d7fc022d1e79974c8dfd
Author: Daniel Gonzalez <email address hidden>
Date: Mon Oct 17 10:22:42 2016 +0200

    Prevent template validate from scanning ports

    The template validation method in the heat API allows to specify the
    template to validate using a URL with the 'template_url' parameter.

    By entering invalid http URLs, like 'http://localhost:22' it is
    possible to scan ports by evaluating the error message of the request.

    For example, the request

    curl -H "Content-Type: application/json" -H "X-Auth-Token: <TOKEN>" \
    -X POST -d '{"template_url": "http://localhost:22"}' \
    http://127.0.0.1:8004/v1/<TENANT_ID>/validate

    causes the following error message to be returned to the user:

    "Could not retrieve template: Failed to retrieve template:
    ('Connection aborted.',
    BadStatusLine('SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.1\\r\\n',))"

    This could be misused by tenants to gain knowledge about the internal
    network the heat API runs in.

    To prevent this information leak, this patch alters the error message
    to not include such details when the url scheme is not 'file'.

    SecurityImpact

    Closes-Bug: #1606500

    Change-Id: Id1f86f41c1e6c028d889eca7ccbb9cde67631950
    (cherry picked from commit eab9a33ce760c55695a5beb2e541487588b08c98)

[Daniel Gonzalez Nothnagel (dgonzalez)]

@Matthew: The fix has now been merged into master, newton, mitaka and liberty.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/393640

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 8.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/393640
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=a8ca0d0e3eaa5a67ea1b9406e6fa274edaff4d84
Submitter: Jenkins
Branch: master

commit a8ca0d0e3eaa5a67ea1b9406e6fa274edaff4d84
Author: Tristan Cacqueray <email address hidden>
Date: Fri Nov 4 08:27:21 2016 +0000

    Adds OSSA-2016-013 (CVE-2016-9185)

    Related-Bug: 1606500
    Change-Id: I252bb88c12db7c6130864fa64a5e73d02439799d

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 7.0.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/heat 6.1.1 release.

[John Herdman (jherdman)]

While this issue appears fixed for any localhost calls, Horizon/Heat still returns additional information when an IP is provided. For instance, when using http://x.x.x.x:22, if the IP is reachable, the SSH banner is returned. It seems like it should only return "Template format version not found". This occurs when using the URL source, as well as when using direct input and providing the source in the user_data / get_file format.

[Jeremy Stanley (fungi)]

John: It might be worth opening a new bug report about your observation, since this was marked as fixed in releases made 7 years ago.