[OSSA-2020-001] Nova can leak consoleauth token into log files (CVE-2015-9543)

[Paul Carlton]

when instance console is accessed auth token is displayed nova-consoleauth.log

nova-consoleauth.log:874:2015-09-02 14:20:36 29941 INFO nova.consoleauth.manager [req-6bc7c116-5681-43ee-828d-4b8ff9d566d0 fe3cd6b7b56f44c9a0d3f5f2546ad4db 37b377441b174b8ba2deda6a6221e399] Received Token: f8ea537c-b924-4d92-935e-4c22ec90d5f7, {'instance_uuid': u'dd29a899-0076-4978-aa50-8fb752f0c3ed', 'access_url': u'http://192.168.245.9:6080/vnc_auto.html?token=f8ea537c-b924-4d92-935e-4c22ec90d5f7', 'token': u'f8ea537c-b924-4d92-935e-4c22ec90d5f7', 'last_activity_at': 1441203636.387588, 'internal_access_path': None, 'console_type': u'novnc', 'host': u'192.168.245.6', 'port': u'5900'}
nova-consoleauth.log:881:2015-09-02 14:20:52 29941 INFO nova.consoleauth.manager [req-a29ab7d8-ab26-4ef2-b942-9bb02d5703a0 None None] Checking Token: f8ea537c-b924-4d92-935e-4c22ec90d5f7, True

and

nova-novncproxy.log:30:2015-09-02 14:20:52 31927 INFO nova.console.websocketproxy [req-a29ab7d8-ab26-4ef2-b942-9bb02d5703a0 None None] 3: connect info: {u'instance_uuid': u'dd29a899-0076-4978-aa50-8fb752f0c3ed', u'internal_access_path': None, u'last_activity_at': 1441203636.387588, u'console_type': u'novnc', u'host': u'192.168.245.6', u'token': u'f8ea537c-b924-4d92-935e-4c22ec90d5f7', u'access_url': u'http://192.168.245.9:6080/vnc_auto.html?token=f8ea537c-b924-4d92-935e-4c22ec90d5f7', u'port': u'5900'}

This token has a short lifetime but the exposure still represents a potential security weakness, especially as the log record in question are INFO level and thus available via centralized logging. A user with real time access to these records could mount a denial of service attack by accessing the instance console and performing a ctl alt del to reboot it

Alternatively data privacy could be compromised if the attacker were able to obtain user credentials

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

I've switched this report from public to public security since it seems to describe a potential vulnerability.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/220622

[Jeremy Stanley (fungi)]

I've added a bugtask for oslo.utils because of partial fix https://review.openstack.org/220620 in that repository.

[Jeremy Stanley (fungi)]

Is this behavior present in stable/kilo (and perhaps earlier) or only master?

[Paul Carlton (paul-carlton2)]

I see it in stable/kilo and earlier too

[Tristan Cacqueray (tristan-cacqueray)]

Assuming this affects all Nova setups, here is the impact description:

Title: Potential leak of consoleauth token into log files
Reporter: Paul Carlton (HP)
Products: Nova
Affects: versions through 2014.2.3, and 2015.1 versions through 2015.1.1

Description:
Paul Carlton from HP reported a vulnerability in Nova. An attacker with read access to the services’ logs may obtain token used for console access. All Nova setups are affected.

[Matt Riedemann (mriedem)]

I'm sort of surprised there is an OSSA for this. We've had many cases of leaked passwords in the nova logs, like when logging connection_info from a block device mapping (might have the admin userid/password in it from the Cinder volume connection) - that kind of thing gets logged quite a bit at debug level. And from what I can remember, we haven't had OSSAs for those changes in the past.

[Tristan Cacqueray (tristan-cacqueray)]

VMT has always considered password leak in logs file OSSA worthy, excepted in DEBUG mode.
Is there a bug for the connection_info from a block device mapping issue ?

[Grant Murphy (gmurphy)]

+1 to impact description with affects line update

[Matt Riedemann (mriedem)]

@Tristan, there is a related nova bug 1321785 for the connection_info field in block_device_mapping - and what comes back from cinder's os-initialize_connection API, which keystoneclient was logging the response, fixed here:

https://review.openstack.org/#/c/219004/

[Jeremy Stanley (fungi)]

Matt, that (bug 1490693) seems to be an exposure only at DEBUG level, for which we've never issued advisories and always classified as security hardening improvements instead.

[Paul Carlton (paul-carlton2)]

Does that mean we will not be back porting changes to juno, kilo and liberty stable?

[Jeremy Stanley (fungi)]

Paul, it means that for bug 1490693 if you backport those we still don't need to issue a security advisory.

For the current bug, the credential leak seems to be at INFO level, which is a situation where we typically do issue an advisory.

[Jeremy Stanley (fungi)]

Note that this bug is stalled waiting for the master branch fix to get unstuck. We need that and working backports to affected stable branches, then the VMT can issue an advisory.

[Paul Carlton (paul-carlton2)]

I am currently working on updating some specs for live migration, which need to be done ASAP to get approved before December deadline for Mitaka, will get back to fixing this as soon as I can

[Tristan Cacqueray (tristan-cacqueray)]

Paul, any progress on that issue ?

[Tristan Cacqueray (tristan-cacqueray)]

Impact description update to include proper affected releases:

Title: Potential leak of consoleauth token into log files
Reporter: Paul Carlton (HP)
Products: Nova
Affects: =>2015.1, <= 2015.1.2, ==12.0.0

Description:
Paul Carlton from HP reported a vulnerability in Nova. An attacker with read access to the services’ logs may obtain token used for console access. All Nova setups are affected.

[Grant Murphy (gmurphy)]

+1 impact description.

[Jeremy Stanley (fungi)]

Tristan's updated impact description in comment #17 looks great.

[Jeremy Stanley (fungi)]

Oh, except the first comma in the affects line is misleading. "=>2015.1 <=2015.1.2, ==12.0.0" would make it clearer the first two versions there indicate the bounds of a range.

[Tristan Cacqueray (tristan-cacqueray)]

@nova-coresec, @Andrea, any progress on this issue ?

[Andrea Rosa (andrea-rosa-m)]

The patch is in conflict, I am going to rebase it then it is up for review

[Paul Carlton (paul-carlton2)]

Fix awaiting reviews

[Tristan Cacqueray (tristan-cacqueray)]

Updated impact description affected versions:

Title: Potential leak of consoleauth token into log files
Reporter: Paul Carlton (HP)
Products: Nova
Affects: <=12.0.3, ==13.0.0

Description:
Paul Carlton from HP reported a vulnerability in Nova. An attacker with read access to the services’ logs may obtain token used for console access. All Nova setups are affected.

[Tristan Cacqueray (tristan-cacqueray)]

@DIMS or @gcb: do you know what is the oslo.utils changeID ?

If it's https://review.openstack.org/#/c/332438/, then is it planned to do backports down to stable/liberty and stable/mitaka ?

[Paul Carlton (paul-carlton2)]

The oslo change was https://review.openstack.org/#/c/220620/ the nova fix no longer depends on this. However the nova fix is still waiting for reviews

[Paul Carlton (paul-carlton2)]

nova fix https://review.openstack.org/#/c/220622/

[Jeremy Stanley (fungi)]

At this point we're going to need a stable/newton backport of https://review.openstack.org/220622 as well.

[Tristan Cacqueray (tristan-cacqueray)]

@nova-coresec: please review https://review.openstack.org/#/c/220622/

[Tristan Cacqueray (tristan-cacqueray)]

Fwiw I just fixed rebase issues and the patch is still up for review: https://review.openstack.org/#/c/220622/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/220622
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=26d4047e17eba9bc271f8868f1d0ffeec97b555e
Submitter: Zuul
Branch: master

commit 26d4047e17eba9bc271f8868f1d0ffeec97b555e
Author: Balazs Gibizer <email address hidden>
Date: Fri Aug 23 15:51:34 2019 +0200

    Mask the token used to allow access to consoles

    Hide the novncproxy token from the logs.

    When backported this patch needs to be extended to handle the same issue
    in the consoleauth service.

    Co-Authored-By:paul-carlton2 <email address hidden>
    Co-Authored-By:Tristan Cacqueray <email address hidden>

    Change-Id: I5b8fa4233d297722c3af08176901d12887bae3de
    Closes-Bug: #1492140

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/696685

[Jeremy Stanley (fungi)]

If we get backports for other supported stable branches, we can probably still issue an advisory for this using Tristan's last impact description in comment #24, it would definitely need the affected versions field updated though.

[melanie witt (melwitt)]

I'm not sure what to think about this one as it's been around for years and we unfortunately have another INFO log exposure [1] of the console auth token that occurred relatively recently [2]:

  INFO nova.console.websocketproxy [-] 10.209.96.111 - - [04/Dec/2019 03:29:50] 10.209.96.111:
  Path: '?token=3e631f39-b5c7-4bba-a5c2-8c76359e71d9'

This one, I don't yet know how to suppress because this logging is coming from the underlying websockify third-party code, not nova code.

So, given that the recent fix doesn't get us out of the woods, I'm not sure whether an advisory at this stage would be useful. I'm not opposed to one though.

[1] https://zuul.opendev.org/t/openstack/build/c32904d9d1424e579a21513cfc66bd7d/log/controller/logs/screen-n-novnc-cell1.txt.gz#11
[2] https://review.opendev.org/649372

[Jeremy Stanley (fungi)]

Thanks for the update, and I agree it sounds like there's not yet a good argument for an advisory. Maybe we need to include a warning somewhere (install docs? security guide?) that short-lived console access credentials can be leaked in hypervisor host logs, and to take appropriate precautions when granting access to those logs.

As for the websockify leak, is there a corresponding bug opened upstream with its maintainers?

[melanie witt (melwitt)]

Short-lived depends on the configured TTL (default is 10 minutes) and they are unintentionally leaked in the INFO logs of the nova-novncproxy console proxy service [1], which does not run on the hypervisor. It's a separate service that is run on a per cell basis.

For the websockify leak, it is not websockify that is directly responsible for the leak because their code is (and has always) logged the "Path" with which the websockify server was called. So, I don't think it's a bug on their part.

To explain the background: earlier this year, novnc made a change to their code to stop supporting built-in token query parameter forwarding to the websockify server. We proposed a partial revert [2] that was NACKed by the maintainers and their guidance was for us to embed our auth token into the "Path" query parameter instead, which is (and has always) been passed through to the websockify server code. There was no other way to get the token passed into websockify so that we could validate it in nova. However, when we made that change, the token began being logged as part of the normal "Path" logging. I personally can't think of another way to stop it other than to adjust our websockify server logging level config to WARN, if that is possible.

[1] https://docs.openstack.org/nova/latest/admin/remote-console-access.html#novnc-based-vnc-console
[2] https://github.com/novnc/noVNC/pull/1220

[Jeremy Stanley (fungi)]

Thanks again for the clarification. So to restate, the novnc maintainers recommend using websockify in a way which exposes authentication tokens in logs, and don't provide any means of mitigating the resulting credential leak. This puts consumers of their software in an unfortunate position, and suggests that we should not continue to encourage use of Nova's novnc integration except in cases where operators are comfortable assuming that risk and taking measures to secure the logs of their per-cell nova-novncproxy daemons.

[melanie witt (melwitt)]

Well, I was thinking maybe we could do something similar to this patch, where we changed the default log level of the oslo.privsep daemon to log at INFO instead of its default DEBUG, in nova:

https://review.opendev.org/#/c/586643/2/nova/config.py

So I was thinking for websockify we could do similar and set the level to WARN instead of INFO.

Doing that isn't ideal for debugging though because we'd miss logs about whether the proxy is running with TLS support, which I have used for troubleshooting before in the past, example:

Dec 05 11:21:47.775193 ubuntu-bionic-inap-mtl01-0013232262 nova-novncproxy[24376]: INFO nova.console.websocketproxy [-] WebSocket server settings:
Dec 05 11:21:47.775615 ubuntu-bionic-inap-mtl01-0013232262 nova-novncproxy[24376]: INFO nova.console.websocketproxy [-] - Listen on 0.0.0.0:6080
Dec 05 11:21:47.776478 ubuntu-bionic-inap-mtl01-0013232262 nova-novncproxy[24376]: INFO nova.console.websocketproxy [-] - Web server (no directory listings). Web root: /usr/share/novnc
Dec 05 11:21:47.776647 ubuntu-bionic-inap-mtl01-0013232262 nova-novncproxy[24376]: INFO nova.console.websocketproxy [-] - SSL/TLS support
Dec 05 11:21:47.777505 ubuntu-bionic-inap-mtl01-0013232262 nova-novncproxy[24376]: INFO nova.console.websocketproxy [-] - proxying from 0.0.0.0:6080 to None:None
...
Dec 05 11:47:49.788915 ubuntu-bionic-inap-mtl01-0013232262 nova-novncproxy[24376]: INFO nova.console.rfb.authvencrypt [None req-d3d9db7c-89fe-44f2-95a6-ef3357a25f1b None None] VeNCrypt security handshake accepted
Dec 05 11:47:49.789176 ubuntu-bionic-inap-mtl01-0013232262 nova-novncproxy[24376]: INFO nova.console.securityproxy.rfb [None req-d3d9db7c-89fe-44f2-95a6-ef3357a25f1b None None] Finished security handshake, resuming normal proxy mode using secured socket

I'll look into the logging issue again to see if I can possibly find another way around.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696685
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d7826bcd761af035f3f76f67c607dde2a1d04e48
Submitter: Zuul
Branch: stable/train

commit d7826bcd761af035f3f76f67c607dde2a1d04e48
Author: Balazs Gibizer <email address hidden>
Date: Fri Aug 23 15:51:34 2019 +0200

    Mask the token used to allow access to consoles

    Hide the novncproxy token from the logs.

    When backported this patch needs to be extended to handle the same issue
    in the consoleauth service.

    Co-Authored-By:paul-carlton2 <email address hidden>
    Co-Authored-By:Tristan Cacqueray <email address hidden>

    Change-Id: I5b8fa4233d297722c3af08176901d12887bae3de
    Closes-Bug: #1492140
    (cherry picked from commit 26d4047e17eba9bc271f8868f1d0ffeec97b555e)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/702181

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702181
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8fbf04f325f593836f8d44b6bbf42b85bde94e3
Submitter: Zuul
Branch: stable/stein

commit d8fbf04f325f593836f8d44b6bbf42b85bde94e3
Author: Balazs Gibizer <email address hidden>
Date: Fri Aug 23 15:51:34 2019 +0200

    Mask the token used to allow access to consoles

    Hide the novncproxy token from the logs.

    When backported this patch needs to be extended to handle the same issue
    in the consoleauth service.

    Co-Authored-By:paul-carlton2 <email address hidden>
    Co-Authored-By:Tristan Cacqueray <email address hidden>

    Conflicts:
      nova/console/websocketproxy.py due to
      I89df8f8fa111b730ddd0aa73ae09a8cd5d152dad missing from stable/stein

      nova/consoleauth/manager.py and
      nova/tests/unit/consoleauth/test_consoleauth.py due to consoleauth is
      removed from stable/train but affected by the bug in stable/stein

    Change-Id: I5b8fa4233d297722c3af08176901d12887bae3de
    Closes-Bug: #1492140
    (cherry picked from commit 26d4047e17eba9bc271f8868f1d0ffeec97b555e)
    (cherry picked from commit d7826bcd761af035f3f76f67c607dde2a1d04e48)

[Jeremy Stanley (fungi)]

If we can also get a patch backported for stable/rocky then we should be able to issue an advisory for this once it merges. Otherwise we can issue an advisory when that branch goes into extended maintenance (estimated one month from today).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/704255

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 20.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/704255
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=08f1f914cc219cf526adfb08c46b8f40b4e78232
Submitter: Zuul
Branch: stable/rocky

commit 08f1f914cc219cf526adfb08c46b8f40b4e78232
Author: Balazs Gibizer <email address hidden>
Date: Fri Aug 23 15:51:34 2019 +0200

    Mask the token used to allow access to consoles

    Hide the novncproxy token from the logs.

    When backported this patch needs to be extended to handle the same issue
    in the consoleauth service.

    Co-Authored-By:paul-carlton2 <email address hidden>
    Co-Authored-By:Tristan Cacqueray <email address hidden>

    Change-Id: I5b8fa4233d297722c3af08176901d12887bae3de
    Closes-Bug: #1492140
    (cherry picked from commit 26d4047e17eba9bc271f8868f1d0ffeec97b555e)
    (cherry picked from commit d7826bcd761af035f3f76f67c607dde2a1d04e48)
    (cherry picked from commit d8fbf04f325f593836f8d44b6bbf42b85bde94e3)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/707845

[Jeremy Stanley (fungi)]

MITRE has assigned CVE-2015-9543 for this vulnerability.

[Jeremy Stanley (fungi)]

The advisory data is being reviewed at https://review.opendev.org/708242 .

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708242
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=28b98cb8337483cc6c48af4cb3f27770b21ac474
Submitter: Zuul
Branch: master

commit 28b98cb8337483cc6c48af4cb3f27770b21ac474
Author: Jeremy Stanley <email address hidden>
Date: Mon Feb 17 20:04:25 2020 +0000

    Add OSSA-2020-001 (CVE-2015-9543)

    Change-Id: If9b675a4cef657f5d4102192821a51bb91d8cbf9
    Closes-Bug: #1492140

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707845
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=366515dcd1090ca2f9f303009c78394b5665ce1f
Submitter: Zuul
Branch: stable/queens

commit 366515dcd1090ca2f9f303009c78394b5665ce1f
Author: Balazs Gibizer <email address hidden>
Date: Fri Aug 23 15:51:34 2019 +0200

    Mask the token used to allow access to consoles

    Hide the novncproxy token from the logs.

    When backported this patch needs to be extended to handle the same issue
    in the consoleauth service.
    Conflicts:
          nova/tests/unit/console/test_websocketproxy.py
    It is due to If1b6e5f20d2ea82d94f5f0550f13189fc9bc16c4 is only
    implemented in rocky

    Co-Authored-By:paul-carlton2 <email address hidden>
    Co-Authored-By:Tristan Cacqueray <email address hidden>

    Change-Id: I5b8fa4233d297722c3af08176901d12887bae3de
    Closes-Bug: #1492140
    (cherry picked from commit 26d4047e17eba9bc271f8868f1d0ffeec97b555e)
    (cherry picked from commit d7826bcd761af035f3f76f67c607dde2a1d04e48)
    (cherry picked from commit d8fbf04f325f593836f8d44b6bbf42b85bde94e3)
    (cherry picked from commit 08f1f914cc219cf526adfb08c46b8f40b4e78232)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.opendev.org/708876

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.3.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/708876
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2927519cf05a40cdd13d21739e39e7aaf574e5c1
Submitter: Zuul
Branch: stable/pike

commit 2927519cf05a40cdd13d21739e39e7aaf574e5c1
Author: Balazs Gibizer <email address hidden>
Date: Fri Aug 23 15:51:34 2019 +0200

    Mask the token used to allow access to consoles

    Hide the novncproxy token from the logs.

    Conflicts:
        nova/tests/unit/consoleauth/test_consoleauth.py
    NOTE: conflict is due to Iffdd4e251bfa2bac1bfd49498e32b738843709de is
    only backported till Queens.

    Co-Authored-By:paul-carlton2 <email address hidden>
    Co-Authored-By:Tristan Cacqueray <email address hidden>

    Change-Id: I5b8fa4233d297722c3af08176901d12887bae3de
    Closes-Bug: #1492140
    (cherry picked from commit 26d4047e17eba9bc271f8868f1d0ffeec97b555e)
    (cherry picked from commit d7826bcd761af035f3f76f67c607dde2a1d04e48)
    (cherry picked from commit d8fbf04f325f593836f8d44b6bbf42b85bde94e3)
    (cherry picked from commit 08f1f914cc219cf526adfb08c46b8f40b4e78232)
    (cherry picked from commit 366515dcd1090ca2f9f303009c78394b5665ce1f)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova pike-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova queens-eol release.