swiftclient disclose token in debug logs

[George Shuklin]

Setup: juno. Nova, glance + swiftclient.

glance-api.conf (important parts):

[DEFAULT]
debug = true
logging_context_format_string=%(name)s[%(process)d]: %(levelname)s %(instance)s%(message)s [%(request_id)s %(user)s %(tenant)s]
logging_default_format_string=%(name)s[%(process)d]: %(levelname)s %(instance)s%(message)s
logging_debug_format_suffix=%(funcName)s %(pathname)s:%(lineno)d
logging_exception_prefix=%(name)s[%(process)d]: TRACE %(instance)s
default_store = swift
use_syslog = True
syslog_log_facility = LOG_LOCAL2
swift_store_auth_address = https://my.hand.disclosing.corporte.url:5000/v2.0
swift_store_user = tenant:user
swift_store_key = sexgodqwerty123456love

Result in remote syslog:

DEBUG REQ: curl -i https://my.hand.disclosing.corporte.url:8080/v1/OMG_47e02d5a461148ef9f9dab62ea0ba64b/region/6a66d8dc-5748-4cb5-9db5-b12ab0d1c698-00007 -X PUT -H "X-Auth-Token: 6f64276e2074726461650a6d" http_log /usr/lib/python2.7/dist-packages/swiftclient/client.py:95

Versions:

ii python-swift 2.2.0-0ubuntu1~cloud0 all distributed virtual object store - Python libraries
ii python-swiftclient 1:2.3.0-0ubuntu1~cloud0 all Client library for Openstack Swift API.
ii glance-api 1:2014.2.3-0-ownbuild all OpenStack Image Registry and Delivery Service - API
ii glance-common 1:2014.2.3-ownbuild all OpenStack Image Registry and Delivery Service - Common
ii python-glance 1:2014.2.3-0ownbuild all OpenStack Image Registry and Delivery Service - Python library
ii python-glance-store 0.1.8-1ubuntu2~cloud0 all OpenStack Image Service store library - Python 2.x
ii python-glanceclient 1:0.14.0-0ubuntu1~cloud0 all Client library for Openstack glance server.

Impact:
1) Unprivileged employee with access to logging facility may get access to glance images, including snapshots of the tenants.
2) Syslog transmitted unencrypted in UDP or TCP and it may be viewed by unauthorized person.

Expected behavior:
Complete or partial token masking in logs, f.e.:

DEBUG REQ: curl -i https://my.hand.disclosing.corporte.url:8080/v1/OMG_47e02d5a461148ef9f9dab62ea0ba64b/region/6a66d8dc-5748-4cb5-9db5-b12ab0d1c698-00007 -X PUT -H "X-Auth-Token: 6****************d" http_log /usr/lib/python2.7/dist-packages/swiftclient/client.py:95

[Tristan Cacqueray (tristan-cacqueray)]

DEBUG logs leak are not considered as a vulnerability and it doesn't warrant a private bug status. Thus I'm marking this bug as public security. Thanks.

[Jeremy Stanley (fungi)]

Added the security tag to indicate a possible hardening opportunity.

[Travis McPeak (travis-mcpeak)]

So just to clarify, what we're basically saying is that logging credentials in DEBUG is not ideal but is also not a vulnerability?

If that's the case I'll propose a more general OSSN that essentially says "all confidentiality bets are off when you run services with log level debug".

[Tristan Cacqueray (tristan-cacqueray)]

That's it, DEBUG log aren't considered "secure". Seems like OSSN-0049 could cover all openstack services.

[George Shuklin (george-shuklin)]

I may agree that local logs with DEBUG is not a big deal, but if use_syslog=True enabled, than, yes, it can cause unexpected consequences.

For example, in our real-world installation I just wanted to see debug logs from glance for short time, and I didn't expected to disclose them to low-clearance support personnel, and this was suddenly a BIG issue for our security department.

I was forced to write down official explanation about accidental credential disclosure and perform in-house audit of all swift access logs to prove there were no attempts of unauthorized access to snapshots with sensitive data.

OSSN is not enough, because it can be necessary to enable debug for service (like glance).

Proposal: perform token masking only if logs are sent to syslog.

[Jeremy Stanley (fungi)]

As a general rule projects should try to mask/redact potentially sensitive information even for DEBUG-level logging, but there are still so many instances of this throughout OpenStack that we rely on recommendations like the red warning box you see at http://docs.openstack.org/developer/horizon/topics/deployment.html#logging to make sure deployers know that setting production service logging to DEBUG or sharing their logs of the same is potentially dangerous. If there is no general OSSN yet with similar recommendations, I agree it's a great addition.

So yes I expect this is a bug, and should be fixed to improve the overall security posture of our software, but fixing it won't elicit a security advisory from the vulnerability management team and may not get backported to older releases unless it can be done very, very cleanly to avoid adverse operational impact.

[Robert Clark (robert-clark)]

In review here: https://review.openstack.org/#/c/219903/

[Nathan Kinder (nkinder)]

This has been published as OSSN-00052:

  https://wiki.openstack.org/wiki/OSSN/OSSN-0052

[Tim Burke (1-tim-z)]

Fixed in https://github.com/openstack/python-swiftclient/commit/4d44dcf / https://review.openstack.org/#/c/282363/ ; see also https://bugs.launchpad.net/python-swiftclient/+bug/1516692