Comment 10 for bug 1787214

Revision history for this message
Ken Giusti (kgiusti) wrote : Re: [Bug 1787214] Re: Using use_json logging leaks credentials

Thanks Jeremy,

/me reads....

A fix is proposed as a patch to the current master branch (as well as any
> affected supported branches) and attached to the private bug report, not
> sent to the public code review system.
>

Aww crap....

On Wed, Aug 22, 2018 at 12:41 PM Jeremy Stanley <email address hidden> wrote:

> Ken: The process recommended by the OpenStack Vulnerability Management
> Team is documented at https://security.openstack.org/vmt-process.html
> (though since oslo.log and oslo.messaging haven't applied for VMT
> oversight they're not necessarily obliged to follow the timeline
> described there).
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1787214
>
> Title:
> Using use_json logging leaks credentials
>
> Status in oslo.log:
> Triaged
> Status in oslo.messaging:
> In Progress
>
> Bug description:
> The following was observed in the OpenStack Ansible CI when use_json
> was used in oslo.log, it looks like it passes out all the items
> serialized which can cause credentials to be leaked:
>
> http://logs.openstack.org/61/591961/2/check/openstack-ansible-
> functional-ubuntu-xenial/f1781ce/logs/host/cinder-
> volume.service.journal.log.txt.gz#_Aug_15_11_10_32
>
> [in case it gets deleted]
>
> Aug 15 11:10:33 ubuntu-xenial-ovh-gra1-0001333882 cinder-
> volume[23478]: {"thread_name": "MainThread", "extra": {"project":
> "unknown", "version": "unknown"}, "process": 23478,
> "relative_created": 2235828.4900188446, "module": "impl_rabbit",
> "message": "[31818cce-51f4-402c-ad62-f3674460d470] Reconnected to AMQP
> server on 10.1.1.101:5672 via [amqp] client with port 54050.",
> "hostname": "ubuntu-xenial-ovh-gra1-0001333882", "filename":
> "impl_rabbit.py", "levelno": 20, "lineno": 778, "asctime": "2018-08-15
> 11:10:33", "msg": "[%(connection_id)s] Reconnected to AMQP server on
> %(hostname)s:%(port)s via [%(transport)s] client with port
> %(client_port)s.", "error_summary": "", "args": {"hostname":
> "10.1.1.101", "userid": "cinder", "password": "secrete",
> "virtual_host": "/cinder", "port": 5672, "insist": false, "ssl":
> false, "transport": "amqp", "connect_timeout": 5, "transport_options":
> {"on_blocked": "<function _on_connection_blocked at 0x7fb2fc3282a8>",
> "on_unblocked": "<function _on_connection_unblocked at
> 0x7fb2fc328320>", "client_properties": {"connection_name": "cinder-
> volume:23478:31818cce-51f4-402c-ad62-f3674460d470", "capabilities":
> {"connection.blocked": true, "authentication_failure_close": true,
> "consumer_cancel_notify": true}}, "confirm_publish": true},
> "login_method": "AMQPLAIN", "uri_prefix": null, "heartbeat": 60.0,
> "failover_strategy": "round-robin", "alternates": [], "client_port":
> 54050, "connection_id": "31818cce-51f4-402c-ad62-f3674460d470"},
> "process_name": "MainProcess", "name":
> "oslo.messaging._drivers.impl_rabbit", "thread": 140406697963024,
> "created": 1534331433.086563, "traceback": null, "msecs":
> 86.5631103515625, "funcname": "on_reconnection", "pathname":
> "/openstack/venvs/cinder-testing/local/lib/python2.7/site-
> packages/oslo_messaging/_drivers/impl_rabbit.py", "context": {},
> "levelname": "INFO"}
>
> Aug 15 11:10:33 ubuntu-xenial-ovh-gra1-0001333882 cinder-
> volume[23478]: {"thread_name": "MainThread", "extra": {"project":
> "unknown", "version": "unknown"}, "process": 23478,
> "relative_created": 2235829.628944397, "module": "impl_rabbit",
> "message": "[b956eec0-15bb-4070-8a23-fd0f3c5e5a8e] Reconnected to AMQP
> server on 10.1.1.101:5672 via [amqp] client with port 54048.",
> "hostname": "ubuntu-xenial-ovh-gra1-0001333882", "filename":
> "impl_rabbit.py", "levelno": 20, "lineno": 778, "asctime": "2018-08-15
> 11:10:33", "msg": "[%(connection_id)s] Reconnected to AMQP server on
> %(hostname)s:%(port)s via [%(transport)s] client with port
> %(client_port)s.", "error_summary": "", "args": {"hostname":
> "10.1.1.101", "userid": "cinder", "password": "secrete",
> "virtual_host": "/cinder", "port": 5672, "insist": false, "ssl":
> false, "transport": "amqp", "connect_timeout": 5, "transport_options":
> {"on_blocked": "<function _on_connection_blocked at 0x7fb2fc3282a8>",
> "on_unblocked": "<function _on_connection_unblocked at
> 0x7fb2fc328320>", "client_properties": {"connection_name": "cinder-
> volume:23478:b956eec0-15bb-4070-8a23-fd0f3c5e5a8e", "capabilities":
> {"connection.blocked": true, "authentication_failure_close": true,
> "consumer_cancel_notify": true}}, "confirm_publish": true},
> "login_method": "AMQPLAIN", "uri_prefix": null, "heartbeat": 60.0,
> "failover_strategy": "round-robin", "alternates": [], "client_port":
> 54048, "connection_id": "b956eec0-15bb-4070-8a23-fd0f3c5e5a8e"},
> "process_name": "MainProcess", "name":
> "oslo.messaging._drivers.impl_rabbit", "thread": 140406697962544,
> "created": 1534331433.087702, "traceback": null, "msecs":
> 87.70203590393066, "funcname": "on_reconnection", "pathname":
> "/openstack/venvs/cinder-testing/local/lib/python2.7/site-
> packages/oslo_messaging/_drivers/impl_rabbit.py", "context": {},
> "levelname": "INFO"}
>
> Aug 15 11:10:33 ubuntu-xenial-ovh-gra1-0001333882 cinder-
> volume[23478]: {"thread_name": "GreenThread-2", "extra": {"project":
> null, "version": "unknown"}, "process": 23478, "relative_created":
> 2235847.9709625244, "module": "impl_rabbit", "message":
> "[a2a29ff1-a431-4db5-9d05-f3ffb68d431d] AMQP server on 10.1.1.101:5672
> is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.",
> "hostname": "ubuntu-xenial-ovh-gra1-0001333882", "filename":
> "impl_rabbit.py", "levelno": 40, "lineno": 751, "asctime": "2018-08-15
> 11:10:33", "msg": "[%(connection_id)s] AMQP server on
> %(hostname)s:%(port)s is unreachable: %(err_str)s. Trying again in
> %(sleep_time)d seconds.", "error_summary": "error: [Errno 32] Broken
> pipe", "args": {"transport_options": {"on_blocked": "<function
> _on_connection_blocked at 0x7fb2fc3282a8>", "on_unblocked": "<function
> _on_connection_unblocked at 0x7fb2fc328320>", "client_properties":
> {"connection_name": "cinder-
> volume:23478:a2a29ff1-a431-4db5-9d05-f3ffb68d431d", "capabilities":
> {"connection.blocked": true, "authentication_failure_close": true,
> "consumer_cancel_notify": true}}, "confirm_publish": true},
> "failover_strategy": "round-robin", "connection_id":
> "a2a29ff1-a431-4db5-9d05-f3ffb68d431d", "insist": false, "ssl": false,
> "client_port": null, "password": "secrete", "port": 5672, "transport":
> "amqp", "alternates": [], "err_str": "error(32, 'Broken pipe')",
> "login_method": "AMQPLAIN", "hostname": "10.1.1.101", "userid":
> "cinder", "connect_timeout": 5, "virtual_host": "/cinder",
> "heartbeat": 60.0, "uri_prefix": null, "sleep_time": 1.0},
> "process_name": "MainProcess", "name":
> "oslo.messaging._drivers.impl_rabbit", "thread": 140406697962864,
> "created": 1534331433.106044, "traceback": null, "msecs":
> 106.04405403137207, "funcname": "on_error", "pathname":
> "/openstack/venvs/cinder-testing/local/lib/python2.7/site-
> packages/oslo_messaging/_drivers/impl_rabbit.py", "context":
> {"domain": null, "project_name": null, "global_request_id": null,
> "project_domain": null, "timestamp": "2018-08-15T10:33:31.089453",
> "user_domain_name": null, "remote_address": null, "quota_class": null,
> "resource_uuid": null, "is_admin": true, "user": null,
> "service_catalog": [], "domain_id": null, "tenant": null, "read_only":
> false, "user_domain": null, "user_id": null, "show_deleted": false,
> "system_scope": null, "user_identity": "- - - - -", "domain_name":
> null, "is_admin_project": true, "project": null, "read_deleted": "no",
> "request_id": "req-e0d7fde8-a2cb-47a8-b12f-725a880d7a83", "roles":
> ["admin"], "project_id": null, "user_name": null, "auth_token": null,
> "project_domain_name": null}, "levelname": "ERROR"}
>
> It looks like it is all happening here:
>
>
> https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L739-L751
>
> More specifically, getting it from this function:
>
>
> https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L740
>
> going up the stack..
>
>
> https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L1143-L1156
>
> going further up where self.connection is defined
>
>
> https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L563-L580
>
> going all the way up to kombu
>
>
> https://github.com/celery/kombu/blob/master/kombu/connection.py#L625-L627
>
> and this is where the leaked data comes from..
>
>
> https://github.com/celery/kombu/blob/master/kombu/connection.py#L595-L623
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/oslo.log/+bug/1787214/+subscriptions
>

--
Ken Giusti (<email address hidden>)