The following was observed in the OpenStack Ansible CI when use_json was used in oslo.log, it looks like it passes out all the items serialized which can cause credentials to be leaked:
http://logs.openstack.org/61/591961/2/check/openstack-ansible-functional-ubuntu-xenial/f1781ce/logs/host/cinder-volume.service.journal.log.txt.gz#_Aug_15_11_10_32
[in case it gets deleted]
Aug 15 11:10:33 ubuntu-xenial-ovh-gra1-0001333882 cinder-volume[23478]: {"thread_name": "MainThread", "extra": {"project": "unknown", "version": "unknown"}, "process": 23478, "relative_created": 2235828.4900188446, "module": "impl_rabbit", "message": "[31818cce-51f4-402c-ad62-f3674460d470] Reconnected to AMQP server on 10.1.1.101:5672 via [amqp] client with port 54050.", "hostname": "ubuntu-xenial-ovh-gra1-0001333882", "filename": "impl_rabbit.py", "levelno": 20, "lineno": 778, "asctime": "2018-08-15 11:10:33", "msg": "[%(connection_id)s] Reconnected to AMQP server on %(hostname)s:%(port)s via [%(transport)s] client with port %(client_port)s.", "error_summary": "", "args": {"hostname": "10.1.1.101", "userid": "cinder", "password": "secrete", "virtual_host": "/cinder", "port": 5672, "insist": false, "ssl": false, "transport": "amqp", "connect_timeout": 5, "transport_options": {"on_blocked": "<function _on_connection_blocked at 0x7fb2fc3282a8>", "on_unblocked": "<function _on_connection_unblocked at 0x7fb2fc328320>", "client_properties": {"connection_name": "cinder-volume:23478:31818cce-51f4-402c-ad62-f3674460d470", "capabilities": {"connection.blocked": true, "authentication_failure_close": true, "consumer_cancel_notify": true}}, "confirm_publish": true}, "login_method": "AMQPLAIN", "uri_prefix": null, "heartbeat": 60.0, "failover_strategy": "round-robin", "alternates": [], "client_port": 54050, "connection_id": "31818cce-51f4-402c-ad62-f3674460d470"}, "process_name": "MainProcess", "name": "oslo.messaging._drivers.impl_rabbit", "thread": 140406697963024, "created": 1534331433.086563, "traceback": null, "msecs": 86.5631103515625, "funcname": "on_reconnection", "pathname": "/openstack/venvs/cinder-testing/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", "context": {}, "levelname": "INFO"}
Aug 15 11:10:33 ubuntu-xenial-ovh-gra1-0001333882 cinder-volume[23478]: {"thread_name": "MainThread", "extra": {"project": "unknown", "version": "unknown"}, "process": 23478, "relative_created": 2235829.628944397, "module": "impl_rabbit", "message": "[b956eec0-15bb-4070-8a23-fd0f3c5e5a8e] Reconnected to AMQP server on 10.1.1.101:5672 via [amqp] client with port 54048.", "hostname": "ubuntu-xenial-ovh-gra1-0001333882", "filename": "impl_rabbit.py", "levelno": 20, "lineno": 778, "asctime": "2018-08-15 11:10:33", "msg": "[%(connection_id)s] Reconnected to AMQP server on %(hostname)s:%(port)s via [%(transport)s] client with port %(client_port)s.", "error_summary": "", "args": {"hostname": "10.1.1.101", "userid": "cinder", "password": "secrete", "virtual_host": "/cinder", "port": 5672, "insist": false, "ssl": false, "transport": "amqp", "connect_timeout": 5, "transport_options": {"on_blocked": "<function _on_connection_blocked at 0x7fb2fc3282a8>", "on_unblocked": "<function _on_connection_unblocked at 0x7fb2fc328320>", "client_properties": {"connection_name": "cinder-volume:23478:b956eec0-15bb-4070-8a23-fd0f3c5e5a8e", "capabilities": {"connection.blocked": true, "authentication_failure_close": true, "consumer_cancel_notify": true}}, "confirm_publish": true}, "login_method": "AMQPLAIN", "uri_prefix": null, "heartbeat": 60.0, "failover_strategy": "round-robin", "alternates": [], "client_port": 54048, "connection_id": "b956eec0-15bb-4070-8a23-fd0f3c5e5a8e"}, "process_name": "MainProcess", "name": "oslo.messaging._drivers.impl_rabbit", "thread": 140406697962544, "created": 1534331433.087702, "traceback": null, "msecs": 87.70203590393066, "funcname": "on_reconnection", "pathname": "/openstack/venvs/cinder-testing/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", "context": {}, "levelname": "INFO"}
Aug 15 11:10:33 ubuntu-xenial-ovh-gra1-0001333882 cinder-volume[23478]: {"thread_name": "GreenThread-2", "extra": {"project": null, "version": "unknown"}, "process": 23478, "relative_created": 2235847.9709625244, "module": "impl_rabbit", "message": "[a2a29ff1-a431-4db5-9d05-f3ffb68d431d] AMQP server on 10.1.1.101:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.", "hostname": "ubuntu-xenial-ovh-gra1-0001333882", "filename": "impl_rabbit.py", "levelno": 40, "lineno": 751, "asctime": "2018-08-15 11:10:33", "msg": "[%(connection_id)s] AMQP server on %(hostname)s:%(port)s is unreachable: %(err_str)s. Trying again in %(sleep_time)d seconds.", "error_summary": "error: [Errno 32] Broken pipe", "args": {"transport_options": {"on_blocked": "<function _on_connection_blocked at 0x7fb2fc3282a8>", "on_unblocked": "<function _on_connection_unblocked at 0x7fb2fc328320>", "client_properties": {"connection_name": "cinder-volume:23478:a2a29ff1-a431-4db5-9d05-f3ffb68d431d", "capabilities": {"connection.blocked": true, "authentication_failure_close": true, "consumer_cancel_notify": true}}, "confirm_publish": true}, "failover_strategy": "round-robin", "connection_id": "a2a29ff1-a431-4db5-9d05-f3ffb68d431d", "insist": false, "ssl": false, "client_port": null, "password": "secrete", "port": 5672, "transport": "amqp", "alternates": [], "err_str": "error(32, 'Broken pipe')", "login_method": "AMQPLAIN", "hostname": "10.1.1.101", "userid": "cinder", "connect_timeout": 5, "virtual_host": "/cinder", "heartbeat": 60.0, "uri_prefix": null, "sleep_time": 1.0}, "process_name": "MainProcess", "name": "oslo.messaging._drivers.impl_rabbit", "thread": 140406697962864, "created": 1534331433.106044, "traceback": null, "msecs": 106.04405403137207, "funcname": "on_error", "pathname": "/openstack/venvs/cinder-testing/local/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", "context": {"domain": null, "project_name": null, "global_request_id": null, "project_domain": null, "timestamp": "2018-08-15T10:33:31.089453", "user_domain_name": null, "remote_address": null, "quota_class": null, "resource_uuid": null, "is_admin": true, "user": null, "service_catalog": [], "domain_id": null, "tenant": null, "read_only": false, "user_domain": null, "user_id": null, "show_deleted": false, "system_scope": null, "user_identity": "- - - - -", "domain_name": null, "is_admin_project": true, "project": null, "read_deleted": "no", "request_id": "req-e0d7fde8-a2cb-47a8-b12f-725a880d7a83", "roles": ["admin"], "project_id": null, "user_name": null, "auth_token": null, "project_domain_name": null}, "levelname": "ERROR"}
It looks like it is all happening here:
https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L739-L751
More specifically, getting it from this function:
https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L740
going up the stack..
https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L1143-L1156
going further up where self.connection is defined
https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L563-L580
going all the way up to kombu
https://github.com/celery/kombu/blob/master/kombu/connection.py#L625-L627
and this is where the leaked data comes from..
https://github.com/celery/kombu/blob/master/kombu/connection.py#L595-L623
FWIW, oslo.log started including the JSON logger in 3.33.0
https:/ /github. com/openstack/ oslo.log/ commit/ 215cc3a8ec5cb61 a365a98d731d08a e2f13d46d1
This has been introduced in the Queens release.