Thanks Jeremy,

/me reads....

A fix is proposed as a patch to the current master branch (as well as any
> affected supported branches) and attached to the private bug report, not
> sent to the public code review system.
>

Aww crap....

On Wed, Aug 22, 2018 at 12:41 PM Jeremy Stanley <email address hidden> wrote:

> Ken: The process recommended by the OpenStack Vulnerability Management
> Team is documented at https://security.openstack.org/vmt-process.html
> (though since oslo.log and oslo.messaging haven't applied for VMT
> oversight they're not necessarily obliged to follow the timeline
> described there).
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1787214
>
> Title:
>   Using use_json logging leaks credentials
>
> Status in oslo.log:
>   Triaged
> Status in oslo.messaging:
>   In Progress
>
> Bug description:
>   The following was observed in the OpenStack Ansible CI when use_json
>   was used in oslo.log, it looks like it passes out all the items
>   serialized which can cause credentials to be leaked:
>
>   http://logs.openstack.org/61/591961/2/check/openstack-ansible-
>   functional-ubuntu-xenial/f1781ce/logs/host/cinder-
>   volume.service.journal.log.txt.gz#_Aug_15_11_10_32
>
>   [in case it gets deleted]
>
>   Aug 15 11:10:33 ubuntu-xenial-ovh-gra1-0001333882 cinder-
>   volume[23478]: {"thread_name": "MainThread", "extra": {"project":
>   "unknown", "version": "unknown"}, "process": 23478,
>   "relative_created": 2235828.4900188446, "module": "impl_rabbit",
>   "message": "[31818cce-51f4-402c-ad62-f3674460d470] Reconnected to AMQP
>   server on 10.1.1.101:5672 via [amqp] client with port 54050.",
>   "hostname": "ubuntu-xenial-ovh-gra1-0001333882", "filename":
>   "impl_rabbit.py", "levelno": 20, "lineno": 778, "asctime": "2018-08-15
>   11:10:33", "msg": "[%(connection_id)s] Reconnected to AMQP server on
>   %(hostname)s:%(port)s via [%(transport)s] client with port
>   %(client_port)s.", "error_summary": "", "args": {"hostname":
>   "10.1.1.101", "userid": "cinder", "password": "secrete",
>   "virtual_host": "/cinder", "port": 5672, "insist": false, "ssl":
>   false, "transport": "amqp", "connect_timeout": 5, "transport_options":
>   {"on_blocked": "<function _on_connection_blocked at 0x7fb2fc3282a8>",
>   "on_unblocked": "<function _on_connection_unblocked at
>   0x7fb2fc328320>", "client_properties": {"connection_name": "cinder-
>   volume:23478:31818cce-51f4-402c-ad62-f3674460d470", "capabilities":
>   {"connection.blocked": true, "authentication_failure_close": true,
>   "consumer_cancel_notify": true}}, "confirm_publish": true},
>   "login_method": "AMQPLAIN", "uri_prefix": null, "heartbeat": 60.0,
>   "failover_strategy": "round-robin", "alternates": [], "client_port":
>   54050, "connection_id": "31818cce-51f4-402c-ad62-f3674460d470"},
>   "process_name": "MainProcess", "name":
>   "oslo.messaging._drivers.impl_rabbit", "thread": 140406697963024,
>   "created": 1534331433.086563, "traceback": null, "msecs":
>   86.5631103515625, "funcname": "on_reconnection", "pathname":
>   "/openstack/venvs/cinder-testing/local/lib/python2.7/site-
>   packages/oslo_messaging/_drivers/impl_rabbit.py", "context": {},
>   "levelname": "INFO"}
>
>   Aug 15 11:10:33 ubuntu-xenial-ovh-gra1-0001333882 cinder-
>   volume[23478]: {"thread_name": "MainThread", "extra": {"project":
>   "unknown", "version": "unknown"}, "process": 23478,
>   "relative_created": 2235829.628944397, "module": "impl_rabbit",
>   "message": "[b956eec0-15bb-4070-8a23-fd0f3c5e5a8e] Reconnected to AMQP
>   server on 10.1.1.101:5672 via [amqp] client with port 54048.",
>   "hostname": "ubuntu-xenial-ovh-gra1-0001333882", "filename":
>   "impl_rabbit.py", "levelno": 20, "lineno": 778, "asctime": "2018-08-15
>   11:10:33", "msg": "[%(connection_id)s] Reconnected to AMQP server on
>   %(hostname)s:%(port)s via [%(transport)s] client with port
>   %(client_port)s.", "error_summary": "", "args": {"hostname":
>   "10.1.1.101", "userid": "cinder", "password": "secrete",
>   "virtual_host": "/cinder", "port": 5672, "insist": false, "ssl":
>   false, "transport": "amqp", "connect_timeout": 5, "transport_options":
>   {"on_blocked": "<function _on_connection_blocked at 0x7fb2fc3282a8>",
>   "on_unblocked": "<function _on_connection_unblocked at
>   0x7fb2fc328320>", "client_properties": {"connection_name": "cinder-
>   volume:23478:b956eec0-15bb-4070-8a23-fd0f3c5e5a8e", "capabilities":
>   {"connection.blocked": true, "authentication_failure_close": true,
>   "consumer_cancel_notify": true}}, "confirm_publish": true},
>   "login_method": "AMQPLAIN", "uri_prefix": null, "heartbeat": 60.0,
>   "failover_strategy": "round-robin", "alternates": [], "client_port":
>   54048, "connection_id": "b956eec0-15bb-4070-8a23-fd0f3c5e5a8e"},
>   "process_name": "MainProcess", "name":
>   "oslo.messaging._drivers.impl_rabbit", "thread": 140406697962544,
>   "created": 1534331433.087702, "traceback": null, "msecs":
>   87.70203590393066, "funcname": "on_reconnection", "pathname":
>   "/openstack/venvs/cinder-testing/local/lib/python2.7/site-
>   packages/oslo_messaging/_drivers/impl_rabbit.py", "context": {},
>   "levelname": "INFO"}
>
>   Aug 15 11:10:33 ubuntu-xenial-ovh-gra1-0001333882 cinder-
>   volume[23478]: {"thread_name": "GreenThread-2", "extra": {"project":
>   null, "version": "unknown"}, "process": 23478, "relative_created":
>   2235847.9709625244, "module": "impl_rabbit", "message":
>   "[a2a29ff1-a431-4db5-9d05-f3ffb68d431d] AMQP server on 10.1.1.101:5672
>   is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds.",
>   "hostname": "ubuntu-xenial-ovh-gra1-0001333882", "filename":
>   "impl_rabbit.py", "levelno": 40, "lineno": 751, "asctime": "2018-08-15
>   11:10:33", "msg": "[%(connection_id)s] AMQP server on
>   %(hostname)s:%(port)s is unreachable: %(err_str)s. Trying again in
>   %(sleep_time)d seconds.", "error_summary": "error: [Errno 32] Broken
>   pipe", "args": {"transport_options": {"on_blocked": "<function
>   _on_connection_blocked at 0x7fb2fc3282a8>", "on_unblocked": "<function
>   _on_connection_unblocked at 0x7fb2fc328320>", "client_properties":
>   {"connection_name": "cinder-
>   volume:23478:a2a29ff1-a431-4db5-9d05-f3ffb68d431d", "capabilities":
>   {"connection.blocked": true, "authentication_failure_close": true,
>   "consumer_cancel_notify": true}}, "confirm_publish": true},
>   "failover_strategy": "round-robin", "connection_id":
>   "a2a29ff1-a431-4db5-9d05-f3ffb68d431d", "insist": false, "ssl": false,
>   "client_port": null, "password": "secrete", "port": 5672, "transport":
>   "amqp", "alternates": [], "err_str": "error(32, 'Broken pipe')",
>   "login_method": "AMQPLAIN", "hostname": "10.1.1.101", "userid":
>   "cinder", "connect_timeout": 5, "virtual_host": "/cinder",
>   "heartbeat": 60.0, "uri_prefix": null, "sleep_time": 1.0},
>   "process_name": "MainProcess", "name":
>   "oslo.messaging._drivers.impl_rabbit", "thread": 140406697962864,
>   "created": 1534331433.106044, "traceback": null, "msecs":
>   106.04405403137207, "funcname": "on_error", "pathname":
>   "/openstack/venvs/cinder-testing/local/lib/python2.7/site-
>   packages/oslo_messaging/_drivers/impl_rabbit.py", "context":
>   {"domain": null, "project_name": null, "global_request_id": null,
>   "project_domain": null, "timestamp": "2018-08-15T10:33:31.089453",
>   "user_domain_name": null, "remote_address": null, "quota_class": null,
>   "resource_uuid": null, "is_admin": true, "user": null,
>   "service_catalog": [], "domain_id": null, "tenant": null, "read_only":
>   false, "user_domain": null, "user_id": null, "show_deleted": false,
>   "system_scope": null, "user_identity": "- - - - -", "domain_name":
>   null, "is_admin_project": true, "project": null, "read_deleted": "no",
>   "request_id": "req-e0d7fde8-a2cb-47a8-b12f-725a880d7a83", "roles":
>   ["admin"], "project_id": null, "user_name": null, "auth_token": null,
>   "project_domain_name": null}, "levelname": "ERROR"}
>
>   It looks like it is all happening here:
>
>
> https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L739-L751
>
>   More specifically, getting it from this function:
>
>
> https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L740
>
>   going up the stack..
>
>
> https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L1143-L1156
>
>   going further up where self.connection is defined
>
>
> https://github.com/openstack/oslo.messaging/blob/master/oslo_messaging/_drivers/impl_rabbit.py#L563-L580
>
>   going all the way up to kombu
>
>
> https://github.com/celery/kombu/blob/master/kombu/connection.py#L625-L627
>
>   and this is where the leaked data comes from..
>
>
> https://github.com/celery/kombu/blob/master/kombu/connection.py#L595-L623
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/oslo.log/+bug/1787214/+subscriptions
>


-- 
Ken Giusti  (<email address hidden>)