Noticed here:
http://logs.openstack.org/08/445308/3/check/gate-tempest-dsvm-py35-ubuntu-xenial/7bf0d72/logs/screen-n-api.txt.gz#_2017-03-16_05_31_09_399
I noticed this while investigating public nova bug 1673375, but it looks like that bug is caused by a ValueError coming from the oslo.messaging notification code, related to a circular reference in the json blob:
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging Traceback (most recent call last): 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/notify/messaging.py", line 70, in notify 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging retry=retry) 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/transport.py", line 104, in _send_notification 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging retry=retry) 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 509, in send_notification 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging envelope=(version == 2.0), notify=True, retry=retry) 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 457, in _send 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging msg = rpc_common.serialize_msg(msg) 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/common.py", line 293, in serialize_msg 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging _MESSAGE_KEY: jsonutils.dumps(raw_msg)} 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_serialization/jsonutils.py", line 190, in dumps 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging return json.dumps(obj, default=default, **kwargs) 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/lib/python3.5/json/__init__.py", line 237, in dumps 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging **kw).encode(obj) 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/lib/python3.5/json/encoder.py", line 198, in encode 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging chunks = self.iterencode(o, _one_shot=True) 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging return _iterencode(o, 0) 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging ValueError: Circular reference detected 2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging
The security issue here is that the notification payload that's logged has all kinds of auth secrets in it, like tokens and passwords.
From logstash it looks like this is only hitting master (pike) right now.
Noticed here:
http:// logs.openstack. org/08/ 445308/ 3/check/ gate-tempest- dsvm-py35- ubuntu- xenial/ 7bf0d72/ logs/screen- n-api.txt. gz#_2017- 03-16_05_ 31_09_399
I noticed this while investigating public nova bug 1673375, but it looks like that bug is caused by a ValueError coming from the oslo.messaging notification code, related to a circular reference in the json blob:
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging. notify. messaging Traceback (most recent call last): notify. messaging File "/usr/local/ lib/python3. 5/dist- packages/ oslo_messaging/ notify/ messaging. py", line 70, in notify notify. messaging retry=retry) notify. messaging File "/usr/local/ lib/python3. 5/dist- packages/ oslo_messaging/ transport. py", line 104, in _send_notification notify. messaging retry=retry) notify. messaging File "/usr/local/ lib/python3. 5/dist- packages/ oslo_messaging/ _drivers/ amqpdriver. py", line 509, in send_notification notify. messaging envelope=(version == 2.0), notify=True, retry=retry) notify. messaging File "/usr/local/ lib/python3. 5/dist- packages/ oslo_messaging/ _drivers/ amqpdriver. py", line 457, in _send notify. messaging msg = rpc_common. serialize_ msg(msg) notify. messaging File "/usr/local/ lib/python3. 5/dist- packages/ oslo_messaging/ _drivers/ common. py", line 293, in serialize_msg notify. messaging _MESSAGE_KEY: jsonutils. dumps(raw_ msg)} notify. messaging File "/usr/local/ lib/python3. 5/dist- packages/ oslo_serializat ion/jsonutils. py", line 190, in dumps notify. messaging return json.dumps(obj, default=default, **kwargs) notify. messaging File "/usr/lib/ python3. 5/json/ __init_ _.py", line 237, in dumps notify. messaging **kw).encode(obj) notify. messaging File "/usr/lib/ python3. 5/json/ encoder. py", line 198, in encode notify. messaging chunks = self.iterencode(o, _one_shot=True) notify. messaging File "/usr/lib/ python3. 5/json/ encoder. py", line 256, in iterencode notify. messaging return _iterencode(o, 0) notify. messaging ValueError: Circular reference detected notify. messaging
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.
The security issue here is that the notification payload that's logged has all kinds of auth secrets in it, like tokens and passwords.
From logstash it looks like this is only hitting master (pike) right now.