[OSSA-2017-002] Failed notification payload is dumped in logs with auth secrets (CVE-2017-7214)

[Matt Riedemann]

Noticed here:

http://logs.openstack.org/08/445308/3/check/gate-tempest-dsvm-py35-ubuntu-xenial/7bf0d72/logs/screen-n-api.txt.gz#_2017-03-16_05_31_09_399

I noticed this while investigating public nova bug 1673375, but it looks like that bug is caused by a ValueError coming from the oslo.messaging notification code, related to a circular reference in the json blob:

2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging Traceback (most recent call last):
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/notify/messaging.py", line 70, in notify
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging retry=retry)
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/transport.py", line 104, in _send_notification
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging retry=retry)
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 509, in send_notification
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging envelope=(version == 2.0), notify=True, retry=retry)
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 457, in _send
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging msg = rpc_common.serialize_msg(msg)
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_messaging/_drivers/common.py", line 293, in serialize_msg
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging _MESSAGE_KEY: jsonutils.dumps(raw_msg)}
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/local/lib/python3.5/dist-packages/oslo_serialization/jsonutils.py", line 190, in dumps
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging return json.dumps(obj, default=default, **kwargs)
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/lib/python3.5/json/__init__.py", line 237, in dumps
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging **kw).encode(obj)
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/lib/python3.5/json/encoder.py", line 198, in encode
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging chunks = self.iterencode(o, _one_shot=True)
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging File "/usr/lib/python3.5/json/encoder.py", line 256, in iterencode
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging return _iterencode(o, 0)
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging ValueError: Circular reference detected
2017-03-16 05:31:09.399 23355 ERROR oslo_messaging.notify.messaging

The security issue here is that the notification payload that's logged has all kinds of auth secrets in it, like tokens and passwords.

From logstash it looks like this is only hitting master (pike) right now.

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

If it can be confirmed for certain that the commit introducing this behavior is master-branch-only impacting then we can drop the embargo and forgo the advisory (Class Y report).

[Ben Nemec (bnemec)]

Hmm, so the problem is that the notifier is hooked directly into the logger, and when it sends the notification it tries to include all of the args to the logger. In this case that includes the Nova context, which contains sensitive data.

The circular reference error aside, it seems bad that we're sending sensitive data in notification payloads. That isn't considered a secure channel, is it?

Maybe we could filter the context out in the notification log handler. I don't think that information is intended to be exposed, except where it is referenced in the log message anyway.

[Matt Riedemann (mriedem)]

I'm pretty sure notifications aren't considered a secure channel. If you're sending over rpc you should have ssl configured. But it's not required, and I remember long ago when the IBM product I was working on was using QEMU we were getting internal security audit flags because QEMU was logging credentials that were sent over the wire.

[Matt Riedemann (mriedem)]

We have a fix here:

https://review.openstack.org/#/c/446948/

[Matt Riedemann (mriedem)]

Looks like the original change to strip the context out of the notification payload happened in Grizzly:

https://github.com/openstack/nova/commit/d1f121265badfffd097ee983d14989b885375a0e

And regressed in Mitaka:

https://github.com/openstack/nova/commit/2910d75b28afd909af3b4ac392729ac3d5e64b65

[Matt Riedemann (mriedem)]

The actual Tempest aggregate negative tests are very old:

https://github.com/openstack/tempest/commit/bf2d51751d778eb7a1b9ddfe1901f15c4cf56192

So why things started getting logged recently I'm still not sure, but my guess would be something changed in oslo.messaging to be logging on failures.

[Matt Riedemann (mriedem)]

https://review.openstack.org/#/c/309679/ in oslo.messaging might have started logging the failure, but that's been around since Newton so it's very odd that we'd just start noticing this now.

[Matt Riedemann (mriedem)]

All of the backports are up too:

https://review.openstack.org/#/q/I1d217620e52d45595a3e0e49ed57b4ab33cd1688,n,z

[Ben Nemec (bnemec)]

Maybe there was a change to the context that caused the circular reference error? I see a comment added in https://github.com/openstack/oslo.context/commit/f25543fcc792ebf155728a91fde06e8dc4e96cea that refers to avoiding circular references, but maybe it's not working in this case.

That didn't merge until January, so it's possible nobody noticed until now.

[Jeremy Stanley (fungi)]

It looks like the backports are starting to get approved and merge now as well. Once they're all merged, I propose switching this bug to Public Security and following our lighter-weight our public report disclosure process from that point instead of continuing to maintain an embargo (the risk posed by this bug seems pretty minimal).

Proposed impact description:

Title: Nova logs sensitive context from notification exceptions
Reporter: Matt Riedemann (Huawei)
Products: Nova
Affects: >=13.0.0 <=13.1.3, >=14.0.0 <=14.0.4, >=15.0.0 <=15.0.1

Description:
Matt Riedemann with Huawei reported a vulnerability in Nova. Legacy notification exception contexts appearing in ERROR level logs may include sensitive information such as account passwords and authorization tokens. All Nova setups are affected.

[Morgan Fainberg (mdrnstm)]

impact statement looks good to me and +1 making it public for the reasons fungi described.

[Matt Riedemann (mriedem)]

Impact statement looks good to me too.

[Matt Riedemann (mriedem)]

The nova fixes are merged on all branches for this now. Can we make it public security and then I can call it out in the release notes for the nova changes on stable and do a release?

[Jeremy Stanley (fungi)]

Thanks Matt, I've switched this to Public Security and am moving forward with the CVE assignment request using the impact statement from comment #10.

[Jeremy Stanley (fungi)]

Removed unneeded security tag, as the bug type is already security (we use the former for hardening opportunities and the latter for vulnerabilities).

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/448295

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/448297

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/448298

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/448299

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/448295
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3f985f1eda6f29180878a3d21c20c5057179486a
Submitter: Jenkins
Branch: master

commit 3f985f1eda6f29180878a3d21c20c5057179486a
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 17:54:43 2017 -0400

    Add release note for CVE-2017-7214

    Change-Id: I1f5b772bff8fa2dc508c4bdd3dd1f32838594786
    Related-Bug: #1673569

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/448297
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=acb19160d4d348e29a21ad57c61c7369352c4d1c
Submitter: Jenkins
Branch: stable/ocata

commit acb19160d4d348e29a21ad57c61c7369352c4d1c
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 17:54:43 2017 -0400

    Add release note for CVE-2017-7214

    Change-Id: I1f5b772bff8fa2dc508c4bdd3dd1f32838594786
    Related-Bug: #1673569
    (cherry picked from commit 3f985f1eda6f29180878a3d21c20c5057179486a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/448298
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c2c91ce44592fc5dc2aacee1cf7f5b5cfd2e9a0a
Submitter: Jenkins
Branch: stable/newton

commit c2c91ce44592fc5dc2aacee1cf7f5b5cfd2e9a0a
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 17:54:43 2017 -0400

    Add release note for CVE-2017-7214

    Change-Id: I1f5b772bff8fa2dc508c4bdd3dd1f32838594786
    Related-Bug: #1673569
    (cherry picked from commit 3f985f1eda6f29180878a3d21c20c5057179486a)
    (cherry picked from commit acb19160d4d348e29a21ad57c61c7369352c4d1c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/448299
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e193201fa1de5b08b29adefd8c149935c5529598
Submitter: Jenkins
Branch: stable/mitaka

commit e193201fa1de5b08b29adefd8c149935c5529598
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 17:54:43 2017 -0400

    Add release note for CVE-2017-7214

    Change-Id: I1f5b772bff8fa2dc508c4bdd3dd1f32838594786
    Related-Bug: #1673569
    (cherry picked from commit 3f985f1eda6f29180878a3d21c20c5057179486a)
    (cherry picked from commit acb19160d4d348e29a21ad57c61c7369352c4d1c)
    (cherry picked from commit c2c91ce44592fc5dc2aacee1cf7f5b5cfd2e9a0a)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/448610
Committed: https://git.openstack.org/cgit/openstack/ossa/commit/?id=c54ed705df6588194d134abe1762aae5e0e5a39b
Submitter: Jenkins
Branch: master

commit c54ed705df6588194d134abe1762aae5e0e5a39b
Author: Jeremy Stanley <email address hidden>
Date: Wed Mar 22 14:23:58 2017 +0000

    OSSA-2017-002 (CVE-2017-7214)

    Nova logs sensitive context from notification exceptions

    Change-Id: Iec1deae6bbe7fc73045c2abf9b3d44bafa86acc0
    Closes-Bug: #1673569

[Brian Murray (brian-murray)]

Hello Matt, or anyone else affected,

Accepted nova into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:14.0.5-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[James Page (james-page)]

Hello Matt, or anyone else affected,

Accepted nova into newton-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:newton-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-newton-needed to verification-newton-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-newton-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

SRU regression testing has completed successfully for yakkety-newton and xenial-newton. Tempest results are below:

yakkety-newton-proposed
-----------------------

======
Totals
======
Ran: 103 tests in 972.3608 sec.
 - Passed: 97
 - Skipped: 6
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 554.7547 sec.

==============
Worker Balance
==============
 - Worker 0 (103 tests) => 0:15:59.401528
___________________________________ summary ____________________________________
  smoke: commands succeeded
  congratulations :)

xenial-newton-proposed
----------------------

======
Totals
======
Ran: 103 tests in 1185.2261 sec.
 - Passed: 97
 - Skipped: 6
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 650.9384 sec.

==============
Worker Balance
==============
 - Worker 0 (103 tests) => 0:19:27.568502
___________________________________ summary ____________________________________
  smoke: commands succeeded
  congratulations :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package nova - 2:14.0.5-0ubuntu1

---------------
nova (2:14.0.5-0ubuntu1) yakkety; urgency=medium

  [Saverio Proto]
  * New upstream point release for OpenStack Newton (LP: #1688557).

  [Corey Bryant]
  * SECURITY UPDATE: Failed notification payload is dumped in logs
    with auth secrets (LP: #1673569).
    - This is included from upstream in the 14.0.5 stable point release.
    - CVE-2017-7214

 -- Corey Bryant <email address hidden> Mon, 15 May 2017 08:37:32 -0400

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[James Page (james-page)]

This bug was fixed in the package nova - 2:14.0.5-0ubuntu1~cloud0
---------------

 nova (2:14.0.5-0ubuntu1~cloud0) xenial-newton; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 nova (2:14.0.5-0ubuntu1) yakkety; urgency=medium
 .
   [Saverio Proto]
   * New upstream point release for OpenStack Newton (LP: #1688557).
 .
   [Corey Bryant]
   * SECURITY UPDATE: Failed notification payload is dumped in logs
     with auth secrets (LP: #1673569).
     - This is included from upstream in the 14.0.5 stable point release.
     - CVE-2017-7214