Cinder logs rabbitmq password on connection log

Bug #1750074 reported by Marga Millet on 2018-02-16
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Undecided
Marga Millet
Manila
Critical
Dustin Schoenbrun
OpenStack Security Advisory
Undecided
Unassigned

Bug Description

Cinder may log rabbitmq password on connection when DEBUG is on.

Example on cinder-scheduler.log file after enabling DEBUG:
(Password has been replaced with XXX)

2018-02-05 19:21:52.721 35 DEBUG cinder.service [req-a2dbe0dd-14c9-4123-a69a-3623e5f0a4d7 - - - - -] transport_url : rabbit://guest:XXX@10.10.10.1:5672,guest:XXX@10.10.10.2:5672,guest:XXX@10.10.10.3:5672 wait /usr/lib/python2.7/site-packages/cinder/service.py:611

In a production environment, this is pretty bad.

Marga Millet (millet) on 2018-02-16
Changed in cinder:
assignee: nobody → Marga Millet (millet)
Marga Millet (millet) wrote :

This applies to all releases.

Fix proposed to branch: master
Review: https://review.openstack.org/545486

Changed in cinder:
status: New → In Progress

Reviewed: https://review.openstack.org/545486
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=919dd16a35ef2d0fd2ee4911398ba7830c02c6fd
Submitter: Zuul
Branch: master

commit 919dd16a35ef2d0fd2ee4911398ba7830c02c6fd
Author: Marga Millet <email address hidden>
Date: Fri Feb 16 23:05:47 2018 +0000

    Cinder logs rabbitmq password on connection log

    Cinder displays rabbitmq password if debug is enabled.

    Closes-Bug: 1750074
    Change-Id: I117319ac12991e4b46170fe71d18a65ea4c98556

Changed in cinder:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/545622
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ca9bf2d1c56605a565ffc468bb91fb5d58ef5ce9
Submitter: Zuul
Branch: stable/queens

commit ca9bf2d1c56605a565ffc468bb91fb5d58ef5ce9
Author: Marga Millet <email address hidden>
Date: Fri Feb 16 23:05:47 2018 +0000

    Cinder logs rabbitmq password on connection log

    Cinder displays rabbitmq password if debug is enabled.

    Closes-Bug: 1750074
    Change-Id: I117319ac12991e4b46170fe71d18a65ea4c98556
    (cherry picked from commit 919dd16a35ef2d0fd2ee4911398ba7830c02c6fd)

tags: added: in-stable-queens
Changed in manila:
assignee: nobody → Dustin Schoenbrun (dschoenb)

Fix proposed to branch: master
Review: https://review.openstack.org/546673

Changed in manila:
status: New → In Progress
Changed in manila:
importance: Undecided → Critical
Eric Harney (eharney) wrote :

I don't think this fix is adequate:
https://review.openstack.org/#/c/545486/

The code in cinder/service.py fails to account for a bunch of other options, such as

transport_url=qpid://...
sql_connection=mysql+pymysql://...

This needs to be leveraging the secret flag for options, like this code does:
https://git.openstack.org/cgit/openstack/oslo.config/tree/oslo_config/cfg.py?h=5.2.0#n2865

Reviewed: https://review.openstack.org/546673
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=05e4f14ea1fbc7ce61c8f57a082080d09c63e357
Submitter: Zuul
Branch: master

commit 05e4f14ea1fbc7ce61c8f57a082080d09c63e357
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 10:53:41 2018 -0500

    Fix manila logging rabbitmq password in debug mode

    Manila will display the rabbitmq password if debugging is enabled.
    This patch will ensure that the rabbitmq password is no longer
    displayed in the connection log for Manila when debugging is
    enabled by looking for the rabbitmq key and not printing it.

    There should likely be an effort to utilize Oslo's secret flag for
    options to truly fix this issue for this and other sensitive options.

    Change-Id: I97cc88354d9b54057350c70c4742055197540d1a
    Closes-Bug: 1750074

Changed in manila:
status: In Progress → Fix Released
tags: added: in-stable-pike

Reviewed: https://review.openstack.org/545621
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d51fabb497df2606f04f90cbd2e842c7692bb1af
Submitter: Zuul
Branch: stable/pike

commit d51fabb497df2606f04f90cbd2e842c7692bb1af
Author: Marga Millet <email address hidden>
Date: Fri Feb 16 23:05:47 2018 +0000

    Cinder logs rabbitmq password on connection log

    Cinder displays rabbitmq password if debug is enabled.

    Closes-Bug: 1750074
    Change-Id: I117319ac12991e4b46170fe71d18a65ea4c98556
    (cherry picked from commit 919dd16a35ef2d0fd2ee4911398ba7830c02c6fd)
    (cherry picked from commit ca9bf2d1c56605a565ffc468bb91fb5d58ef5ce9)

This issue was fixed in the openstack/cinder 12.0.0.0rc2 release candidate.

Reviewed: https://review.openstack.org/546755
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=be8580b350c34c91494cb553eb15e6b61ef250b7
Submitter: Zuul
Branch: stable/queens

commit be8580b350c34c91494cb553eb15e6b61ef250b7
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 10:53:41 2018 -0500

    Fix manila logging rabbitmq password in debug mode

    Manila will display the rabbitmq password if debugging is enabled.
    This patch will ensure that the rabbitmq password is no longer
    displayed in the connection log for Manila when debugging is
    enabled by looking for the rabbitmq key and not printing it.

    There should likely be an effort to utilize Oslo's secret flag for
    options to truly fix this issue for this and other sensitive options.

    Change-Id: I97cc88354d9b54057350c70c4742055197540d1a
    Closes-Bug: 1750074
    (cherry picked from commit 05e4f14ea1fbc7ce61c8f57a082080d09c63e357)

This issue was fixed in the openstack/manila 6.0.0.0rc3 release candidate.

Dirk Mueller (dmllr) wrote :

This patch to cinder broke cinder:

2018-02-22 17:56:30.587 32083 ERROR cinder Traceback (most recent call last):
2018-02-22 17:56:30.587 32083 ERROR cinder File "/usr/bin/cinder-scheduler", line 10, in <module>
2018-02-22 17:56:30.587 32083 ERROR cinder sys.exit(main())
2018-02-22 17:56:30.587 32083 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/cmd/scheduler.py", line 56, in main
2018-02-22 17:56:30.587 32083 ERROR cinder service.wait()
2018-02-22 17:56:30.587 32083 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/service.py", line 624, in wait
2018-02-22 17:56:30.587 32083 ERROR cinder (flag == "transport_url" and "rabbit:" in flag_get) or
2018-02-22 17:56:30.587 32083 ERROR cinder TypeError: argument of type 'NoneType' is not iterable
2018-02-22 17:56:30.587 32083 ERROR cinder

Reviewed: https://review.openstack.org/546754
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f4e2efb58dd805c2b970c3705535ee27c37a84c5
Submitter: Zuul
Branch: master

commit f4e2efb58dd805c2b970c3705535ee27c37a84c5
Author: Eric Harney <email address hidden>
Date: Wed Feb 21 14:27:11 2018 -0500

    Log config options with oslo.config

    This removes some custom Cinder code which
    handles filtering secret config options in a flaky way.

    Filtering will now be based on the "secret=True" option
    flag.

    Related-Bug: #1750074
    Change-Id: I1c404b057d1471c85bd7eaf5c096f5912293460a

Reviewed: https://review.openstack.org/547229
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=7d278042c5280e40d5ed68f504f45ef023f05e18
Submitter: Zuul
Branch: stable/queens

commit 7d278042c5280e40d5ed68f504f45ef023f05e18
Author: Eric Harney <email address hidden>
Date: Wed Feb 21 14:27:11 2018 -0500

    Log config options with oslo.config

    This removes some custom Cinder code which
    handles filtering secret config options in a flaky way.

    Filtering will now be based on the "secret=True" option
    flag.

    Related-Bug: #1750074
    Change-Id: I1c404b057d1471c85bd7eaf5c096f5912293460a

Reviewed: https://review.openstack.org/549596
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=c0f1af11b102eabdf7b21d79c80eb8945f03dee8
Submitter: Zuul
Branch: stable/pike

commit c0f1af11b102eabdf7b21d79c80eb8945f03dee8
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 10:53:41 2018 -0500

    Fix manila logging rabbitmq password in debug mode

    Manila will display the rabbitmq password if debugging is enabled.
    This patch will ensure that the rabbitmq password is no longer
    displayed in the connection log for Manila when debugging is
    enabled by looking for the rabbitmq key and not printing it.

    There should likely be an effort to utilize Oslo's secret flag for
    options to truly fix this issue for this and other sensitive options.

    Change-Id: I97cc88354d9b54057350c70c4742055197540d1a
    Closes-Bug: 1750074
    (cherry picked from commit 05e4f14ea1fbc7ce61c8f57a082080d09c63e357)
    (cherry picked from commit be8580b350c34c91494cb553eb15e6b61ef250b7)

Eric Harney (eharney) on 2018-03-05
information type: Public → Public Security
Jeremy Stanley (fungi) wrote :

I'm marking the advisory task won't fix and triaging this as a potential security hardening opportunity. In the past we've considered information leaking in DEBUG level logs to fit the B3 classification (a vulnerability in experimental or debugging features not intended for production use) in our report taxonomy: https://security.openstack.org/vmt-process.html#incident-report-taxonomy

information type: Public Security → Public
tags: added: security
Changed in ossa:
status: New → Won't Fix

Reviewed: https://review.openstack.org/548891
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9ba486370b00e131086162265c4a0e7dd85bf8ec
Submitter: Zuul
Branch: stable/ocata

commit 9ba486370b00e131086162265c4a0e7dd85bf8ec
Author: Eric Harney <email address hidden>
Date: Wed Feb 21 14:27:11 2018 -0500

    Log config options with oslo.config

    This removes some custom Cinder code which
    handles filtering secret config options in a flaky way.

    Filtering will now be based on the "secret=True" option
    flag.

    Related-Bug: #1750074
    Change-Id: I1c404b057d1471c85bd7eaf5c096f5912293460a
    (cherry picked from commit 7d278042c5280e40d5ed68f504f45ef023f05e18)
    (cherry picked from commit 4bc52eb7ba35da9005c7d28c341b0ce408216572)

tags: added: in-stable-ocata

Change abandoned by Eric Harney (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/545620
Reason: https://review.openstack.org/#/c/548891/

Reviewed: https://review.openstack.org/546786
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=3d7909deb21a1f0be4cd6eca13dc9e8d070f71e2
Submitter: Zuul
Branch: master

commit 3d7909deb21a1f0be4cd6eca13dc9e8d070f71e2
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 17:02:31 2018 -0500

    Log config options with oslo.config

    This removes some custom code inherited from Cinder which was handling
    the output of secret options in a bad way. This patch utilizes Oslo's
    existing utilities to output the Manila configuration options securely.

    Filtering will be done with the "secret=True" option flag.

    Major thanks to Eric Harney for introducing this fix to Cinder.

    Change-Id: I894e011680661c0b73b9592f70a6457e403f18c6
    Related-Bug: #1750074

Reviewed: https://review.openstack.org/549990
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=1949b403e9feb134d0fb2b9d65271292277351ee
Submitter: Zuul
Branch: stable/queens

commit 1949b403e9feb134d0fb2b9d65271292277351ee
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 17:02:31 2018 -0500

    Log config options with oslo.config

    This removes some custom code inherited from Cinder which was handling
    the output of secret options in a bad way. This patch utilizes Oslo's
    existing utilities to output the Manila configuration options securely.

    Filtering will be done with the "secret=True" option flag.

    Major thanks to Eric Harney for introducing this fix to Cinder.

    Change-Id: I894e011680661c0b73b9592f70a6457e403f18c6
    Related-Bug: #1750074
    (cherry picked from commit 3d7909deb21a1f0be4cd6eca13dc9e8d070f71e2)

Reviewed: https://review.openstack.org/549989
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=19aeba1f63f4e864eebda61bf16a078055c79cb0
Submitter: Zuul
Branch: stable/pike

commit 19aeba1f63f4e864eebda61bf16a078055c79cb0
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 17:02:31 2018 -0500

    Log config options with oslo.config

    This removes some custom code inherited from Cinder which was handling
    the output of secret options in a bad way. This patch utilizes Oslo's
    existing utilities to output the Manila configuration options securely.

    Filtering will be done with the "secret=True" option flag.

    Major thanks to Eric Harney for introducing this fix to Cinder.

    Change-Id: I894e011680661c0b73b9592f70a6457e403f18c6
    Related-Bug: #1750074
    (cherry picked from commit 3d7909deb21a1f0be4cd6eca13dc9e8d070f71e2)
    (cherry picked from commit 1949b403e9feb134d0fb2b9d65271292277351ee)

Reviewed: https://review.openstack.org/556276
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=4f5811de9684d2b868d5969eaba983a12679ab81
Submitter: Zuul
Branch: stable/ocata

commit 4f5811de9684d2b868d5969eaba983a12679ab81
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 17:02:31 2018 -0500

    Log config options with oslo.config

    This removes some custom code inherited from Cinder which was handling
    the output of secret options in a bad way. This patch utilizes Oslo's
    existing utilities to output the Manila configuration options securely.

    Filtering will be done with the "secret=True" option flag.

    Major thanks to Eric Harney for introducing this fix to Cinder.

    Change-Id: I894e011680661c0b73b9592f70a6457e403f18c6
    Related-Bug: #1750074
    (cherry picked from commit 3d7909deb21a1f0be4cd6eca13dc9e8d070f71e2)
    (cherry picked from commit 1949b403e9feb134d0fb2b9d65271292277351ee)
    (cherry picked from commit 19aeba1f63f4e864eebda61bf16a078055c79cb0)

This issue was fixed in the openstack/cinder 13.0.0.0b1 development milestone.

This issue was fixed in the openstack/manila 7.0.0.0b1 development milestone.

This issue was fixed in the openstack/cinder 11.1.1 release.

This issue was fixed in the openstack/manila 5.0.2 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers