Cinder logs rabbitmq password on connection log

[Marga Millet]

Cinder may log rabbitmq password on connection when DEBUG is on.

Example on cinder-scheduler.log file after enabling DEBUG:
(Password has been replaced with XXX)

2018-02-05 19:21:52.721 35 DEBUG cinder.service [req-a2dbe0dd-14c9-4123-a69a-3623e5f0a4d7 - - - - -] transport_url : rabbit://guest:XXX@10.10.10.1:5672,guest:XXX@10.10.10.2:5672,guest:XXX@10.10.10.3:5672 wait /usr/lib/python2.7/site-packages/cinder/service.py:611

In a production environment, this is pretty bad.

[Marga Millet (millet)]

This applies to all releases.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/545486

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/545620

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/545621

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/545622

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545486
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=919dd16a35ef2d0fd2ee4911398ba7830c02c6fd
Submitter: Zuul
Branch: master

commit 919dd16a35ef2d0fd2ee4911398ba7830c02c6fd
Author: Marga Millet <email address hidden>
Date: Fri Feb 16 23:05:47 2018 +0000

    Cinder logs rabbitmq password on connection log

    Cinder displays rabbitmq password if debug is enabled.

    Closes-Bug: 1750074
    Change-Id: I117319ac12991e4b46170fe71d18a65ea4c98556

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545622
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=ca9bf2d1c56605a565ffc468bb91fb5d58ef5ce9
Submitter: Zuul
Branch: stable/queens

commit ca9bf2d1c56605a565ffc468bb91fb5d58ef5ce9
Author: Marga Millet <email address hidden>
Date: Fri Feb 16 23:05:47 2018 +0000

    Cinder logs rabbitmq password on connection log

    Cinder displays rabbitmq password if debug is enabled.

    Closes-Bug: 1750074
    Change-Id: I117319ac12991e4b46170fe71d18a65ea4c98556
    (cherry picked from commit 919dd16a35ef2d0fd2ee4911398ba7830c02c6fd)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/546673

[Eric Harney (eharney)]

I don't think this fix is adequate:
https://review.openstack.org/#/c/545486/

The code in cinder/service.py fails to account for a bunch of other options, such as

transport_url=qpid://...
sql_connection=mysql+pymysql://...

This needs to be leveraging the secret flag for options, like this code does:
https://git.openstack.org/cgit/openstack/oslo.config/tree/oslo_config/cfg.py?h=5.2.0#n2865

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546673
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=05e4f14ea1fbc7ce61c8f57a082080d09c63e357
Submitter: Zuul
Branch: master

commit 05e4f14ea1fbc7ce61c8f57a082080d09c63e357
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 10:53:41 2018 -0500

    Fix manila logging rabbitmq password in debug mode

    Manila will display the rabbitmq password if debugging is enabled.
    This patch will ensure that the rabbitmq password is no longer
    displayed in the connection log for Manila when debugging is
    enabled by looking for the rabbitmq key and not printing it.

    There should likely be an effort to utilize Oslo's secret flag for
    options to truly fix this issue for this and other sensitive options.

    Change-Id: I97cc88354d9b54057350c70c4742055197540d1a
    Closes-Bug: 1750074

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545621
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d51fabb497df2606f04f90cbd2e842c7692bb1af
Submitter: Zuul
Branch: stable/pike

commit d51fabb497df2606f04f90cbd2e842c7692bb1af
Author: Marga Millet <email address hidden>
Date: Fri Feb 16 23:05:47 2018 +0000

    Cinder logs rabbitmq password on connection log

    Cinder displays rabbitmq password if debug is enabled.

    Closes-Bug: 1750074
    Change-Id: I117319ac12991e4b46170fe71d18a65ea4c98556
    (cherry picked from commit 919dd16a35ef2d0fd2ee4911398ba7830c02c6fd)
    (cherry picked from commit ca9bf2d1c56605a565ffc468bb91fb5d58ef5ce9)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/546754

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/546755

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 12.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/546786

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546755
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=be8580b350c34c91494cb553eb15e6b61ef250b7
Submitter: Zuul
Branch: stable/queens

commit be8580b350c34c91494cb553eb15e6b61ef250b7
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 10:53:41 2018 -0500

    Fix manila logging rabbitmq password in debug mode

    Manila will display the rabbitmq password if debugging is enabled.
    This patch will ensure that the rabbitmq password is no longer
    displayed in the connection log for Manila when debugging is
    enabled by looking for the rabbitmq key and not printing it.

    There should likely be an effort to utilize Oslo's secret flag for
    options to truly fix this issue for this and other sensitive options.

    Change-Id: I97cc88354d9b54057350c70c4742055197540d1a
    Closes-Bug: 1750074
    (cherry picked from commit 05e4f14ea1fbc7ce61c8f57a082080d09c63e357)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 6.0.0.0rc3 release candidate.

[Dirk Mueller (dmllr)]

This patch to cinder broke cinder:

2018-02-22 17:56:30.587 32083 ERROR cinder Traceback (most recent call last):
2018-02-22 17:56:30.587 32083 ERROR cinder File "/usr/bin/cinder-scheduler", line 10, in <module>
2018-02-22 17:56:30.587 32083 ERROR cinder sys.exit(main())
2018-02-22 17:56:30.587 32083 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/cmd/scheduler.py", line 56, in main
2018-02-22 17:56:30.587 32083 ERROR cinder service.wait()
2018-02-22 17:56:30.587 32083 ERROR cinder File "/usr/lib/python2.7/site-packages/cinder/service.py", line 624, in wait
2018-02-22 17:56:30.587 32083 ERROR cinder (flag == "transport_url" and "rabbit:" in flag_get) or
2018-02-22 17:56:30.587 32083 ERROR cinder TypeError: argument of type 'NoneType' is not iterable
2018-02-22 17:56:30.587 32083 ERROR cinder

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/547229

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/547230

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546754
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=f4e2efb58dd805c2b970c3705535ee27c37a84c5
Submitter: Zuul
Branch: master

commit f4e2efb58dd805c2b970c3705535ee27c37a84c5
Author: Eric Harney <email address hidden>
Date: Wed Feb 21 14:27:11 2018 -0500

    Log config options with oslo.config

    This removes some custom Cinder code which
    handles filtering secret config options in a flaky way.

    Filtering will now be based on the "secret=True" option
    flag.

    Related-Bug: #1750074
    Change-Id: I1c404b057d1471c85bd7eaf5c096f5912293460a

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/547229
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=7d278042c5280e40d5ed68f504f45ef023f05e18
Submitter: Zuul
Branch: stable/queens

commit 7d278042c5280e40d5ed68f504f45ef023f05e18
Author: Eric Harney <email address hidden>
Date: Wed Feb 21 14:27:11 2018 -0500

    Log config options with oslo.config

    This removes some custom Cinder code which
    handles filtering secret config options in a flaky way.

    Filtering will now be based on the "secret=True" option
    flag.

    Related-Bug: #1750074
    Change-Id: I1c404b057d1471c85bd7eaf5c096f5912293460a

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/548891

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/549596

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549596
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=c0f1af11b102eabdf7b21d79c80eb8945f03dee8
Submitter: Zuul
Branch: stable/pike

commit c0f1af11b102eabdf7b21d79c80eb8945f03dee8
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 10:53:41 2018 -0500

    Fix manila logging rabbitmq password in debug mode

    Manila will display the rabbitmq password if debugging is enabled.
    This patch will ensure that the rabbitmq password is no longer
    displayed in the connection log for Manila when debugging is
    enabled by looking for the rabbitmq key and not printing it.

    There should likely be an effort to utilize Oslo's secret flag for
    options to truly fix this issue for this and other sensitive options.

    Change-Id: I97cc88354d9b54057350c70c4742055197540d1a
    Closes-Bug: 1750074
    (cherry picked from commit 05e4f14ea1fbc7ce61c8f57a082080d09c63e357)
    (cherry picked from commit be8580b350c34c91494cb553eb15e6b61ef250b7)

[Jeremy Stanley (fungi)]

I'm marking the advisory task won't fix and triaging this as a potential security hardening opportunity. In the past we've considered information leaking in DEBUG level logs to fit the B3 classification (a vulnerability in experimental or debugging features not intended for production use) in our report taxonomy: https://security.openstack.org/vmt-process.html#incident-report-taxonomy

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/549989

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/549990

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/548891
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9ba486370b00e131086162265c4a0e7dd85bf8ec
Submitter: Zuul
Branch: stable/ocata

commit 9ba486370b00e131086162265c4a0e7dd85bf8ec
Author: Eric Harney <email address hidden>
Date: Wed Feb 21 14:27:11 2018 -0500

    Log config options with oslo.config

    This removes some custom Cinder code which
    handles filtering secret config options in a flaky way.

    Filtering will now be based on the "secret=True" option
    flag.

    Related-Bug: #1750074
    Change-Id: I1c404b057d1471c85bd7eaf5c096f5912293460a
    (cherry picked from commit 7d278042c5280e40d5ed68f504f45ef023f05e18)
    (cherry picked from commit 4bc52eb7ba35da9005c7d28c341b0ce408216572)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Eric Harney (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/545620
Reason: https://review.openstack.org/#/c/548891/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546786
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=3d7909deb21a1f0be4cd6eca13dc9e8d070f71e2
Submitter: Zuul
Branch: master

commit 3d7909deb21a1f0be4cd6eca13dc9e8d070f71e2
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 17:02:31 2018 -0500

    Log config options with oslo.config

    This removes some custom code inherited from Cinder which was handling
    the output of secret options in a bad way. This patch utilizes Oslo's
    existing utilities to output the Manila configuration options securely.

    Filtering will be done with the "secret=True" option flag.

    Major thanks to Eric Harney for introducing this fix to Cinder.

    Change-Id: I894e011680661c0b73b9592f70a6457e403f18c6
    Related-Bug: #1750074

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549990
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=1949b403e9feb134d0fb2b9d65271292277351ee
Submitter: Zuul
Branch: stable/queens

commit 1949b403e9feb134d0fb2b9d65271292277351ee
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 17:02:31 2018 -0500

    Log config options with oslo.config

    This removes some custom code inherited from Cinder which was handling
    the output of secret options in a bad way. This patch utilizes Oslo's
    existing utilities to output the Manila configuration options securely.

    Filtering will be done with the "secret=True" option flag.

    Major thanks to Eric Harney for introducing this fix to Cinder.

    Change-Id: I894e011680661c0b73b9592f70a6457e403f18c6
    Related-Bug: #1750074
    (cherry picked from commit 3d7909deb21a1f0be4cd6eca13dc9e8d070f71e2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/549989
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=19aeba1f63f4e864eebda61bf16a078055c79cb0
Submitter: Zuul
Branch: stable/pike

commit 19aeba1f63f4e864eebda61bf16a078055c79cb0
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 17:02:31 2018 -0500

    Log config options with oslo.config

    This removes some custom code inherited from Cinder which was handling
    the output of secret options in a bad way. This patch utilizes Oslo's
    existing utilities to output the Manila configuration options securely.

    Filtering will be done with the "secret=True" option flag.

    Major thanks to Eric Harney for introducing this fix to Cinder.

    Change-Id: I894e011680661c0b73b9592f70a6457e403f18c6
    Related-Bug: #1750074
    (cherry picked from commit 3d7909deb21a1f0be4cd6eca13dc9e8d070f71e2)
    (cherry picked from commit 1949b403e9feb134d0fb2b9d65271292277351ee)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/556276

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/556276
Committed: https://git.openstack.org/cgit/openstack/manila/commit/?id=4f5811de9684d2b868d5969eaba983a12679ab81
Submitter: Zuul
Branch: stable/ocata

commit 4f5811de9684d2b868d5969eaba983a12679ab81
Author: Dustin Schoenbrun <email address hidden>
Date: Wed Feb 21 17:02:31 2018 -0500

    Log config options with oslo.config

    This removes some custom code inherited from Cinder which was handling
    the output of secret options in a bad way. This patch utilizes Oslo's
    existing utilities to output the Manila configuration options securely.

    Filtering will be done with the "secret=True" option flag.

    Major thanks to Eric Harney for introducing this fix to Cinder.

    Change-Id: I894e011680661c0b73b9592f70a6457e403f18c6
    Related-Bug: #1750074
    (cherry picked from commit 3d7909deb21a1f0be4cd6eca13dc9e8d070f71e2)
    (cherry picked from commit 1949b403e9feb134d0fb2b9d65271292277351ee)
    (cherry picked from commit 19aeba1f63f4e864eebda61bf16a078055c79cb0)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 13.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 7.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/cinder 11.1.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/manila 5.0.2 release.