Sensitive information leaked in mistral logs

Bug #1850843 reported by Gauvain Pocentek on 2019-10-31
260
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.utils
Critical
Unassigned
tripleo
Critical
Dougal Matthews

Bug Description

Hi,

We discovered that all the data from the TripleO heat stack (user provided and generated passwords, certificates, ssh keys) are available in the mistral logs on the undercloud, in clear text.

We are running TripleO Rocky.

Truncated output:

2019-10-30 07:16:10.083 1 INFO mistral.engine.engine_server [req-1f9f6b40-3f82-479d-9960-3a9b13f57f88 734f9f46f9bd4298951bc155c6ba42bd 9481b04b8a9b44bc91bd2f7a78cc57a8 - default default] Received RPC request 'on_action_complete'[action_ex_id=90af917e-96f6-4bfe-ac86-93c473414336, result=Result [data={passwords: {u'KeystoneFernetKey1': u'ctgEMxw0.... u'OctaviaCaKeyPassphrase': u'Uk... , u'SSLKey': u'-----BEGIN PRIVATE KEY-----.... }

We don't have custom settings or modifications on the undercloud, and we use the containers from dockerhub.

From what I understand this is the normal mistral behaviour, but this type of information should not be logged at all.

Did we do something wrong or is it an actual security problem?

I reported the bug on TripleO because we use mistral in the undercloud context only, feel free to change the target project.

Let me know if you need more information.

Emilien Macchi (emilienm) wrote :

I've asked Dougal, our Mistral expert to take a look asap.

Changed in tripleo:
status: New → Triaged
importance: Undecided → Critical
milestone: none → ussuri-1
Changed in tripleo:
assignee: nobody → Dougal Matthews (d0ugal)
Cédric Jeanneret (cjeanner) wrote :

This patch against oslo.utils adds some more patterns for the mask_password method, and ensures it's actually case-insensitive.

It will be used for a next patch, hitting both:
- mistral-lib (newer releases)
- mistral (older releases)
The goal for those two future patches are to call the oslo.utils.mask_(dict_)password() method where needed/mandatory.

Cédric Jeanneret (cjeanner) wrote :

Mistral-lib patch calling the oslo_utils "mask_password" before formatting the logs.
Thank you Dougal for your help on that!

Cédric Jeanneret (cjeanner) wrote :

Mistral patch for Ocata and older versions.

BEWARE: this couldn't be tested because tox failed to install some deps (system too recent). But it should be good anyway©

Dougal Matthews (d0ugal) wrote :

These three patches LGTM.

Gauvain Pocentek (gpocentek) wrote :

Hi,

Thanks for taking care of this.

I patched the mistral_engine container with the patches from comments #2 and #3, but I still see the data in the logs. Is there another container I should update?

Dougal Matthews (d0ugal) wrote :

Gauvain, that is odd. I would have expected that to resolve it. I would patch all of the containers to be sure.

I'll spin up a environment now to test patching it in

Gauvain Pocentek (gpocentek) wrote :

I'll test again with all containers patched on Monday.

Dougal Matthews (d0ugal) on 2019-11-05
information type: Private Security → Public Security
Gauvain Pocentek (gpocentek) wrote :

Hi. I haven't been able to test the patches again, but will try to do it soon.

Ben Nemec (bnemec) 9 hours ago
Changed in oslo.utils:
status: New → Triaged
importance: Undecided → Critical

Reviewed: https://review.opendev.org/692965
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=b41268417cecb12d1d5955ee3107067edf050221
Submitter: Zuul
Branch: master

commit b41268417cecb12d1d5955ee3107067edf050221
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>

Reviewed: https://review.opendev.org/692968
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=cae9aa72377713c2fc93b5cf3fad05b873a55d6d
Submitter: Zuul
Branch: stable/train

commit cae9aa72377713c2fc93b5cf3fad05b873a55d6d
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>
    (cherry picked from commit b41268417cecb12d1d5955ee3107067edf050221)

tags: added: in-stable-train
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers