Sensitive information leaked in mistral logs

Bug #1850843 reported by Gauvain Pocentek on 2019-10-31
260
This bug affects 1 person
Affects Status Importance Assigned to Milestone
oslo.utils
Critical
Dougal Matthews
tripleo
Critical
Dougal Matthews

Bug Description

Hi,

We discovered that all the data from the TripleO heat stack (user provided and generated passwords, certificates, ssh keys) are available in the mistral logs on the undercloud, in clear text.

We are running TripleO Rocky.

Truncated output:

2019-10-30 07:16:10.083 1 INFO mistral.engine.engine_server [req-1f9f6b40-3f82-479d-9960-3a9b13f57f88 734f9f46f9bd4298951bc155c6ba42bd 9481b04b8a9b44bc91bd2f7a78cc57a8 - default default] Received RPC request 'on_action_complete'[action_ex_id=90af917e-96f6-4bfe-ac86-93c473414336, result=Result [data={passwords: {u'KeystoneFernetKey1': u'ctgEMxw0.... u'OctaviaCaKeyPassphrase': u'Uk... , u'SSLKey': u'-----BEGIN PRIVATE KEY-----.... }

We don't have custom settings or modifications on the undercloud, and we use the containers from dockerhub.

From what I understand this is the normal mistral behaviour, but this type of information should not be logged at all.

Did we do something wrong or is it an actual security problem?

I reported the bug on TripleO because we use mistral in the undercloud context only, feel free to change the target project.

Let me know if you need more information.

Emilien Macchi (emilienm) wrote :

I've asked Dougal, our Mistral expert to take a look asap.

Changed in tripleo:
status: New → Triaged
importance: Undecided → Critical
milestone: none → ussuri-1
Changed in tripleo:
assignee: nobody → Dougal Matthews (d0ugal)
Cédric Jeanneret (cjeanner) wrote :

This patch against oslo.utils adds some more patterns for the mask_password method, and ensures it's actually case-insensitive.

It will be used for a next patch, hitting both:
- mistral-lib (newer releases)
- mistral (older releases)
The goal for those two future patches are to call the oslo.utils.mask_(dict_)password() method where needed/mandatory.

Cédric Jeanneret (cjeanner) wrote :

Mistral-lib patch calling the oslo_utils "mask_password" before formatting the logs.
Thank you Dougal for your help on that!

Cédric Jeanneret (cjeanner) wrote :

Mistral patch for Ocata and older versions.

BEWARE: this couldn't be tested because tox failed to install some deps (system too recent). But it should be good anyway©

Dougal Matthews (d0ugal) wrote :

These three patches LGTM.

Gauvain Pocentek (gpocentek) wrote :

Hi,

Thanks for taking care of this.

I patched the mistral_engine container with the patches from comments #2 and #3, but I still see the data in the logs. Is there another container I should update?

Dougal Matthews (d0ugal) wrote :

Gauvain, that is odd. I would have expected that to resolve it. I would patch all of the containers to be sure.

I'll spin up a environment now to test patching it in

Gauvain Pocentek (gpocentek) wrote :

I'll test again with all containers patched on Monday.

Dougal Matthews (d0ugal) on 2019-11-05
information type: Private Security → Public Security
Gauvain Pocentek (gpocentek) wrote :

Hi. I haven't been able to test the patches again, but will try to do it soon.

Ben Nemec (bnemec) on 2019-11-12
Changed in oslo.utils:
status: New → Triaged
importance: Undecided → Critical

Reviewed: https://review.opendev.org/692965
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=b41268417cecb12d1d5955ee3107067edf050221
Submitter: Zuul
Branch: master

commit b41268417cecb12d1d5955ee3107067edf050221
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>

Reviewed: https://review.opendev.org/692968
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=cae9aa72377713c2fc93b5cf3fad05b873a55d6d
Submitter: Zuul
Branch: stable/train

commit cae9aa72377713c2fc93b5cf3fad05b873a55d6d
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>
    (cherry picked from commit b41268417cecb12d1d5955ee3107067edf050221)

tags: added: in-stable-train

Reviewed: https://review.opendev.org/692971
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=f6d5df161fc7eaffe57e34e9fc20575b93a6f166
Submitter: Zuul
Branch: stable/stein

commit f6d5df161fc7eaffe57e34e9fc20575b93a6f166
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>

tags: added: in-stable-stein

Reviewed: https://review.opendev.org/692972
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=95f1b88c90ba0654e1353669b8b0f1d170391a25
Submitter: Zuul
Branch: stable/rocky

commit 95f1b88c90ba0654e1353669b8b0f1d170391a25
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>
    (cherry picked from commit b41268417cecb12d1d5955ee3107067edf050221)

tags: added: in-stable-rocky

Reviewed: https://review.opendev.org/694515
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=ed70bd3cd10eae2a34a5e9bd5d1fe0a6791ab3de
Submitter: Zuul
Branch: master

commit ed70bd3cd10eae2a34a5e9bd5d1fe0a6791ab3de
Author: Dougal Matthews <email address hidden>
Date: Fri Nov 15 11:11:07 2019 +0000

    Make mask_dict_password case insensitive and add new patterns

    In Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d mask_password was made case
    insensitive but mask_dict_password wasn't. This update makes the
    behaviour of these functions the same.

    Instead of lowering _SANITIZE_KEYS each time the source list is lowered.

    New password patterns from realworld logs were added to the patterns.

    Change-Id: Ic3ee301857630a15b9c26fd5d0fc907c43199517
    Related-Bug: #1850843

Reviewed: https://review.opendev.org/694574
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=566e14cb96ef9fa520f007efdeb5e855587a1b98
Submitter: Zuul
Branch: stable/stein

commit 566e14cb96ef9fa520f007efdeb5e855587a1b98
Author: Dougal Matthews <email address hidden>
Date: Fri Nov 15 11:11:07 2019 +0000

    Make mask_dict_password case insensitive and add new patterns

    In Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d mask_password was made case
    insensitive but mask_dict_password wasn't. This update makes the
    behaviour of these functions the same.

    Instead of lowering _SANITIZE_KEYS each time the source list is lowered.

    New password patterns from realworld logs were added to the patterns.

    Change-Id: Ic3ee301857630a15b9c26fd5d0fc907c43199517
    Related-Bug: #1850843
    (cherry picked from commit ed70bd3cd10eae2a34a5e9bd5d1fe0a6791ab3de)

Reviewed: https://review.opendev.org/694575
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=c49a426b6618426f9260eea10c1b2a9e1c5a4d65
Submitter: Zuul
Branch: stable/rocky

commit c49a426b6618426f9260eea10c1b2a9e1c5a4d65
Author: Dougal Matthews <email address hidden>
Date: Fri Nov 15 11:11:07 2019 +0000

    Make mask_dict_password case insensitive and add new patterns

    In Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d mask_password was made case
    insensitive but mask_dict_password wasn't. This update makes the
    behaviour of these functions the same.

    Instead of lowering _SANITIZE_KEYS each time the source list is lowered.

    New password patterns from realworld logs were added to the patterns.

    Change-Id: Ic3ee301857630a15b9c26fd5d0fc907c43199517
    Related-Bug: #1850843
    (cherry picked from commit ed70bd3cd10eae2a34a5e9bd5d1fe0a6791ab3de)

Dougal Matthews (d0ugal) on 2019-11-21
Changed in oslo.utils:
assignee: nobody → Dougal Matthews (d0ugal)
Dougal Matthews (d0ugal) on 2019-11-21
Changed in oslo.utils:
status: Triaged → Fix Released
Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Other bug subscribers