Sensitive information leaked in mistral logs

[Gauvain Pocentek]

Hi,

We discovered that all the data from the TripleO heat stack (user provided and generated passwords, certificates, ssh keys) are available in the mistral logs on the undercloud, in clear text.

We are running TripleO Rocky.

Truncated output:

2019-10-30 07:16:10.083 1 INFO mistral.engine.engine_server [req-1f9f6b40-3f82-479d-9960-3a9b13f57f88 734f9f46f9bd4298951bc155c6ba42bd 9481b04b8a9b44bc91bd2f7a78cc57a8 - default default] Received RPC request 'on_action_complete'[action_ex_id=90af917e-96f6-4bfe-ac86-93c473414336, result=Result [data={passwords: {u'KeystoneFernetKey1': u'ctgEMxw0.... u'OctaviaCaKeyPassphrase': u'Uk... , u'SSLKey': u'-----BEGIN PRIVATE KEY-----.... }

We don't have custom settings or modifications on the undercloud, and we use the containers from dockerhub.

From what I understand this is the normal mistral behaviour, but this type of information should not be logged at all.

Did we do something wrong or is it an actual security problem?

I reported the bug on TripleO because we use mistral in the undercloud context only, feel free to change the target project.

Let me know if you need more information.

[Emilien Macchi (emilienm)]

I've asked Dougal, our Mistral expert to take a look asap.

[Cédric Jeanneret (cjeanner)]

This patch against oslo.utils adds some more patterns for the mask_password method, and ensures it's actually case-insensitive.

It will be used for a next patch, hitting both:
- mistral-lib (newer releases)
- mistral (older releases)
The goal for those two future patches are to call the oslo.utils.mask_(dict_)password() method where needed/mandatory.

[Cédric Jeanneret (cjeanner)]

Mistral-lib patch calling the oslo_utils "mask_password" before formatting the logs.
Thank you Dougal for your help on that!

[Cédric Jeanneret (cjeanner)]

Mistral patch for Ocata and older versions.

BEWARE: this couldn't be tested because tox failed to install some deps (system too recent). But it should be good anyway©

[Dougal Matthews (d0ugal)]

These three patches LGTM.

[Gauvain Pocentek (gpocentek)]

Hi,

Thanks for taking care of this.

I patched the mistral_engine container with the patches from comments #2 and #3, but I still see the data in the logs. Is there another container I should update?

[Dougal Matthews (d0ugal)]

Gauvain, that is odd. I would have expected that to resolve it. I would patch all of the containers to be sure.

I'll spin up a environment now to test patching it in

[Gauvain Pocentek (gpocentek)]

I'll test again with all containers patched on Monday.

[Gauvain Pocentek (gpocentek)]

Hi. I haven't been able to test the patches again, but will try to do it soon.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692965
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=b41268417cecb12d1d5955ee3107067edf050221
Submitter: Zuul
Branch: master

commit b41268417cecb12d1d5955ee3107067edf050221
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692968
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=cae9aa72377713c2fc93b5cf3fad05b873a55d6d
Submitter: Zuul
Branch: stable/train

commit cae9aa72377713c2fc93b5cf3fad05b873a55d6d
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>
    (cherry picked from commit b41268417cecb12d1d5955ee3107067edf050221)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692971
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=f6d5df161fc7eaffe57e34e9fc20575b93a6f166
Submitter: Zuul
Branch: stable/stein

commit f6d5df161fc7eaffe57e34e9fc20575b93a6f166
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/692972
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=95f1b88c90ba0654e1353669b8b0f1d170391a25
Submitter: Zuul
Branch: stable/rocky

commit 95f1b88c90ba0654e1353669b8b0f1d170391a25
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>
    (cherry picked from commit b41268417cecb12d1d5955ee3107067edf050221)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/694515

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/694573

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/694574

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/694575

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694515
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=ed70bd3cd10eae2a34a5e9bd5d1fe0a6791ab3de
Submitter: Zuul
Branch: master

commit ed70bd3cd10eae2a34a5e9bd5d1fe0a6791ab3de
Author: Dougal Matthews <email address hidden>
Date: Fri Nov 15 11:11:07 2019 +0000

    Make mask_dict_password case insensitive and add new patterns

    In Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d mask_password was made case
    insensitive but mask_dict_password wasn't. This update makes the
    behaviour of these functions the same.

    Instead of lowering _SANITIZE_KEYS each time the source list is lowered.

    New password patterns from realworld logs were added to the patterns.

    Change-Id: Ic3ee301857630a15b9c26fd5d0fc907c43199517
    Related-Bug: #1850843

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694574
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=566e14cb96ef9fa520f007efdeb5e855587a1b98
Submitter: Zuul
Branch: stable/stein

commit 566e14cb96ef9fa520f007efdeb5e855587a1b98
Author: Dougal Matthews <email address hidden>
Date: Fri Nov 15 11:11:07 2019 +0000

    Make mask_dict_password case insensitive and add new patterns

    In Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d mask_password was made case
    insensitive but mask_dict_password wasn't. This update makes the
    behaviour of these functions the same.

    Instead of lowering _SANITIZE_KEYS each time the source list is lowered.

    New password patterns from realworld logs were added to the patterns.

    Change-Id: Ic3ee301857630a15b9c26fd5d0fc907c43199517
    Related-Bug: #1850843
    (cherry picked from commit ed70bd3cd10eae2a34a5e9bd5d1fe0a6791ab3de)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694575
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=c49a426b6618426f9260eea10c1b2a9e1c5a4d65
Submitter: Zuul
Branch: stable/rocky

commit c49a426b6618426f9260eea10c1b2a9e1c5a4d65
Author: Dougal Matthews <email address hidden>
Date: Fri Nov 15 11:11:07 2019 +0000

    Make mask_dict_password case insensitive and add new patterns

    In Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d mask_password was made case
    insensitive but mask_dict_password wasn't. This update makes the
    behaviour of these functions the same.

    Instead of lowering _SANITIZE_KEYS each time the source list is lowered.

    New password patterns from realworld logs were added to the patterns.

    Change-Id: Ic3ee301857630a15b9c26fd5d0fc907c43199517
    Related-Bug: #1850843
    (cherry picked from commit ed70bd3cd10eae2a34a5e9bd5d1fe0a6791ab3de)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/718440

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/718441

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/718440
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=9d2d9a8949122a034f4cee8a058bc90382c0621d
Submitter: Zuul
Branch: stable/queens

commit 9d2d9a8949122a034f4cee8a058bc90382c0621d
Author: Cédric Jeanneret <email address hidden>
Date: Fri Nov 1 11:27:37 2019 +0100

    Make mask_password case insensitive, and add new patterns

    It appears that Mistral service logs everything, and doesn't use yet
    the mask_password (nor mask_dict_password) method. In order to ensure
    all is properly masked, we have to add some new patterns, and make it
    case insensitive in order to simplify and avoid duplicated entries.

    Change-Id: Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d
    Related-Bug: #1850843
    Signed-off-by: Cédric Jeanneret <email address hidden>
    (cherry picked from commit b41268417cecb12d1d5955ee3107067edf050221)
    (cherry picked from commit 95f1b88c90ba0654e1353669b8b0f1d170391a25)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/718441
Committed: https://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=d0b99227d9bf67316905d973d49cc85aa9a93c04
Submitter: Zuul
Branch: stable/queens

commit d0b99227d9bf67316905d973d49cc85aa9a93c04
Author: Dougal Matthews <email address hidden>
Date: Fri Nov 15 11:11:07 2019 +0000

    Make mask_dict_password case insensitive and add new patterns

    In Icc19b7c8bdb6a3182939d5e9fdef21288b19f43d mask_password was made case
    insensitive but mask_dict_password wasn't. This update makes the
    behaviour of these functions the same.

    Instead of lowering _SANITIZE_KEYS each time the source list is lowered.

    New password patterns from realworld logs were added to the patterns.

    Change-Id: Ic3ee301857630a15b9c26fd5d0fc907c43199517
    Related-Bug: #1850843
    (cherry picked from commit ed70bd3cd10eae2a34a5e9bd5d1fe0a6791ab3de)
    (cherry picked from commit c49a426b6618426f9260eea10c1b2a9e1c5a4d65)