Unredacted X-Auth-Token logged at level DEBUG in nova-api when HTTP status code != 2xx

[melanie witt]

Noticed this while working on something else, if the API is going to return a non 2xx HTTP success status code, a lot of request details are logged including the user's unsanitized auth token. In the past, operators considered this to be a security issue despite logging only at level DEBUG. For this reason I am opening a bug for review.

This particular logging code was added in the Zed release:

https://review.opendev.org/c/openstack/nova/+/806683

These are logged a lot when using OSC + server names because OSC always tries to lookup a name as a UUID (which will fail with 404) before it falls back on trying it as an ID. So commands such as 'openstack server show MyServer' will produce debug logs like the following.

Example log for GET /servers HTTP 404:

Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: INFO nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] HTTP exception thrown: Instance test could not be found.
Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: DEBUG nova.api.openstack.wsgi [None req-f5dbcb16-318b-4b19-96a6-62492ac94677 demo demo] Request method failure captured:
Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: request: GET /compute/v2.1/servers/test HTTP/1.1
Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: Accept: application/json
Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: Accept-Encoding: gzip, deflate
Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: Connection: keep-alive
Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: Content-Length: 0
Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: Host: 192.168.44.11
Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: Openstack-System-Scope: None
Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: User-Agent: python-novaclient
Mar 28 01:11:57 ubuntu-focal <email address hidden>[3890335]: X-Auth-Token: gAAAAABkIj7cGDJYKbnQalHot3qfNuAY1AMwMdKFP0kVQB6_8HRCSizDQpxMfspjx2S7t8rMWPSwYwg0-Yox2QM9E3KPWVq72YPl-cr8XwlDIn-ev9WjpmkmCtlOqOs0M0rOSvQggxdNB0xLx
2-gYiWUyMAYW6A1vXcup7Rvs8-YFetPKr2vGnY

[...]

Full log trace:

https://paste.openstack.org/show/bx9CmbgOsDNrIn16PfUW

[Balazs Gibizer (balazs-gibizer)]

I think we should revert https://review.opendev.org/c/openstack/nova/+/806683 asap and backport that revert. Leaking user tokens in the log (even in DEBUG) is a no-no for me.

[Sylvain Bauza (sylvain-bauza)]

Agreed with gibi here, and IMHO we should also add the SMT team to know whether we should have a CVE.

[Sylvain Bauza (sylvain-bauza)]

Also adding nova-coresec group here so Dan will see this bug report.

[Sylvain Bauza (sylvain-bauza)]

The other possibility instead of reverting https://review.opendev.org/c/openstack/nova/+/806683/ is to use mask_password from oslo.utils [1] like we do for all of the logs in the wsgi.py module...

[1] https://docs.openstack.org/oslo.utils/latest/reference/strutils.html#oslo_utils.strutils.mask_password

[Dan Smith (danms)]

Yeah, I don't know why we started doing that. I've been meaning to ask about it while debugging gate failures because we're now logging a ton of tracebacks on each run. That's something we used to specifically avoid because it just blows up the logs for silly things.

So yeah, +1 to reverting that.

[Tony Breeds (o-tony)]

FWIW I think reverting is the correct solution. The logging adds a lot of noise to the logs, it's a trivial (clean) revert and it's be very easy to do a release on each branch as the both have fewer than 10 changes merged.

I Also think this qualifies as a "Class A" security issue and and warrents both an OSSA and a CVE

[Jeremy Stanley (fungi)]

On 2023-04-13 00:22:49 -0000 (-0000), Tony Breeds wrote:
> FWIW I think reverting is the correct solution. The logging adds a lot
> of noise to the logs, it's a trivial (clean) revert and it's be very
> easy to do a release on each branch as the both have fewer than 10
> changes merged.
>
> I Also think this qualifies as a "Class A" security issue and and
> warrents both an OSSA and a CVE
[...]

I haven't been following closely (on vacation all week), but keep in
mind that if it only happens in "DEBUG" level logging we normally
categorize it as a class B3 report per our taxonomy[*], "a
vulnerability in experimental or debugging features not intended for
production use" (this was meant to cover leaking sensitive
information in DEBUG logs or command output when debug options are
turned on, as well as in features clearly documented as
experimental). As such, we wouldn't normally distribute a security
advisory but drafting a security note about it might still make
sense.

[*] https://security.openstack.org/vmt-process.html#report-taxonomy
--
Jeremy Stanley

[Sylvain Bauza (sylvain-bauza)]

Fungi, if I read you correctly, we wouldn't need an OSSA for this bug report, right?
If so, no CVE, right?

What's the timing then for the embargo ?

[Jeremy Stanley (fungi)]

This seems like something we would normally just switch to public immediately and fix in master as a security hardening measure, backporting if possible whenever convenient. If Tony or others involved in the report disagree with my assessment, perhaps if I've misunderstood the nature of the reported vulnerability for example, then we shouldn't switch it to public yet until we're sure everyone is on the same page.

[Dan Smith (danms)]

Yes, I would think this should be something we can just publicize and fix. I'm not familiar with all the classifications, so might be missing other reasoning that Tony has for doing a CVE.

[Tony Breeds (o-tony)]

I'm just out of practise and misunderstood the issue taxonomy.

[Sylvain Bauza (sylvain-bauza)]

Given for the moment, we have some CI hard issue, we need to wait for it to be fixed before modifying this bug to be publicized but I'm OK.

I also think we found the way : just reverting https://review.opendev.org/c/openstack/nova/+/806683

Do you want me to prepare a patch for this and provide it in this bug report ?

[Jeremy Stanley (fungi)]

Are we still concerned about CI problems blocking merge of fixes? If not, let's go ahead and switch this to public and push a change to Gerrit.

[Sylvain Bauza (sylvain-bauza)]

Okay, proposed https://review.opendev.org/c/openstack/nova/+/882052 and switching this bug report to Public.

[Jeremy Stanley (fungi)]

I've set the VMT's advisory tab to Won't Fix and switched the bug from Public Security to normal Public state, consistent with a hardening opportunity. We normally also add the "security" tag but it's already present.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/882052
Committed: https://opendev.org/openstack/nova/commit/6833695e70bba31b84a0a19301657bc59ae1710b
Submitter: "Zuul (22348)"
Branch: master

commit 6833695e70bba31b84a0a19301657bc59ae1710b
Author: Sylvain Bauza <email address hidden>
Date: Tue May 2 15:51:28 2023 +0000

    Revert "Debug Nova APIs call failures"

    This reverts commit afb0f774841d30dcae9c074d524e7fa9be840678.

    Reason for revert:

    We unfortunately leak the token in the logs which is considered a security flaw, even if only provided on DEBUG level.

    Change-Id: I52b52e65b689dadbdb08122c94652c491f850de6
    Closes-Bug: #2012993

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/nova/+/882783

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/882783
Committed: https://opendev.org/openstack/nova/commit/a02f96687350ad74d9921406a525ee991bbe8882
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit a02f96687350ad74d9921406a525ee991bbe8882
Author: Sylvain Bauza <email address hidden>
Date: Tue May 2 15:51:28 2023 +0000

    Revert "Debug Nova APIs call failures"

    This reverts commit afb0f774841d30dcae9c074d524e7fa9be840678.

    Reason for revert:

    We unfortunately leak the token in the logs which is considered a security flaw, even if only provided on DEBUG level.

    Change-Id: I52b52e65b689dadbdb08122c94652c491f850de6
    Closes-Bug: #2012993
    (cherry picked from commit 6833695e70bba31b84a0a19301657bc59ae1710b)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/nova/+/882786

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/nova/+/882786
Committed: https://opendev.org/openstack/nova/commit/91ee67019e0eb89290e599b2283f1f421718b796
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 91ee67019e0eb89290e599b2283f1f421718b796
Author: Sylvain Bauza <email address hidden>
Date: Tue May 2 15:51:28 2023 +0000

    Revert "Debug Nova APIs call failures"

    This reverts commit afb0f774841d30dcae9c074d524e7fa9be840678.

    Reason for revert:

    We unfortunately leak the token in the logs which is considered a security flaw, even if only provided on DEBUG level.

    Change-Id: I52b52e65b689dadbdb08122c94652c491f850de6
    Closes-Bug: #2012993
    (cherry picked from commit 6833695e70bba31b84a0a19301657bc59ae1710b)
    (cherry picked from commit a02f96687350ad74d9921406a525ee991bbe8882)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 26.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 27.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 28.0.0.0rc1 release candidate.