Aodh can be used to launder Keystone trusts

Bug #1649333 reported by Zane Bitter
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Aodh
Fix Released
Undecided
Zane Bitter
OpenStack Security Notes
Fix Released
Undecided
Luke Hinds

Bug Description

When adding an alarm action with the scheme trust+http:, Aodh allows the user to provide a trust ID to use to acquire a token with which to make the webhook request. (If no trust ID is provided then Aodh creates a trust internally, in which case there is no issue.) However, Aodh makes no attempt to check that the user creating the alarm is the trustor or has the same rights as the trustor - it doesn't even attempt to check that the trust is for the same project as the alarm.

The nature of the trust+http: alarm notifier is that it allows the user to obtain a token given the ID of a trust for which Aodh is the trustee, since the URL is arbitrary and not limited to services in the Keystone catalog.

It is my understanding (though I'm open to correction) that trust IDs are not meant to be secret - i.e. that security should rely on the secrecy of the trustee's credentials, not of the trust ID.

The current setup requires that the security posture be equivalent to, or even stronger than, that for a bearer token, since the trust ID can be used to obtain a token and it may never expire. (The vulnerability is limited to Aodh though, since it only applies to trusts where Aodh is the trustee.) In principle this *could* be a valid choice, although I personally would disagree with it. In practice, that is not the current security posture - for example, trust IDs are stored unencrypted in the database and routinely logged unobfuscated.

The easiest solution would be to simply disallow users supplying their own trust IDs. This may well be feasible - it's likely that in most clouds non-admin users will not have the necessary permissions to get the ID of the Aodh service user, which is required to create a trust with that user as the trustee. Heat does not supply a trust ID in its convenience code for setting up an autoscaling alarm. So there are probably no users of this feature in the wild.

To eliminate the vulnerability without eliminating the feature, I believe Aodh would need to check that the user creating the alarm possesses all of the trust privileges - i.e. their token is for the same project ID and they have all of the roles that the trust does. Or, alternatively, check that the user *is* (i.e. has the same user ID as) the trustor.

Revision history for this message
Thomas Herve (therve) wrote :

I introduced the feature back in the day where it was in ceilometer, and we were not sure whether or not the trust ought to be created by ceilometer. I think it's safe to always create the trust now; I'd be shocked if anyone ever tried to pass a pre-created one.

Revision history for this message
Zane Bitter (zaneb) wrote :

I'm attaching a fix.

What is the process for fixing security issues in Aodh? Is it managed by the VMT?

Changed in aodh:
status: New → Confirmed
assignee: nobody → Zane Bitter (zaneb)
Zane Bitter (zaneb)
Changed in aodh:
status: Confirmed → In Progress
Revision history for this message
Julien Danjou (jdanjou) wrote :

That seems fair enough indeed.

According to https://governance.openstack.org/tc/reference/tags/vulnerability_managed.html it's not managed by VMT. So I'm not sure what to do next (other than applying your fix)

Revision history for this message
Robert Clark (robert-clark) wrote :

We can create an embargoed OpenStack Security Note https://wiki.openstack.org/wiki/Security#Security_Notes_-_OSSN that will serve in place of a Security Advisory (that you'd get from the VMT if you were security managed).

The VMT has a list of downstream stakeholders (mainly large OpenStack operators). We can send them the OSSN, along with the patch and then publish both a few days later.

I'll add this to the OSSN queue now, so that it's on Luke Hinds' (Our OSSN manager) radar.

Revision history for this message
Zane Bitter (zaneb) wrote :

Attaching a backport for stable/newton. The same patch also applies to stable/mitaka with a minimum of fuzzing.

Revision history for this message
Zane Bitter (zaneb) wrote :

Any progress on the security note? We've released Ocata with the bug since this was last discussed, and we're getting close to releasing Pike. I'd still like to have the OSSN, but at this point I'd rather have the issue fixed.

Revision history for this message
Robert Clark (robert-clark) wrote : Re: [Bug 1649333] Re: Aodh can be used to launder Keystone trusts
Download full text (3.3 KiB)

Sorry for dropping the ball on this one guys, I currently don't have LP
access due to MFA issues!

Luke Hinds <email address hidden> manages the OSSNs and can get this processed
for you quickly.

I haven't CC'd him on this thread as permission should be controlled
through LP but please add him.

-Rob

On Tue, Jul 25, 2017 at 4:38 PM, Zane Bitter <email address hidden> wrote:

> Any progress on the security note? We've released Ocata with the bug
> since this was last discussed, and we're getting close to releasing
> Pike. I'd still like to have the OSSN, but at this point I'd rather have
> the issue fixed.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1649333
>
> Title:
> Aodh can be used to launder Keystone trusts
>
> Status in Aodh:
> In Progress
> Status in OpenStack Security Notes:
> New
>
> Bug description:
> When adding an alarm action with the scheme trust+http:, Aodh allows
> the user to provide a trust ID to use to acquire a token with which to
> make the webhook request. (If no trust ID is provided then Aodh
> creates a trust internally, in which case there is no issue.) However,
> Aodh makes no attempt to check that the user creating the alarm is the
> trustor or has the same rights as the trustor - it doesn't even
> attempt to check that the trust is for the same project as the alarm.
>
> The nature of the trust+http: alarm notifier is that it allows the
> user to obtain a token given the ID of a trust for which Aodh is the
> trustee, since the URL is arbitrary and not limited to services in the
> Keystone catalog.
>
> It is my understanding (though I'm open to correction) that trust IDs
> are not meant to be secret - i.e. that security should rely on the
> secrecy of the trustee's credentials, not of the trust ID.
>
> The current setup requires that the security posture be equivalent to,
> or even stronger than, that for a bearer token, since the trust ID can
> be used to obtain a token and it may never expire. (The vulnerability
> is limited to Aodh though, since it only applies to trusts where Aodh
> is the trustee.) In principle this *could* be a valid choice, although
> I personally would disagree with it. In practice, that is not the
> current security posture - for example, trust IDs are stored
> unencrypted in the database and routinely logged unobfuscated.
>
>
> The easiest solution would be to simply disallow users supplying their
> own trust IDs. This may well be feasible - it's likely that in most clouds
> non-admin users will not have the necessary permissions to get the ID of
> the Aodh service user, which is required to create a trust with that user
> as the trustee. Heat does not supply a trust ID in its convenience code for
> setting up an autoscaling alarm. So there are probably no users of this
> feature in the wild.
>
> To eliminate the vulnerability without eliminating the feature, I
> believe Aodh would need to check that the user creating the alarm
> possesses all of the trust privileges - i.e. their token is for the
> same project ID and they have all of the roles that the tr...

Read more...

Revision history for this message
Zane Bitter (zaneb) wrote :

Thanks Robert, I just subscribed Luke to this issue.

Luke Hinds (lhinds)
Changed in ossn:
assignee: nobody → Luke Hinds (lhinds)
Revision history for this message
Luke Hinds (lhinds) wrote :

Hi Zane,

What is the latest you can merge this into pike? As Rob says, we typically give downstream steakholders a few days to patch before going public, so I would like to make it so they get that window, but importantly you have time to get this in for any code freeze there might be for Aodh in pike.

Also I am not that familiar with aodh, so I will verify the OSSN content in this LP, before sending. Main part where I will need some assistance on applying the patch (possible posix paths, dry runs , args etc).

Luke

Revision history for this message
Zane Bitter (zaneb) wrote :

Pike-rc1 is the week of August 11: https://releases.openstack.org/pike/schedule.html

There's no restriction on when we can commit bug fixes up until the final release, so any time before then is fine I think.

Revision history for this message
Adam Young (ayoung) wrote :

It sounds like the fix should be that either Aodh always triggers the trust creation, or that Aodh validates the trust it is given. I think both are acceptabled.

To validate the trust, Aodh should immediately activate the trust (use it to get a token) and in the token requests response, check the value of the trustor, confirming it matches the expected user.

Blindly accepting any Trust ID this way is certainly an error.

Revision history for this message
Luke Hinds (lhinds) wrote :

Hi, Please review the note below.

@Zane, please recommend how specifically an operator should apply your patch, and list any services will need restarting. You can see the sections under "Recommended Actions".

~~~~BEGIN~~~~

This is an advance warning of a potential issue discovered in OpenStack, to give you, as downstream stakeholders, a chance to coordinate a response.

Please note, this is not an OpenStack Security Advisory (OSSA), but advanced notice of an upcoming OpenStack Security Note (OSSN).

OSSNs are generally released publicly, but given the nature of this note we've opted to provide advanced notification.

Please treat the following information as confidential until the proposed public disclosure date, which is 7 days after embargo notice (this email).

Aodh can be used to launder Keystone trusts
---

### Summary ###

When adding an alarm action with the scheme `trust+http:` Aodh does not verify that the user creating the alarm is the trustor or has the same rights as the trustor, not that the trust is for the same project as the alarm.

### Affected Services / Software ###

Aodh is the alarm engine of the Telemetry project.

### Discussion ###

When adding an alarm action with the scheme `trust+http:`, Aodh allows the user to provide a trust ID to acquire a token with which to make a webhook request. (If no trust ID is provided then Aodh creates a trust internally, in which case the issue is not present.) However, Aodh makes no attempt to verify that the user creating the alarm is the trustor or has the same rights as the trustor - it also does not attempt to check that the trust is for the same project as the alarm.

The nature of the `trust+http:` alarm notifier is that it allows the user to obtain a token given the ID of a trust for which Aodh is the trustee, since the URL is arbitrary and not limited to services in the Keystone catalog.

### Recommended Actions ###

The patchfile attached to this OSSN when applied, will block use of trust URLs which contain trust ID's and log an error message of "trust URL cannot contain a trust ID.". Any trust action without a trust ID will result in Aodh internally creating a trust ID as before.

#### Patch file instructions ####

{patch CLI line and args}

{Services that require restart}

### Contacts / References ###
Author: Luke Hinds, Red Hat
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0080
Original LaunchPad Bug : https://bugs.launchpad.net/ossn/+bug/1649333
OpenStack Security Project : https://launchpad.net/~openstack-ossg

~~~~END~~~~

Revision history for this message
Luke Hinds (lhinds) wrote :

Sorry noted my own nits

"not that the trust is for the same project as the alarm." = "nor that the trust is for the same project as the alarm."

"Aodh is the alarm engine of the Telemetry project." = "Aodh the alarm engine of the Telemetry project."

Revision history for this message
Zane Bitter (zaneb) wrote :

JD, can you provide the info on how to apply the patch and which services to restart? I will probably get it wrong ;)

Revision history for this message
Luke Hinds (lhinds) wrote :

Second thoughts and in the interest of times, let's just list the services that require a bounce (if there are any). We can then supply the patch, and leave it to the distribution / provider to implement that patch in whatever packaging system they use to install / manage openstack.

I would like to get this out EOB tomorrow, as I will be on leave next week. We can then go public next Friday the 11th and merge into master at the same time.

I will amend the above message to state 5 working days of embargo.

Revision history for this message
Zane Bitter (zaneb) wrote :

IIRC it runs partly under Apache and partly not, and I always guess wrong which part runs which code, so I'd like an Aodh developer to answer that. Julien is on PTO until 11 August, so I added Pradeep who can hopefully find somebody to answer.

Revision history for this message
Zane Bitter (zaneb) wrote :

Forgot to mention, rest of the note LGTM.

Revision history for this message
Pradeep Kilambi (pkilambi) wrote :

The note looks good to me. As for the services to restart, Aodh api is recommended to be run under apache. If So, restarting apache should pick up the change. In cases where its still run as eventlet (in older versions) they will need to restart openstack-aodh-api for centos/RHEL/Suse and aodh-api for ubuntu.

Revision history for this message
Luke Hinds (lhinds) wrote :
Changed in ossn:
status: New → In Progress
Luke Hinds (lhinds)
information type: Private Security → Public
Luke Hinds (lhinds)
Changed in aodh:
status: In Progress → Fix Released
Changed in ossn:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/aodh 5.0.0

This issue was fixed in the openstack/aodh 5.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/aodh 3.0.4

This issue was fixed in the openstack/aodh 3.0.4 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/aodh 4.0.2

This issue was fixed in the openstack/aodh 4.0.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to aodh (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/555500

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to aodh (master)

Reviewed: https://review.openstack.org/555500
Committed: https://git.openstack.org/cgit/openstack/aodh/commit/?id=92182de328d1f088c5f5a68326d2b207b21e06ea
Submitter: Zuul
Branch: master

commit 92182de328d1f088c5f5a68326d2b207b21e06ea
Author: Zane Bitter <email address hidden>
Date: Thu Mar 22 17:12:43 2018 -0400

    Don't include trust IDs in Alarm action output

    It was originally envisaged that we would allow users to specify trust
    IDs in trust+*:// alarm action URLs, and only create trusts internally
    when none was specified. This made storage of trust IDs in the username
    field of the URL part of the API interface, and it made sense that the
    user would have access to the IDs both for trusts they had created
    themselves and those created internally.

    However, this proved to be a security hole that was closed by
    cb90d3ad472bba8d648803ca94a9196dff97f0e8, and now we forbid users from
    passing in a trust ID. Any trust IDs that appear in the URL were created
    internally.

    This causes issues when updating alarms because aodhclient uses a
    read-modify-write pattern to do so. The action URLs output from a GET
    may contain trust IDs, while the input to a PUT is forbidden from
    containing them. This means the data is not round-trippable, so updates
    even to unrelated fields fail.

    Since storage of the trust ID in action URLs is purely an internal
    implementation detail at this point, scrub it out of data returned by
    the REST API. This means the output of e.g. a GET will match the
    original input data, as the user probably expects, and can be fed back
    into a PUT successfully.

    Change-Id: I230c0d80a9440c7d002b9a623048ef715a9df743
    Closes-Bug: #1737186
    Related-Bug: #1649333

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to aodh (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/564020

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to aodh (stable/queens)

Reviewed: https://review.openstack.org/564020
Committed: https://git.openstack.org/cgit/openstack/aodh/commit/?id=149d3ad2193b4d17df801f82a0a6be62dba564db
Submitter: Zuul
Branch: stable/queens

commit 149d3ad2193b4d17df801f82a0a6be62dba564db
Author: Zane Bitter <email address hidden>
Date: Thu Mar 22 17:12:43 2018 -0400

    Don't include trust IDs in Alarm action output

    It was originally envisaged that we would allow users to specify trust
    IDs in trust+*:// alarm action URLs, and only create trusts internally
    when none was specified. This made storage of trust IDs in the username
    field of the URL part of the API interface, and it made sense that the
    user would have access to the IDs both for trusts they had created
    themselves and those created internally.

    However, this proved to be a security hole that was closed by
    cb90d3ad472bba8d648803ca94a9196dff97f0e8, and now we forbid users from
    passing in a trust ID. Any trust IDs that appear in the URL were created
    internally.

    This causes issues when updating alarms because aodhclient uses a
    read-modify-write pattern to do so. The action URLs output from a GET
    may contain trust IDs, while the input to a PUT is forbidden from
    containing them. This means the data is not round-trippable, so updates
    even to unrelated fields fail.

    Since storage of the trust ID in action URLs is purely an internal
    implementation detail at this point, scrub it out of data returned by
    the REST API. This means the output of e.g. a GET will match the
    original input data, as the user probably expects, and can be fed back
    into a PUT successfully.

    Change-Id: I230c0d80a9440c7d002b9a623048ef715a9df743
    Closes-Bug: #1737186
    Related-Bug: #1649333
    (cherry picked from commit 92182de328d1f088c5f5a68326d2b207b21e06ea)

tags: added: in-stable-queens
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.