[EDP] Swift credentials passed in plain text

[Trevor McKay]

For Sahara, we support job binaries and data sources in Swift. Job binaries are accessed from the Sahara process, and data sources are accessed from Hadoop at job execution time. Username/password credentials are required for swift access. These credentials might be/are compromised in the following ways:

1) For both job binaries and data sources, objects are created and stored in the Sahara database that contain the path and the associated credentials in plain text. Anyone gaining access to the database can therefore read the username/password credentials stored there with the swift path.

2) For data sources, the credentials are passed as part of the Hadoop job configuration. Currently all Hadoop jobs are run as Oozie workflows. The swift username and password values are set in the workflow.xml file, and are visible to anyone that can access the Oozie UI console, use the Oozie command line to retrieve the workflow.xml, or even use hadoop fs to look at the files uploaded for the job (which include the workflow.xml)

We need a way for Sahara and Hadoop to access swift objects securely, without exposing swift credentials in workflow.xml or storing them in the database in plain text. In the future we will support mechanisms other than Oozie so this is not just an Oozie issue per se.

For further background, here is the Hadoop patch that allows Hadoop to access swift paths. It uses a service suffix in the netlocation portion of the URL to match the URL against credential values in the job configuration. Any solution to this issue will require a new patch to Hadoop itself, as well as changes to the Sahara code base.

https://issues.apache.org/jira/browse/HADOOP-8545

It's been suggested within the Sahara team that we can potentially accomplish this with trusts.

Note, this vulnerability isn't really a secret to anyone observant who is familiar with Sahara EDP, but it is probably better not to trumpet it too loudly.

[John Dickinson (notmyname)]

It sounds like Swift's tempurl feature would work perfectly for this. It's based on a shared secret stored in the Swift account metadata.

http://docs.openstack.org/developer/swift/middleware.html#tempurl

[Robert Clark (robert-clark)]

+1 I think Tempurl could be a solution for most of what you've described :)

[Thierry Carrez (ttx)]

not covered by OpenStack VMT yet (no stable release yet)

[Sergey Lukjanov (slukjanov)]

Yup, it sounds like tempurl could be used instead of credentials in our case.

[Trevor McKay (tmckay)]

Sergey,

  I have a few concerns about it in our case...

1) We have data objects stored for a long time in Sahara -- could be weeks, months, years, who knows. That's okay, though, because we could generate the tempurl on the fly when a job is submitted and put the updated url in the job

2) How do we set the expiration date? Really hard to know when a job will run, or how long it will take. Could be a huge job, and a busy cluster. Or a tiny job on a quiet cluster.

3) Is there a way to invalidate the url when the job is finished?

4) Although we make the problem a lot better, we still have the issue that Oozie/Hadoop logs would expose an authorized URL, so that someone could read (or write in the case of an output url, authorized for PUT) the data sources. True, it's only the particular data source, and not the whole swift instance, but it's still an exposure.

I had an idea about the above -- what if we used tempurls, and then encrypted them using public/private keys created for the hadoop user? Hadoop could decrypt the tempurl, and then access it. Then log entries wouldn't matter. This would require the Hadoop swift patch to recognize an encrypted url, look up the key file for decryption (probably a configured path) and decrypt. Too crazy?

[Michael McCune (mimccune)]

After doing some digging around I'm curious if we can't solve this problem using either trusts and delegation(if we are able to with respect to backward compatibility) or just using Keystone projects.

I can imagine a workflow where Sahara would create a new project on cluster creation. The project would contain the access rights for the Swift data sources. Then as instances get deployed, Sahara would gain a token for each instance and place it on the instance during provisioning. During job execution the instances would use the token to authenticate with Swift for access to the data objects. At job completion, cluster destruction, or when the user is finished, the project and tokens could be revoked from Keystone.

I think this would solve some of the issues surrounding passing a cleartext credential around, and it would give Sahara greater control over the duration of access to the data.

Some difficulties I can see with this method surround the orchestration of the Keystone projects. Especially with regards to the Swift object stores. The Swift objects would need to be included in the project during cluster creation and this might create a disjointed workflow for users if they wish to reuse data objects. Also Hadoop would likely need to be patched to use a Keystone token instead of username/password for communicating with Swift. All of these changes would have an impact on the UI for an entire job process.

Additionally we would still be storing a credential file on the instance. Sahara would just be avoiding passing it through the workflow.xml in favor of using ssh during provisioning to place the credential.

If we are able to use the newer trust/delegation mechanisms then this task could be made easier by allowing Sahara to perform one-time delegation of tokens to the instances.

Another point to consider for the future is multi-tenancy hierarchies, wherein Sahara could create more complex structures of projects to help deliniate which clusters and jobs have access to specific data objects.

[Trevor McKay (tmckay)]

mimccune,

  reusability of data objects is definitely something we want to preserve, in my mind it's the main reason they exist as separate objects and are persisted in the database. Otherwise we would just specify paths on job launch.

  I like the idea of "newer trust/delegation" mechanisms and one time delegation to the instances. That sounds the cleanest to me.

  What do you think of my earlier suggestion, to encrypt credentials using public/private keys set up during cluster generation? The Hadoop patch could decrypt username and password stored in the hadoop config using the private key, and lookup the credentials as it does now. Sahara could even store them encrypted in the database. The rest of the current mechanism stays the same.

  It may not be great, but if something stops us from using one-time delegation it would be a bridge which is at least superior to plaintext.

  Thoughts, everyone?

[Michael McCune (mimccune)]

Trevor,

I agree that hashing the credentials, even if it's just with the known public/private keys is better than leaving them as cleartext. I'm wondering if we can reach a solution where the credentials to the Swift obejct are not needed and the security gating is performed solely by Keysotne. I'm not sure if that's possible but I think it would simplify things.

I also agree that if we can't use the trust/delegation methodology it will be beneficial to hide the credentials. Is it possible we could load an instance with a file based credential that is outside the workflow.xml and change the Hadoop portion to look for credentials outside the xml?

[Andrew Lazarev (alazarev)]

>Sahara could even store them encrypted in the database.
Sahara stores hadoop private key in the same database. So, sahara side will have everything to decrypt and hadoop side will have everything... what is the point? just hashing?

[Trevor McKay (tmckay)]

Why would we store the private key in the Sahara database?

If we generate the private key on the instances during cluster launch, and store only the public key(s) on the node running Sahara, we can encrypt using the public key, and only the instances can decrypt it, right? Isn't that the whole point of public/private keys?

I'm imagining that Sahara encrypts the username/password before writing to the database. It never needs to access data source objects directly.

For job binaries, Sahara still needs a way to get the binaries without exposing credentials

[Michael McCune (mimccune)]

Andrew,

I think the larger point here is to not have cleartext credentials in the workflow.xml. That would be the main point to encrypting the credentials for the instances, then they could perform the decryption at the time of Swift interaction. Thus eliminating the cleartext credentials.

Trevor,

To do what you are suggesting we will need to get the public keys from the instances when they are spawned, probably during provisioning, and then generate the ciphertext credentials for the database/distribution. We will also need to keep track of which credentials belong to each specific instance, unless the instances share a private key(which I don't think they do).

I'm still of the feeling that if we can use Keystone projects to limit access to the data sources, that will give us a clean to way gate all these transactions. Even if we don't use the trust/delegation model we can still create projects that will contain all the instances which need access(including the root Sahara node). What are the limitations to using Keystone in this manner?

[Trevor McKay (tmckay)]

Mike,

> To do what you are suggesting we will need to get the public keys from the instances when they are spawned, probably during > provisioning, and then generate the ciphertext credentials for the database/distribution. We will also need to keep track of > which credentials belong to each specific instance, unless the instances share a private key(which I don't think they do).

Ack, this is what I was thinking. Again, not great but maybe a fallback if we are waiting for something not quite consumable from keystone.

[Michael McCune (mimccune)]

I've been doing some testing with the Keystone trust delegation mechanism and I think there is merit to using it for this solution. Here is a general overview of how we might solve this issue:

1. When a new job that involves Swift objects is created, a trust for the context user’s role is delegated from the context user to the Sahara admin user.

2. A periodic task is started that will acquire a token associated with the delegated trust. This task will place the token in a key/value store on all instances in the cluster. It will update the token based on the timeout value dictated by Keystone.

3. When an instance needs to access a Swift object it uses the token in it’s local key/value store to access that object.

4. When a job has ended, the delegated trust will be revoked from the Sahara admin user, the stored token on all cluster instances will be removed, and the periodic task associated with the job will be terminated.

There are some options we could excercise regarding this solution.

Step 2 could be changed to distribute the trust id instead of an authentication token. In this manner, each instance would need to acquire an authentication token from the trust id. This might be simpler from the perspective of not needing a periodic task in Sahara, but would create much more traffic to the Keystone controller, and would require the Swift-Hadoop plugin to become more aware of when a token needs to be updated.

The key/value store on each instance has not been defined. I was thinking we could use a system wherein the authentication token for a job is stored relative to it's job id, assuming this is unique. In this respect the Swift-Hadoop plugin on each instance would only need to pull the id from it's workflow, look up the token, and then access the Swift objects.

Some issues to consider:

I can imagine a case where a user is part of multiple Keystone projects. It may become difficult to determine what project to delegate trust from. In these cases we might need to end up delegating more trust than is need, or figure out a way to determine which project the Swift endpoint belongs to. I'm not sure there is a clean way to solve this currently.

This also assumes that delegating a "Member" role to the Sahara user will be sufficient to access the Swift objects. There could be configurations where this is not strictly true. In these cases we would need to make sure that there is documentation to support the assumptions we make about a user's access to Swift objects.

This solution will also require that we use Keystone v3 to gain access to the trust/delegation methods.

Another point to consider is how Sahara will keep hold of the trust id. It might be worthwhile to ensure that Sahara only keeps a trust id in memory instead of committing to a database. Because the trust can always be regenerated it might not be necessary for Sahara to store the trust id for long periods of time. In the case of a catastrophic failure of the Sahara controller, it could always regenerate the trust and distribute the tokens as usual.

[Dmitry Mescheryakov (dmitrymex)]

Michael,

I also think that trusts could be a good solution to the problem. But I have a slightly different suggestion in mind:

    1. When a job is submitted by user, Sahara creates a user in keystone (later - 'sahara user') in service (defined in config file) tenant. Also a trust is delegated from context user to the sahara user.

    2. Sahara puts on the VMs credentials for the sahara user and trust id. They are used to access swift for data.

    3. Once the job is done, Sahare deletes Sahara user.

That is close to your suggestion, except I propose to use temporary user for the job. The reason is that in step 2 we need to pass that temporary user's credentials along with trust id to the VMs. Passing credentials of 'persistent' admin user is much riskier that of temporary 'powerless' one.

My main concern here is that:
    a. Swift does not support Keystone v3. There is only a blueprint for that: https://blueprints.launchpad.net/swift/+spec/keystone-v3-support There is already some code merged, but it is unclear how much there is still to do.
    b. Hadoop SwiftFileSystem also does not support trusts. Right now it accepts only username/password.

Regarding some of the concerns you expressed:

-----
>> I can imagine a case where a user is part of multiple Keystone projects. It may become difficult to determine what project to delegate trust from.

What are the difficulties here? Swift endpoint reliably defines the tenant to be used.

-----
>> This solution will also require that we use Keystone v3 to gain access to the trust/delegation methods.

Agree. While all 'modern' OpenStack deployments will have v3 available, there is still much work to be done to implement v3 for Swift and Hadoop SwiftFS.

-----
>> Another point to consider is how Sahara will keep hold of the trust id.

It is my understanding that trust id is not a secret at all. It can be used only by user it is delegated to. So only that user's credentials are secret, not trust id. But in that case your concern applies to the temporary user's credentials and I don't have a good option on how to store them.

[Michael McCune (mimccune)]

Dmitry,

Thanks for taking a look. i originally thought about using a temporary user to assign the trust onto, but after talking with the Keystone developers there is a problem with that method. If Keystone is backed by LDAP then the Sahara user might not have sufficient privilege to create a new user. If we can guarantee that the Sahara user will always have permission to create a new Keystone user, then I agree.

>> The reason is that in step 2 we need to pass that temporary user's credentials along with trust id to the VMs. Passing credentials of 'persistent' admin user is much riskier that of temporary 'powerless' one.

Assuming we aren't using a temporary user, I only propose that we distribute the trust based token generated from the delegation not the Sahara admin user's token. Once a trust has been established it is possible to generate tokens scoped by the trust. These tokens would be safer to distribute as they would be limited to the trust delegation.

>>a. Swift does not support Keystone v3. There is only a blueprint for that: https://blueprints.launchpad.net/swift/+spec/keystone-v3-support There is already some code merged, but it is unclear how much there is still to do.

I don't think we need v3 support in Swift. I have been testing this out with Devstack, and Swift is able to consume the trust based tokens without an issue.

>>b. Hadoop SwiftFileSystem also does not support trusts. Right now it accepts only username/password.

Yes. We would need to change the Hadoop SwiftFileSystem patch to gain the token from the local key/value store instead of taking the credentials from the workflow.xml. Then it would need to change the format of the Swift call to use token authentication instead of credentials.

>>What are the difficulties here? Swift endpoint reliably defines the tenant to be used.

Well, if that's the case then there is no issue. Sahara would determine the project from the endpoint and gain the trust from the user that way. There could be an issue if a job included Swift objects that are from multiple projects. In these cases we would need a trust that could span the projects and I'm not sure that is possible currently. (I will investigate)

Also, I'm not sure how common it would be for a user to have Swift objects in separate projects. If this is a common occurance then we will need to account for it. If it is uncommon, then we will need to document heavily.

>>Agree. While all 'modern' OpenStack deployments will have v3 available, there is still much work to be done to implement v3 for Swift and Hadoop SwiftFS.

I'm not sure how much v3 support we will need from Swift. As long as we are using tokens generated from Keystone by Sahara and passed to the nodes I think we will be able to supply those tokens as part of the Swift ReST call. I've been trying this locally and haven't had issue with Swift taking the tokens.

As for Hadoop SwiftFS, we will need to modify that patch. No question.

>>It is my understanding that trust id is not a secret at all. It can be used only by user it is delegated to. So only that user's credentials are secret, not trust id. But in that case your concern applies to the temporary user's cre...

[Sergey Lukjanov (slukjanov)]

Fix has been committed as part of new feature - https://blueprints.launchpad.net/sahara/+spec/edp-swift-trust-authentication