Swift+Glance stops working after changing service password

[differentlocal]

Hello!

We have some trouble with glance+swift storage.

After changing password for account, used for Keystone authentication in Glance and Swift, glance stops working with errors 500 (HTTPInternalServerError) and 401 (HTTPUnauthorized).

I investigated this issue and found that Glance stores image or snapshot location in database (mysql or sqlite) with _full_ swift URI with login and password.

Example:
swift+http://admin%3Aadmin:%PASSWORD%@%HOST%:5000/v2.0/glance/357a3fe7-313c-411c-b0b2-bcd6491d12a1

When we changed password in Keystone, this credentials are outdated BUT Glance STILL USE IT for authenticating in Swift, ignoring glance-api.conf and glance-api-paste. In result, we got HTTP500 error in reply to any request to glance (like glance image-download) and HTTP401 error in glance-api.log

I can find only one method to workaround this - I manually changed this credentials in MySQL. In our situation (5 images) this way is idiotic, but real. But what if we have 500 or 5000 images and snapshots?

I think, glance MUST have any method to change credentials without manual changing thousands of DB records.

[Alex Meade (alex-meade)]

This is a pretty major design flaw that we definitely want to rectify.

It's also even more of a hassle if a deployer is encrypting the stored location information (ie: conf.metadata_encryption_key), if we provide a tool as a solution then we will want to keep that in mind.

[Jay Pipes (jaypipes)]

Agreed. It's a major design flaw that the login and password are even stored in the database at all...

[Flavio Percoco (flaper87)]

This is something that should be handled by every store implementation. Stores should store the remote path without credentials or host information, IMHO. The thing is that those parameters can vary (as the example used for this issue) and also, having them stored in the database represent a security issue as well, again IMHO.

Perhaps we should create a single bug for every store doing this and make this depend on those.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/22182

[Mark Washenberger (markwash)]

While I agree that your experience is the result of a bug in glance, I do want to highlight that there is a valid use case for not having all swift images use the same set of credentials.

In particular, it is quite possible to imagine that you would start out a glance deployment with one default swift store that gradually fills up. Suppose that once that swift store is full, you want to start putting new images in a new swift deployment. It is useful to be able to have all new images going to the new store, while continuing to serve old images out of the old.

However, I think we can and should find a way to support multiple swift storage points for a single glance deployment *without* having to store credentials in the database. And we can probably provide some configuration or tooling around swift store locations that will save the effort the OP was complaining about.

[Flavio Percoco (flaper87)]

I agree with Mark here.

The review proposed as (WIP) is meant to be read as a POC / start idea of how it could work. We should definitely find a way to support both cases or at least a way to update credentials when / if needed.

I suggested on gerrit (on some review comment) to write a blueprint for these changes and use that as reference for addressing all these bugs.,

The other thing to consider is that this issue has an impact on all store backends.

[differentlocal (differentlocal)]

Mark,

I know that. But if I want to change credentials of some storages (for security reasons, after changing IP of swift-proxy or for some other purposes), now I dont have any tool to do it correctly, without direct access to database.

As for me, glance can store credentials (and paths, IPs, and anything other) in database (or in any other place), but must provide some tool/interface for viewing and editing associated storages, credentials and paths.

[Hemanth Makkapati (hemanth-makkapati)]

Isn't this addressed by the fix to remove creds from location? Of course, the fix itself won't help entirely unless the migrations are run.

[Max Lobur (max-lobur)]

 Still reproducible - we hit that on Icehouse code.

[Bobby Yakovich (bgyako)]

Was this ever resolved? I am seeing this in GLance DB swift+http://service%3Aswift:swift_p%40ss@192.168.8.2:5000/v2.0/glance/1568499f-bf25-43dc-a0db-2de7c029ace1

Also having issues with error 500 trying to do basic glance image-list when pointing glance to keystone.
When not pointed to keystone everything works except snapshots that are in private mode not view able by owner.

[Ian Cordasco (icordasc)]

Bobby, it would appear it wasn't resolved.

[Stuart McLaren (stuart-mclaren)]

You can configure credentials in a different way which avoids the password being stored in the database (even for the single tenant swift store).

This can be done via the glance-swift.conf file, eg:

https://github.com/openstack/glance/blob/master/etc/glance-swift.conf.sample

when you do this, the 'reference' is stored in the database rather than the credentials.
The current username/password are swapped in when required.

I think this means you can do a 'rolling' password change by using two users in the same tenant. While changing password some glance nodes will have tenantX:user1 (valid) and some nodes will have tenantX:user2 (also valid). (Though I haven't tested that).

Should we consider moving to this as the default configuration?

(There is no solution to the previously existing entries with passwords - but I would see that as a different bug.)

[Ian Cordasco (icordasc)]

Yeah I think the previously existing entries will need a migration at least.

[Nikhil Komawar (nikhil-komawar)]

Someone needs to pick this work up https://review.openstack.org/#/c/103981/ . I'm running low on cycles at this point so, any effort here would be appreciated.

[Thierry Carrez (ttx)]

Agree cleaning up old entries would be a good thing to do -- not advisory material though

[Flavio Percoco (flaper87)]

triage bump. This is still a valid issue.

I wonder if we can remove the possibility of storing credentials in the location in favor of just using the new way that doesn't require that.

[Nikhil Komawar (nikhil-komawar)]

@Flavio: I think not, not unless we provide a clean db migration for operators that takes care of such locations.

[Brian Rosmaita (brian-rosmaita)]

I think we should do something about this in Ocata using Nikhil's spec proposal (for Kilo!) as a start. I agree with Nikhil that it's essential to provide a way to migrate old-style locations to new-style locations. I don't see time for this to happen in Newton, though.

[Ian Cordasco (icordasc)]

@Brian, perhaps we should do this in Pike instead. ;-)