'No user for access key admin' when using the Openstack API

[Dan Prince]

Using bzr revision 716.

I'm unable to use the openstack-compute Ruby bindings with the latest codebase. I get the following error in nova-api.log when attempting to authenticate with the bindings:

2011-02-23 18:46:44,037 DEBUG nova.utils [-] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/pymodules/python2.6/nova/db/sqlalchemy/api.pyc'> from MainProcess (pid=14442) __get_backend /usr/lib/pymodules/python2.6/nova/utils.py:374
2011-02-23 18:46:44,114 ERROR nova.api.openstack [-] Caught error: No user for access key admin
(nova.api.openstack): TRACE: Traceback (most recent call last):
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/__init__.py", line 53, in __call__
(nova.api.openstack): TRACE: return req.get_response(self.application)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
(nova.api.openstack): TRACE: application, catch_exc_info=False)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
(nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/auth.py", line 51, in __call__
(nova.api.openstack): TRACE: return self.authenticate(req)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/auth.py", line 81, in authenticate
(nova.api.openstack): TRACE: token, user = self._authorize_user(username, key, req)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/auth.py", line 123, in _authorize_user

[Dan Prince (dan-prince)]

Okay. I've tracked this down to commit 709. Reverting that commit seems to fix everything. I suspect issue causing that commit was a result of an issue in either the Python bindings being used or the arguments supplied to them.

I've been using the Ruby bindings for regression testing this for quite a while now (no changes on that front).

[Jay Pipes (jaypipes)]

jpipes@serialcoder:~/repos/nova/trunk$ bzr log -r709
------------------------------------------------------------
revno: 709 [merge]
author: Justin Santa Barbara <email address hidden>
committer: Tarmac
branch nick: nova
timestamp: Mon 2011-02-21 22:39:53 +0000
message:
  The OpenStack API was using the 'secret' as the 'access key'. There
is an 'access key' and there is a 'secret key'. Access key ~=
username. Secret key ~= password. This fix is necessary for the
OpenStack Python API bindings to log in.
------------------------------------------------------------

Are we sure the Ruby binding isn't incorrect as well?

-jay

On Wed, Feb 23, 2011 at 2:30 PM, Dan Prince <email address hidden> wrote:
> Okay. I've tracked this down to commit 709. Reverting that commit seems
> to fix everything. I suspect issue causing that commit was a result of
> an issue in either the Python bindings being used or the arguments
> supplied to them.
>
> I've been using the Ruby bindings for regression testing this for quite
> a while now (no changes on that front).
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/723897
>
> Title:
>  'No user for access key admin' when using the Openstack API
>

[Monsyne Dragon (mdragon)]

I think there is some confusion here as to the meanings of these values.
The python openstack bindings (novatools ) were working before this patch (#709). They are broken now.

The source of this seems to be the oddity that nova stores the openstack 'api-key' as 'User.access' , when it is really a secret key value (the username being User.name) Whereas for the ec2 api access is essentially the user identifier, and secret is the secret key.

There seems to be some conceptual gear grinding here.

[Dan Prince (dan-prince)]

Hey Jay.

The Ruby bindings are based directly on the Cloud Servers Bindings. They haven't change in quite a while and should work fine.

Another guys on our team hit this same issue so I'm fairly positive 709 is causing the issue.

[Jay Pipes (jaypipes)]

So, looks like r709 fixed EC2 and broke OpenStack API... :(

On Wed, Feb 23, 2011 at 3:03 PM, Dan Prince <email address hidden> wrote:
> Hey Jay.
>
> The Ruby bindings are based directly on the Cloud Servers Bindings. They
> haven't change in quite a while and should work fine.
>
> Another guys on our team hit this same issue so I'm fairly positive 709
> is causing the issue.
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/723897
>
> Title:
>  'No user for access key admin' when using the Openstack API
>

[justinsb (justin-fathomdb)]

Had a discussion with Monsyne Dragon on IRC about this. Fixing this is next in my queue.

The issue, I believe, is that the novarc exported user id is now incorrect.

Some background: there are 3 fields in the User object: username, access_key and secret_key. (They get renamed a bit depending on the module)

EC2 uses access_key as the username and secret_key as the secret password

OpenStack pre 709 was using username as the username and access_key (the EC2 username) as the password.

These are incompatible - you can't have a non-secret on one API (EC2) being the secret for another (OpenStack).

709 fixes it so that OpenStack expects the same credentials.

However, I think the export is now broken.

This also now leaves open the question of what 'username' is for.

Monsyne Dragon also raised a concern that perhaps an EC2 secret and OpenStack secret were not compatible (due to different formats or similar) I don't believe that is the case.

I think the best thing is to have one set of credentials across APIs. I think that it's just the export that's broken, so that's what I'm going to look at right now.

I believe if you use the EC2 credentials as your username and password everything should work just as it did before. It's just that the credentials in novarc aren't quite right...

[Dan Prince (dan-prince)]

Hmm. I'm not sure that is the case here. All of this code is in the Openstack API (no EC2 stuff). Additionally only Openstack bindings and tools are mentioned in this thread.

The code in 709 used this:

   user = self.auth.get_user_from_access_key(username)

The following code works:

   user = self.auth.get_user_from_access_key(key)

----

Like Monsyne said above there are some naming differences in nova. Here is how things are stored in SQL:

mysql> select access_key, secret_key from users;
+----------------------+----------------------+
| access_key | secret_key |
+----------------------+----------------------+
| 9rAVqhvvLXCALygDPayB | JnpxuXTZNJfJWcuFJSDn |
+----------------------+----------------------+
1 row in set (0.00 sec)

Here is what shows up in my novarc:

cat novarc | grep "^export NOVA"
export NOVA_CERT=${NOVA_KEY_DIR}/cacert.pem
export NOVA_API_KEY="9rAVqhvvLXCALygDPayB"
export NOVA_USERNAME="admin"
export NOVA_URL="http://172.19.0.3:8774/v1.0/"

In other words the API_KEY is the 'access_key'.

[justinsb (justin-fathomdb)]

Here's the proposed fix, though I haven't tried it yet:
https://code.launchpad.net/~justin-fathomdb/nova/bug723897

As you've already shared your password with us ( ;-) ), would you mind sharing the complete novarc. My hypothesis is that the (secret) NOVA_API_KEY is in fact your (non-secret) EC2 username, and that your EC2 credentials (removing the ':{project}' from the EC2 username) should now work for the OpenStack API.

[justinsb (justin-fathomdb)]

Or... simpler... this should work:

export NOVA_USERNAME="9rAVqhvvLXCALygDPayB"
export NOVA_API_KEY="JnpxuXTZNJfJWcuFJSDn"

As a secondary question, I believe this is what we want the OpenStack Username and Password to be...

[Dan Prince (dan-prince)]

Okay. Couple of things. It appears you are trying to make EC2 and Openstack act alike. The are different API's and will always behave differently.

Where we have the following SQL records:

mysql> select id, access_key, secret_key from users;
+-------+----------------------+----------------------+
| id | access_key | secret_key |
+-------+----------------------+----------------------+
| admin | 9rAVqhvvLXCALygDPayB | JnpxuXTZNJfJWcuFJSDn |
+-------+----------------------+----------------------+
1 row in set (0.00 sec)

I'm pretty sure we want the username to be 'admin' and the API key to be '9rAVqhvvLXCALygDPayB' as it was before 709.

In other words this should be correct:

export NOVA_USERNAME="admin"
export NOVA_API_KEY="9rAVqhvvLXCALygDPayB"

---

The Openstack EC2 API does things a bit differently (as does Amazon's EC2). My config uses the following:

export EC2_ACCESS_KEY="9rAVqhvvLXCALygDPayB:admin"
export EC2_SECRET_KEY="JnpxuXTZNJfJWcuFJSDn"

Which should be correct as well.

[justinsb (justin-fathomdb)]

I think there are 2 issues here:

The credentials exported in novarc are wrong post r709, and I've proposed a bug fix merge to fix that without reverting r709. If you could confirm that the NOVA_USERNAME & API_KEY I suggests above do work for you, then I think this will fix the issue where you can't log in.

There's a second question of what the username & password should be. I think the situation pre r709 was untenable - the Amazon access_key is not normally considered a secret (it is transmitted as a request parameter, and the AWS API could securely be run without SSL because it uses signatures), but it was being used as the secret by OpenStack. I changed it so that the credentials for OpenStack are the same as for EC2, which seems to me a rational approach. I think having different credentials is confusing; what's the advantage in having different credentials based on the front-end API you're using?

[Monsyne Dragon (mdragon)]

Hmm,,

This change still seems problematic to me. When we want to authenticate via the openstack API, we will be passing in a username (i.e. 'admin') and an API key. This change changes the meaning of existing data in the database, and does not have a migration to keep the meaning of that data intact.

WHile I agree that it is somewhat of a design wart that nova use(s/ed) the same value for the (conceptually public) ec2 access key, and the (conceptually private ) openstack api key, this change just moves the conflation to between the username (conceptually an easy-to remember, per user, human readable, not -often mutable identifier) and the ec2 access key which may not be thought of the same way.

I would suggest reverting the openstack api to the pre-change 709 functionality, since that works with current data, and proposing a blueprint to refactor the two api credentials for the DIablo release. My suggestion would be that the two sets of credentials are really entirely separate, and that User should have separate ec2api[_access_key, _secret] and openstack[_username, _api_key] fields

[Dan Prince (dan-prince)]

Sure. I would agree the credentials in novarc need to be fixed post r709. Your branch would do that.

On your second question: My main point is that we want the USERNAME in the rackspace API to match the account id. This is how the Rackspace API behaves and most bindings behave that way (and have examples accordingly).

If you wanted to swap out the implementation so that 'secret_key' is used instead of 'access_key' that would probably be fine. That way at least both implementation would use secret_key in a similar manner.

I really don't like this code in the openstack API:

   user = self.auth.get_user_from_access_key(username)

Sorry I didn't get wind of this sooner. I've been in training for a couple days...

[Dan Prince (dan-prince)]

To be clear... I still think we should revert 709....

[Jay Pipes (jaypipes)]

On Wed, Feb 23, 2011 at 5:22 PM, Monsyne Dragon
<email address hidden> wrote:
> I would suggest reverting the openstack api to the pre-change 709
> functionality, since that works with current data, and proposing a
> blueprint to refactor the two api credentials for the DIablo release. My
> suggestion would be that the two sets of credentials are really entirely
> separate, and that User should have separate ec2api[_access_key,
> _secret] and openstack[_username, _api_key] fields

+5

[Dan Prince (dan-prince)]

I like what Monsyne has proposed as well. Agree this is bigger than we want to handle in this ticket. I still like reverting.

[Brian Waldon (bcwaldon)]

I'm right there with you for reverting it. A great deal of thought needs to go into this before any patches are applied.

[justinsb (justin-fathomdb)]

On the issue in this bug, which is 'can't login':
I think the right thing to do is to apply the patch which fixes novarc to be compatible with r709. If someone affected by the bug could confirm that this solves the problem for them, that would be helpful.

On the issue that 'OpenStack was using questionable credentials pre r709':
There's no fundamental reason that the access_key can't be human readable or the account id or whatever additional meaning you want to give it, for both EC2 and OpenStack. A good reason not to, is to support large-scale clouds where it's likely I can't be 'justin' and therefore have to be 'justin1267' or something equally meaningless to me. The secret is always going to be hard to remember, so it's not like people will be committing these things to memory anyway, so I don't see the benefit in having the username be easy, but if you want to make it be the account id you can do so.

We're not changing the meanings of values within a database, we're using the fields that were already present consistently. It is true that anyone that was using the OpenStack API and had saved their credentials will now need to re-enter them, but better that we do this now than that we do this later when we're just going to affect more users. The Python binding is pretty new, and I don't even know where the Ruby binding is (link please!), so I can't imagine that many people are affected if we do it now. Either way, we do have to do it for security reasons.

If someone wants to propose a blueprint/patch whereby the OpenStack API has a different set of credentials to the EC2 API, feel free. It doesn't make sense to me, because then users would have to know whether a particular application is written to the OpenStack or the EC2 APIs, but do go ahead and propose it - that's what open source is all about!

To borrow Jay's terminology:

What we have post r709 is:
ec2api[api_key, api_secret] and openstack[api_key, api_secret]

I think changing to this would keep everyone happy and is OK, but we break all the EC2 users:
ec2api[username, api_secret] and openstack[username, api_secret]

If we revert, we're back to this, which I don't think is acceptable long-term:
ec2api[api_key, api_secret] and openstack[username, api_key]

I also don't think having separate credentials is acceptable - why should the end-user know/care which API an app is using?

[Brian Waldon (bcwaldon)]

This conversation shouldn't be happening in this form, but through a blueprint. I don't want to see the changes from r709 in trunk because they don't seem very well thought out and broke important functionality. I feel everybody would agree that the pre-709 situation is not good long-term, but a series of small patches is not any better. Let's get a discussion started with a larger group.

[Rick Harris (rconradharris)]

+1 on Brian's comment.

I think we should revert 709 ASAP to get trunk working again so we're not blocking anybody.

Then, offline, through the ML or blueprint, we can figure out the best long-term approach. From the looks of this, we'll certainly have an active discussion on this :-)

[justinsb (justin-fathomdb)]

I think people might be overthinking this. The EC2 API and the OpenStack API should use the same credentials - nothing else makes any sense. r709 does that. Reverting it is a step backwards.

[Brian Waldon (bcwaldon)]

Getting a consensus from a larger group of people is priority number one. In my opinion, revision 709 does not solve fully the problem. If it did, it wouldn't have broken trunk and we wouldn't be discussing how to fix it right now.

And I apologize if I sounded a little harsh in my last comment. I mean no harm :)

[justinsb (justin-fathomdb)]

Trunk is not broken. The behaviour of trunk has changed. The old behaviour was FUBAR. I patched it so that the behaviour was more sane and consistent, although some people would like to take those changes further. People that relied on the old behaviour can't use it any more. To those people: I am sorry. At the same time, the OpenStack API is not yet stable nor is it locked down: expect things to break.

I'm fixing a lot of bugs in the OpenStack API, and my understanding is that it is still okay to be non-backwards compatible at this stage. Once we ship the OpenStack API, then we have to jump through hoops so that we never break anyone, but until we do, we need to stay focused on getting the API right. If that's incorrect, then please let me know.

[Raffael Schmid (raffael-yux)]

Just installed OpenStack from packages on ubuntu lucid 64.

When I want to allow SSH to my instances, using the following command:
root@cloud1:~ # euca-authorize -P tcp -p 22 default

I get the error

Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
EC2ResponseError: 403 Forbidden
403 Forbidden

Access was denied to this resource.

In /var/log/nova/nova-api.log I find
2011-03-28 11:40:11,958 DEBUG nova.auth.manager [-] Looking up user: '2615507a-f5aa-4e73-8500-de3653e27f5c' from (pid=6982) authenticate /usr/lib/pymodules/python2.6/nova/auth/manager.py:268
2011-03-28 11:40:11,963 AUDIT nova.api [-] Authentication Failure: No user for access key 2615507a-f5aa-4e73-8500-de3653e27f5c
2011-03-28 11:40:11,964 INFO nova.api [-] 0.18146s 178.209.34.84 GET /services/Cloud/ None:None 403 [Boto/1.9b (linux2)] text/plain text/html

Database looks like
mysql> select id, access_key, secret_key from users;
+------+--------------------------------------+--------------------------------------+
| id | access_key | secret_key |
+------+--------------------------------------+--------------------------------------+
| nine | 2615507a-f5aa-4e73-8500-de3653e27f5c | afa6cd05-6c...........9-580e65f8e956 |
+------+--------------------------------------+--------------------------------------+

Am I hitting another bug or does this belong to the discussion here? Should I create another Bugreport? I already changed the lines in my .bashrc to reflect http://bazaar.launchpad.net/~justin-fathomdb/nova/bug723897/revision/723 (Not yet in the package, btw?), but did not help.

[Vish Ishaya (vishvananda)]

It looks like your user is a regular user. You need to give a user then netadmin role in order to do network commands.

Enter both of the following commands:

nova-manage role add nine netadmin
nova-manage role add nine netadmin <project_id>

Where project_id is the project user 'nine' is using

Vish

On Mar 28, 2011, at 2:48 AM, Raffael Schmid wrote:

> Just installed OpenStack from packages on ubuntu lucid 64.
>
> When I want to allow SSH to my instances, using the following command:
> root@cloud1:~ # euca-authorize -P tcp -p 22 default
>
> I get the error
>
> Warning: failed to parse error message from AWS: <unknown>:1:0: syntax error
> EC2ResponseError: 403 Forbidden
> 403 Forbidden
>
> Access was denied to this resource.
>
> In /var/log/nova/nova-api.log I find
> 2011-03-28 11:40:11,958 DEBUG nova.auth.manager [-] Looking up user: '2615507a-f5aa-4e73-8500-de3653e27f5c' from (pid=6982) authenticate /usr/lib/pymodules/python2.6/nova/auth/manager.py:268
> 2011-03-28 11:40:11,963 AUDIT nova.api [-] Authentication Failure: No user for access key 2615507a-f5aa-4e73-8500-de3653e27f5c
> 2011-03-28 11:40:11,964 INFO nova.api [-] 0.18146s 178.209.34.84 GET /services/Cloud/ None:None 403 [Boto/1.9b (linux2)] text/plain text/html
>
> Database looks like
> mysql> select id, access_key, secret_key from users;
> +------+--------------------------------------+--------------------------------------+
> | id | access_key | secret_key |
> +------+--------------------------------------+--------------------------------------+
> | nine | 2615507a-f5aa-4e73-8500-de3653e27f5c | afa6cd05-6c...........9-580e65f8e956 |
> +------+--------------------------------------+--------------------------------------+
>
>
> Am I hitting another bug or does this belong to the discussion here? Should I create another Bugreport? I already changed the lines in my .bashrc to reflect http://bazaar.launchpad.net/~justin-fathomdb/nova/bug723897/revision/723 (Not yet in the package, btw?), but did not help.
>
>
> ** Changed in: nova
> Status: Fix Committed => In Progress
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/723897
>
> Title:
> 'No user for access key admin' when using the Openstack API
>
> Status in OpenStack Compute (Nova):
> In Progress
>
> Bug description:
> Using bzr revision 716.
>
> I'm unable to use the openstack-compute Ruby bindings with the latest
> codebase. I get the following error in nova-api.log when attempting to
> authenticate with the bindings:
>
>
> 2011-02-23 18:46:44,037 DEBUG nova.utils [-] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/pymodules/python2.6/nova/db/sqlalchemy/api.pyc'> from MainProcess (pid=14442) __get_backend /usr/lib/pymodules/python2.6/nova/utils.py:374
> 2011-02-23 18:46:44,114 ERROR nova.api.openstack [-] Caught error: No user for access key admin
> (nova.api.openstack): TRACE: Traceback (most recent call last):
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/__init__.py", li...

[Raffael Schmid (raffael-yux)]

I did this. Still the same error.

The only "special"-thing i did was editing the networks-table-content to fit to my wanted network layout.

[Raffael Schmid (raffael-yux)]

ok, fixed in the new package.