ERROR n/a (HTTP 401) when issuing "nova volume-list" command

[John D'Esposito]

I am installing Ubuntu 12.04 Folsom (2012.2). I am at the point where I am configuring nova-volumes. I have sourced a file that sets my credentials(OS_USERNAME, OS_PASSWORD etc, I have update the api-paste.ini to reflect admin user, password and service.

When I run the "nova volume-list", I get an ERROR n/a (HTTP 401).

The nova-api.log has the following:

2012-10-10 14:37:22 WARNING keystone.middleware.auth_token [-] Unexpected response from keystone service: {u'error': {u'message': u'The request you have made requires authentication.', u'code': 401, u'title': u'Not Authorized'}}
2012-10-10 14:37:22 WARNING keystone.middleware.auth_token [-] Authorization failed for token c7d4ced421b24813afb58216f686fe8e
2012-10-10 14:37:22 INFO keystone.middleware.auth_token [-] Invalid user token - rejecting request
2012-10-10 14:37:22 INFO nova.osapi_volume.wsgi.server [-] 172.31.254.27 - - [10/Oct/2012 14:37:22] "GET /v1/2eb21fe687e5462ea15af42584b0dd34/volumes/detail HTTP/1.1" 401 461 0.008179

Keystone seems to be working fine for all other services and requests. All of my services are configured to the correct tenant.

Can anyone help with this problem?

Also, how can I turn on DEBUG mode for the nova-api.log?

thanks

John

[Doug Goldstein (cardoe)]

Try editing /etc/nova/api-paste.ini and replace:

admin_tenant_name = %SERVICE_TENANT_NAME%
admin_user = %SERVICE_USER%
admin_password = %SERVICE_PASSWORD%

with the actual values (below is assuming you used the defaults from the Folsom doc):

admin_tenant_name = service
admin_user = nova
admin_password = nova

[Doug Goldstein (cardoe)]

To turn on debugging add:

debug=True

in /etc/nova/nova.conf

[John D'Esposito (johndesp)]

Doug,

Thanks for responding. This is what I currently have in nova.conf and it is not producing debug..

# LOGS/STATE
verbose=True
debug=True
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova

Any idea why?

thanks

John

[John D'Esposito (johndesp)]

I modified the api-paste.ini as instructed.

Still no luck. I think I can get further if I get the log to produce Debug statements.

Does debug=True need to go in the api-paste.ini?

I am not sure where it would go.

thanks

John

[Doug Goldstein (cardoe)]

No it should be in /etc/nova/nova.conf. Have you made sure to restart the related nova services?

restart nova-api
restart nova-cert
restart nova-scheduler
restart nova-compute
restart nova-volume

[Vignesh sankaran (viki-infotech01)]

hi,
 i'm also gettinf the same error but in cinder did u solved your pblm.pls tell me the steps to rectify these error

[freakalad (freakalad)]

I'm hit up with this error too, and I'm having an exceedingly hard time working around it.
For me the issue stems form ambiguity in the docco's: http://docs.openstack.org/trunk/openstack-compute/install/apt/content/compute-minimum-configuration-settings.html
... as well as cryptic error messages & std-out.

I've been using KeePass to keep track of all my auth credential, so I'm *relatively* sure that I've gotten this right.

I've set some environmental vars to be source with .bashrc ; keystone & glance seem OK (I can run those commands to query the system) & `nova-manage service list` reports everything :-)

for `nova --debug list` I get (redacted): http://pastebin.com/Eq2STQKm
& during a `tail -f /var/log/nova/nova-api.log` while the above is executing, I get: http://pastebin.com/JbvDp4Hv

This would suggest that this is a "simple" authentication issue, and that re-entering a new/correct admin_password value in /etc/nova/nova.conf could resolve the issue. (not sure yet how to go about to do this)

In an issue that could possibly be related (but I'm not sure), I keep getting the following message (error?) on my stdout:
DEBUG nova.utils [req-$HASH$ None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.pyc'> from (pid=16650) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:494

If I disable verbose in the nova.conf, this disappears, but that that does not mean the issue is resolved - merely that it's an issue related to the nova service.

Unfortunately this seems to be an issue that's preventing me from continuing my Folsom installation on Ubuntu 12.04

Will probably report back if I manage to reset the nova auth (not MySQL) & it resolves the issue.

[freakalad (freakalad)]

I've tried resetting *all* system/service user passwords to those I have on record in my configs & keepass db (& did a complete reboot to ensure services come up), via the mechanism detailed here: http://docs.openstack.org/developer/keystone/configuration.html
`keystone user-list`
`keystone user-password-update --pass $PASSWORD $USERID`

problem persists

[freakalad (freakalad)]

For reference, my:
* nova.conf : http://pastebin.com/dSWBg9uZ
* api-paste.ini: http://pastebin.com/N0qzvWuj

I've now been staring at these configs for so long, I cannot be certain that I'm not about to break anything.

Update: According to a RackSpace employee(?) on IRC, the aforementioned command will not update the auth/password that could be causing this. Unfortunately there does not seem to be concise information dealing with the change of such information (i.e. in the even of a compromise)

[freakalad (freakalad)]

UPDATE:
I found the following post:
https://answers.launchpad.net/glance/+question/203967 (comment #4)

on a whim, I tried the curl command outputted bu the failing `nova --debug $ARGS`, to ensure the auth token/password is correct & to possibly try different passwords/tokens in the event they fail.

I simply ran those commands, none of them failed with a 401/404, so I ran the original command, and it seems to be working now.

I've not made *any* changes in my configs, so, despite the issue being seemingly OK, I'm still none the wiser as to *what* caused the issue or how to address it in future.

There were a few other options listed via `nova help | grep list` that failed before, that seem OK now

[freakalad (freakalad)]

ADDITIONAL UPDATE:
The DEBUG message to stdout I had before (comment #7) has also mysteriously disappeared once this `nova $ARGS` issue resolved itself:
DEBUG nova.utils [req-$HASH$ None None] backend <module 'nova.db.sqlalchemy.api' from '/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.pyc'> from (pid=16650) __get_backend /usr/lib/python2.7/dist-packages/nova/utils.py:494

I can appreciate that correlation does not necessarily imply causation, but I don't like coincidences I can't explain.
This definitely seems like a bug (one I'm not keen on reproducing atm) - either in the Auth or API somewhere, or possibly in the output (i.e. lack of useful information to assist in troubleshooting)

[Doug Goldstein (cardoe)]

You didn't actually set your username/password and service name in api-paste.ini according to what you pasted and that's likely your issue.

[Thierry Carrez (ttx)]

Does Doug answer solve this for you ?

[Etienne Bersac (bersace03)]

Doug, you're right. I hit this error and fixed it by setting properly api-paste.ini. Looks like documentation and error message is'nt clear enought about that.

http://docs.openstack.org/trunk/openstack-compute/install/apt/content/compute-minimum-configuration-settings.html only covers nova.conf, no mention of api-paste.ini.

[Tom Fifield (fifieldt)]

https://review.openstack.org/#/c/24629/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25872
Committed: http://github.com/openstack/openstack-manuals/commit/1082fdfb0023bcddf5e4c85d569cf98e0e0f0696
Submitter: Jenkins
Branch: master

commit 1082fdfb0023bcddf5e4c85d569cf98e0e0f0696
Author: Lorin Hochstein <email address hidden>
Date: Mon Apr 1 22:59:14 2013 -0400

    Document removing auth from api-paste.ini

    Tell admins to remove auth information from api-paste.ini, and
    specify this all in nova.conf instead.

    Fixes bug 1065211

    Change-Id: I42c968c8a442a372a2567585c8be723da6323a3f