Kyestone: Auth token not in the request header

[Nicolae Paladi]

Hi, I am using CentOS 6.4, deployed OpenStack Icehouse with Packstack.

After the deployment, they admin user is not authorized for some commands,
e.g. nova list, neutron net-list, etc.

Similar to the bug described in https://bugs.launchpad.net/keystone/+bug/1289935,
however the solution patch does not apply.

Some output:

2014-07-08 16:52:11.063 1649 INFO eventlet.wsgi.server [-] 10.0.230.14 - - [08/Jul/2014 16:52:11] "POST /v2.0/tokens HTTP/1.1" 200 7520 0.201348
2014-07-08 16:52:11.079 1649 DEBUG keystone.middleware.core [-] Auth token not in the request header. Will not build auth context. process_request /usr/lib/python2.6/site-packages/keystone/middleware/core.py:271
2014-07-08 16:52:11.081 1649 DEBUG keystone.common.wsgi [-] arg_dict: {} __call__ /usr/lib/python2.6/site-packages/keystone/common/wsgi.py:181
2014-07-08 16:52:11.086 1649 DEBUG keystone.notifications [-] CADF Event: {'typeURI': 'http://schemas.dmtf.org/cloud/audit/1.0/event', 'initiator': {'typeURI': 'service/security/account/user', 'host': {'agent': 'python-neutronclient', 'address': '10.0.230.14'}, 'id': 'openstack:ca12b898-95bb-4705-8455-6122aae81752', 'name': u'77aabd14a2e1453489dec37d7b174e58'}, 'target': {'typeURI': 'service/security/account/user', 'id': 'openstack:c9028777-2e4b-4c8a-bf07-4175e1c1f5e9'}, 'observer': {'typeURI': 'service/security', 'id': 'openstack:669df929-fca7-4f71-99cf-0e2af4e981fa'}, 'eventType': 'activity', 'eventTime': '2014-07-08T14:52:11.086573+0000', 'action': 'authenticate', 'outcome': 'pending', 'id': 'openstack:0d35b838-3cc9-46ed-bdf6-e384583d0982'} _send_audit_notification /usr/lib/python2.6/site-packages/keystone/notifications.py:289

Identical to the issue mentioned here:
https://www.redhat.com/archives/rdo-list/2014-June/msg00067.html

[Dolph Mathews (dolph)]

What is the request being made to keystone to reproduce this behavior?

[Zoltan Martha (marthazoli)]

Managed to reproduce this on Debian Jessie with Keystone version 2014.1.1-3

For me, every keystone operation hangs indefinitely. For example 'strace keystone tenant-list' hangs here:

stat("/usr/lib/python2.7/encodings/re", 0x7fff38e1b810) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/encodings/re.x86_64-linux-gnu.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/encodings/re.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/encodings/remodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/encodings/re.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/encodings/re.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/usr/lib/python2.7/encodings/unicodedata", 0x7fff38e1b810) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/encodings/unicodedata.x86_64-linux-gnu.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/encodings/unicodedata.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/encodings/unicodedatamodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/encodings/unicodedata.py", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.7/encodings/unicodedata.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
brk(0x2b09000) = 0x2b09000
close(3) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(3, {sa_family=AF_INET, sin_port=htons(5000), sin_addr=inet_addr("192.168.3.2")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll([{fd=3, events=POLLOUT}], 1, 600000) = 1 ([{fd=3, revents=POLLOUT}])
getsockopt(3, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
poll([{fd=3, events=POLLOUT}], 1, 600000) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "POST /v2.0/tokens HTTP/1.1\r\nHost"..., 309, 0, NULL, 0) = 309
fcntl(3, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
poll([{fd=3, events=POLLIN}], 1, 600000

However the API listens on the correct ports:
curl 192.168.3.2:5000/v2.0/
{"version": {"status": "stable", "updated": "2014-04-17T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "http://192.168.3.2:5000/v2.0/", "rel": "self"}, {"href": "http://docs.openstack.org/api/openstack-identity-service/2.0/content/", "type": "text/html", "rel": "describedby"}, {"href": "http://docs.openstack.org/api/openstack-identity-service/2.0/identity-dev-guide-2.0.pdf", "type": "application/pdf", "rel": "describedby"}]}}

Same for port 35357.

[Dolph Mathews (dolph)]

Actually, I totally overlooked that the request was in the logs, to POST /v2.0/tokens. There should not be an X-Auth-Token in a request to POST /v2.0/tokens anyway, so that's completely normal. The rest of the logs in the problem description are also completely normal, so far as I can tell.

Without further details, all I can say regarding the "admin user is not authorized for some commands" is that there must be some sort of other misconfiguration in the deployment - likely something that should have been handled by packstack?

Finally, I don't see how https://www.redhat.com/archives/rdo-list/2014-June/msg00067.html is related?