keystone returns 'NoneType' object is unsubscriptable

[Robin Goldstone]

I am trying to install the RDO OpenStack distribution on a RHEL 6.4 system. The install is failing during keystone configuration as follows:
^[[1;35merr: /Stage[main]/Cinder::Keystone::Auth/Keystone_service[cinder]: Could not evaluate: Execution of '/usr/bin/keystone --endpoint http://127.0.0.1:35357/v2.0/ service-list' returned 1: 'NoneType' object is unsubscriptable

Running this command interactively produces the same error:
# brutus1 /etc/keystone > export OS_SERVICE_TOKEN=49c056f8688540fcb933cf7335537676
# brutus1 /etc/keystone > export OS_SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0
# brutus1 /etc/keystone > keystone service-list
'NoneType' object is unsubscriptable

The keystone service appears to be running and connected to the mysql database:
# brutus1 /root > ps -ef|grep keystone
keystone 8451 1 0 17:20 ? 00:00:00 /usr/bin/python /usr/bin/keystone-all --config-file /etc/keystone/keystone.conf
# brutus1 /root > lsof -i|grep keystone
keystone- 8451 keystone 4u IPv4 21614 0t0 TCP *:35357 (LISTEN)
keystone- 8451 keystone 6u IPv4 21616 0t0 TCP *:commplex-main (LISTEN)
keystone- 8451 keystone 8u IPv4 21961 0t0 TCP ebrutus1:54531->ebrutus1:mysql (ESTABLISHED)

The keystone mysql database appears to have been properly initialized:

mysql> use keystone;
Database changed
mysql> show tables;
+------------------------+
| Tables_in_keystone |
+------------------------+
| credential |
| domain |
| ec2_credential |
| endpoint |
| group |
| group_domain_metadata |
| group_project_metadata |
| migrate_version |
| policy |
| project |
| role |
| service |
| token |
| trust |
| trust_role |
| user |
| user_domain_metadata |
| user_group_membership |
| user_project_metadata |
+------------------------+
19 rows in set (0.00 sec)

mysql> select * from service;
Empty set (0.00 sec)

I cranked up the debug level in the keystone log file and here is what I see:
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** REQUEST ENVIRON ********************
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SCRIPT_NAME = /v2.0
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] webob.adhoc_attrs = {'response': <Response at 0x2373510 200 OK>}
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] REQUEST_METHOD = GET
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] PATH_INFO = /OS-KSADM/services
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SERVER_PROTOCOL = HTTP/1.0
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_X_AUTH_TOKEN = 49c056f8688540fcb933cf7335537676
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_USER_AGENT = python-keystoneclient
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] eventlet.posthooks = []
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SERVER_NAME = 127.0.0.1
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] REMOTE_ADDR = 127.0.0.1
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] eventlet.input = <eventlet.wsgi.Input object at 0x236b8d0>
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.url_scheme = http
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] SERVER_PORT = 35357
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.input = <eventlet.wsgi.Input object at 0x236b8d0>
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_HOST = 127.0.0.1:35357
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.multithread = True
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_ACCEPT = */*
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.version = (1, 0)
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] openstack.context = {'token_id': '49c056f8688540fcb933cf7335537676', 'is_admin': True}
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] GATEWAY_INTERFACE = CGI/1.1
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.run_once = False
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.errors = <open file '<stderr>', mode 'w' at 0x2aaaaaae81e0>
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] wsgi.multiprocess = False
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] CONTENT_TYPE = text/plain
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] HTTP_ACCEPT_ENCODING = identity, deflate, compress, gzip
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi]
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** REQUEST BODY ********************
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi]
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] arg_dict: {}
2013-09-04 17:37:48 INFO [sqlalchemy.engine.base.Engine] SELECT service.id AS service_id, service.type AS service_type, service.extra AS service_extra
FROM service
2013-09-04 17:37:48 INFO [sqlalchemy.engine.base.Engine] ()
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** RESPONSE HEADERS ********************
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] Vary = X-Auth-Token
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] Content-Type = application/json
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] Content-Length = 25
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi]
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] ******************** RESPONSE BODY ********************
2013-09-04 17:37:48 DEBUG [keystone.common.wsgi] {"OS-KSADM:services": []}
2013-09-04 17:37:48 INFO [access] 127.0.0.1 - - [05/Sep/2013:00:37:48 +0000] "GET http://127.0.0.1:35357/v2.0/OS-KSADM/services HTTP/1.0" 200 25
2013-09-04 17:37:48 DEBUG [eventlet.wsgi.server] 127.0.0.1 - - [04/Sep/2013 17:37:48] "GET /v2.0/OS-KSADM/services HTTP/1.1" 200 153 0.008193

Next I completely wiped out the RDO installation and reinstalled with the same result.

I reported this to the Openstack mailing list and got back a response from Sean McCully asking me to try curl, which I did successfully as follows:
# brutus1 /root > curl -X POST -H "X-Auth-Token: 49c056f8688540fcb933cf7335537676" -H "Content-type: application/json" --data '{"OS-KSADM:service": {"name":"Test","type":"test","extra":"{}"}}' http://0.0.0.0:35357/v2.0/OS-KSADM/services
{"OS-KSADM:service": {"type": "test", "name": "Test", "id": "6d54ea4d2752435aba19893a9de24695"}}
# brutus1 /root > curl -X GET -H "X-Auth-Token: 49c056f8688540fcb933cf7335537676" http://0.0.0.0:35357/v2.0/OS-KSADM/services
{"OS-KSADM:services": [{"type": "test", "name": "Test", "id": "6d54ea4d2752435aba19893a9de24695"}]}

But access via the keystone client still fails with the same error:
# brutus1 /root > keystone service-list
'NoneType' object is unsubscriptable

[Adam Young (ayoung)]

You should also report this bug against RDO in RH's Bugzilla. It is likely caused by an out-of-sync version of Keystone Client. You can link to this bug report. We'll keep this bug open for a short while, but it is likely to be closed in the near future as the behavior is not seen upstream.

[Robin Goldstone (goldstone1)]

Hi Adam,

Thank you for looking at this. I have reported this on the RDO Forum and
not received any response. Can you tell me how I get to the RDO Bugzilla?
I am at a total standstill trying to get OpenStack installed and have been
frustrated by their lack of response, which is why I decided to try
reporting it upstream. If you have any suggestions on how I might debug
this further, I would be grateful. Otherwise I guess I will have to wait
for the RDO folks to respond, hopefully in my lifetimeŠ

Regards,
Robin

On 9/6/13 11:01 AM, "Adam Young" <email address hidden> wrote:

>You should also report this bug against RDO in RH's Bugzilla. It is
>likely caused by an out-of-sync version of Keystone Client. You can
>link to this bug report. We'll keep this bug open for a short while,
>but it is likely to be closed in the near future as the behavior is not
>seen upstream.
>
>--
>You received this bug notification because you are subscribed to the bug
>report.
>https://bugs.launchpad.net/bugs/1221331
>
>Title:
> keystone returns 'NoneType' object is unsubscriptable
>
>Status in OpenStack Identity (Keystone):
> New
>Status in Python client library for Keystone:
> New
>
>Bug description:
> I am trying to install the RDO OpenStack distribution on a RHEL 6.4
>system. The install is failing during keystone configuration as follows:
> ^[[1;35merr:
>/Stage[main]/Cinder::Keystone::Auth/Keystone_service[cinder]: Could not
>evaluate: Execution of '/usr/bin/keystone --endpoint
>http://127.0.0.1:35357/v2.0/ service-list' returned 1: 'NoneType' object
>is unsubscriptable
>
> Running this command interactively produces the same error:
> # brutus1 /etc/keystone > export
>OS_SERVICE_TOKEN=49c056f8688540fcb933cf7335537676
> # brutus1 /etc/keystone > export
>OS_SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0
> # brutus1 /etc/keystone > keystone service-list
> 'NoneType' object is unsubscriptable
>
> The keystone service appears to be running and connected to the mysql
>database:
> # brutus1 /root > ps -ef|grep keystone
> keystone 8451 1 0 17:20 ? 00:00:00 /usr/bin/python
>/usr/bin/keystone-all --config-file /etc/keystone/keystone.conf
> # brutus1 /root > lsof -i|grep keystone
> keystone- 8451 keystone 4u IPv4 21614 0t0 TCP *:35357
>(LISTEN)
> keystone- 8451 keystone 6u IPv4 21616 0t0 TCP
>*:commplex-main (LISTEN)
> keystone- 8451 keystone 8u IPv4 21961 0t0 TCP
>ebrutus1:54531->ebrutus1:mysql (ESTABLISHED)
>
> The keystone mysql database appears to have been properly initialized:
>
> mysql> use keystone;
> Database changed
> mysql> show tables;
> +------------------------+
> | Tables_in_keystone |
> +------------------------+
> | credential |
> | domain |
> | ec2_credential |
> | endpoint |
> | group |
> | group_domain_metadata |
> | group_project_metadata |
> | migrate_version |
> | policy |
> | project |
> | role |
> | service |
> | token |
> | trust |
> | trust_role |
> | user ...

[Dolph Mathews (dolph)]

There's no reason to avoid changing the status of a bug if it's known to be inaccurate. Discussion can still occur here and the permalink will work either way.

I assume Adam is correct... you can workaround an old package by simply installing the latest version of keystoneclient from pypi:

  > pip install python-keystoneclient

https://pypi.python.org/pypi/python-keystoneclient/

The latest client should be compatible with much older releases of keystone.

[Robin Goldstone (goldstone1)]

Thank you, Dolph. I would be glad to try out a newer keystone client. The one being distributed with RDO is version 0.2.3-7 while the latest appears to be 0.3.2. Unfortunately I know very little about python and the pip command is not installed on my system. I was wondering if this package is available in RPM format. If not, can you offer any additional assistance on how I can install this? I downloaded the tarball but it doesn't have the typical configure/make files that I am used to.

[Dolph Mathews (dolph)]

This should get you pip:

  $ sudo yum install python-pip

If not:

  http://www.pip-installer.org/en/latest/installing.html

[Dolph Mathews (dolph)]

Or extract python-keystoneclient-0.3.2.tar.gz and run:

  $ python setup.py install

[Robin Goldstone (goldstone1)]

I can't do a yum install because none of the repositories I have access to have python-pip in them. I tried the other method you suggested but no joy:

 brutus1 /root/python-keystoneclient-0.3.2 > python setup.py install

Installed /root/python-keystoneclient-0.3.2/pbr-0.5.21-py2.6.egg
Searching for pip>=1.0
Reading http://pypi.python.org/simple/pip/
Best match: pip 1.4.1
Downloading https://pypi.python.org/packages/source/p/pip/pip-1.4.1.tar.gz#md5=6afbb46aeb48abac658d4df742bff714
Processing pip-1.4.1.tar.gz
Running pip-1.4.1/setup.py -q bdist_egg --dist-dir /var/tmp/root/easy_install-4K2ywY/pip-1.4.1/egg-dist-tmp-3XI2R_
warning: no files found matching '*.html' under directory 'docs'
warning: no previously-included files matching '*.rst' found under directory 'docs/_build'
no previously-included directories found matching 'docs/_build/_sources'

Installed /root/python-keystoneclient-0.3.2/pip-1.4.1-py2.6.egg
running install
/usr/bin/python: No module named pip
error: /usr/bin/python -m pip.__init__ install 'pbr>=0.5.21,<1.0' 'argparse' 'iso8601>=0.1.4' 'PrettyTable>=0.6,<0.8' 'requests>=1.1' 'simplejson>=2.0.9' 'six' 'oslo.config>=1.1.0' 'netaddr' returned 1

[Dolph Mathews (dolph)]

Hrm, that's a little outside my scope of expertise, but I find it alarming that it appears to install pip and then immediately can't find it (?!). That behavior *may* be a bug in pbr... I'd suggest filing the entirely of comment #7 over there and seeing what they have to say: https://bugs.launchpad.net/pbr

[Robin Goldstone (goldstone1)]

Hmm. I really don't want to go down the road of opening another bug report, this time on a piece of software that I never wanted to use to begin with. I did manage to find a newer version of the python-keystone client RPM in the RDO Havana repository. I downloaded and installed version 0.2.5-2 (currently running 0.2.3-7) but running it produced a bunch of errors suggesting that I can't just "drop in" a new version of this client and expect it to work with the rest of the RDO distribution:
# brutus1 /root/robing/rpms > keystone service-list
Traceback (most recent call last):
  File "/usr/bin/keystone", line 5, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2659, in <module>
    parse_requirements(__requires__), Environment()
  File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
    raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: prettytable>=0.6,<0.8

I have reported this bug to RedHat, bugid #1005440. I cross referenced this bug report as suggested. So I guess I will hope that RedHat can help sort this out. Thank you for your help!

[Robin Goldstone (goldstone1)]

I went through the installation again on a "fresh" OS image. This time I am back to the first error I reported on the RDO forum namely packstack appears to install everything correctly but in reality it does absolutely nothing because puppet is not installed on my system. On the RDO forum they told me that packstack is supposed to install puppet if it is not already installed but as you can see, it does not do that. Not only does it fail to install puppet, but it apparently does not even check return codes when it runs the puppet commands. If it did, you would think it would notice that puppet is not even installed. Instead it reports "installation completed successfully" when in fact nothing is installed.
# brutus1 /root > yum install http://repos.fedorapeople.org/repos/openstack/openstack-havana/epel-6/python-keystoneclient-0.3.2-3.el6.noarch.rpm
Loaded plugins: changelog, priorities, ps, refresh-packagekit, security
73 packages excluded due to repository priority protections
Setting up Install Process
python-keystoneclient-0.3.2-3.el6.noarch.rpm | 151 kB 00:00
Examining /var/tmp/yum-root-1w2Xeq/python-keystoneclient-0.3.2-3.el6.noarch.rpm: 1:python-keystoneclient-0.3.2-3.el6.noarch
Marking /var/tmp/yum-root-1w2Xeq/python-keystoneclient-0.3.2-3.el6.noarch.rpm as an update to 1:python-keystoneclient-0.2.3-7.el6.noarch
Resolving Dependencies
--> Running transaction check
---> Package python-keystoneclient.noarch 1:0.2.3-7.el6 will be updated
---> Package python-keystoneclient.noarch 1:0.3.2-3.el6 will be an update
--> Processing Dependency: python-httplib2 for package: 1:python-keystoneclient-0.3.2-3.el6.noarch
--> Running transaction check
---> Package python-httplib2.noarch 0:0.7.7-1.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================
 Package Arch Version Repository Size
========================================================================================================================
Updating:
 python-keystoneclient noarch 1:0.3.2-3.el6 /python-keystoneclient-0.3.2-3.el6.noarch 592 k
Installing for dependencies:
 python-httplib2 noarch 0.7.7-1.el6 epel 70 k

Transaction Summary
========================================================================================================================
Install 1 Package(s)
Upgrade 1 Package(s)

Total size: 662 k
Total download size: 70 k
Is this ok [y/N]: y
Downloading Packages:
python-httplib2-0.7.7-1.el6.noarch.rpm | 70 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : python-httplib2-0.7.7-1.el6.noarch 1/3
  Updating : 1:python-keystoneclient-0.3.2-3.el6.noarch ...

[Robin Goldstone (goldstone1)]

Sorry, the first part of my posting above was a repeat of some info I posted previously when I tried to update to the newer keystone client. The new stuff starts with the line that says: "Clean reinstall on brutus2:"

[Dolph Mathews (dolph)]

It looks like the above should be filed against RDO's bug tracker:

  https://bugzilla.redhat.com/page.cgi?id=browse.html&product=RDO&bug_status=open&tab=recents

[Robin Goldstone (goldstone1)]

My apologies, I pasted it in the wrong window. :(

I am going to close this bug report since I am working this directly with the RDO folks now.

[Robin Goldstone (goldstone1)]

Hmm. Not sure I can close this myself. Can you go ahead and close this out? The RDO bug ID is 1005440.
Regards,
Robin

[Dolph Mathews (dolph)]

This is already "invalid"/closed in keystone -- comments are never blocked or anything

[Andy Riebs (andy-candooz)]

In my case, the problem proved to be that the FQDN wasn't available. "hostname --fqdn" is a quick check for this situation.

In my case, the fix was to reverse the entries in /etc/host, so that the long form of the hostname appears first; i.e.,

127.0.0.1 localhost.localdomain localhost

[Robin Goldstone (goldstone1)]

Thank you, Andy, for suggesting this possibility. It is unfortunately not my issue:
brutus1@robing:hostname --fqdn
brutus1.llnl.gov

My issue turned out to be the result of missing requisites, which packstack failed to install (because my repo was incomplete) but did not report any errors, instead blindly proceeding with the rest of the installation and leaving me with a broken system.

I am working with the RDO folks to try to resolve the underlying issue with their packaging.
Regards,
Robin