memorycache.get_client() expects a list from auth_token, not string

[Sam Morrison]

On a new grizzly install I get the following error: (this is cinder but it also happens in nova)

  File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/fault.py", line 73, in __call__
    return req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 441, in __call__
    self._init_cache(env)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 365, in _init_cache
    self._cache = memorycache.get_client(memcache_servers)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/openstack/common/memorycache.py", line 47, in get_client
    return client_cls(memcached_servers, debug=0)
  File "/usr/lib/python2.7/dist-packages/memcache.py", line 192, in __init__
    self.debug = debug
  File "/usr/lib/python2.7/dist-packages/eventlet/corolocal.py", line 45, in __setattr__
    _patch(self)
  File "/usr/lib/python2.7/dist-packages/eventlet/corolocal.py", line 35, in _patch
    thrl.__init__(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/memcache.py", line 195, in __init__
    self.set_servers(servers)
  File "/usr/lib/python2.7/dist-packages/memcache.py", line 239, in set_servers
    for s in servers]
  File "/usr/lib/python2.7/dist-packages/memcache.py", line 1058, in __init__
    self.port = int(hostData.get('port', 11211))
TypeError: int() argument must be a string or a number, not 'NoneType'

The problem is that it's passing it though to oslo as a string when it should be a list

[Dolph Mathews (dolph)]

This issue may be dependent on the version of memcache package - which version do you have installed?

[Sam Morrison (sorrison)]

We have version 1.4.13 memcached server and 1.48 version of python-memcache
I tried with version 1.4.14 of memcached and got the same issue.

The problem isn't with memcache.

In the init_cache method of auth_token it does a:

memcache_servers = self._conf_get('memcache_servers')

Which returns a string, it then passes this to memorycache.get_client which expects a list.

[Dolph Mathews (dolph)]

Related to bug 1172723

[Dolph Mathews (dolph)]

The latest python-memcached (1.53) supports a number of different values (including strings), not just a list:

  http://paste.openstack.org/raw/39262/

I'm not sure if that's different from the version you're using (1.48).

It also looks like it is already defined to be a ListOpt:

 https://github.com/openstack/python-keystoneclient/blob/d1ba41d66f46a5306b8ce9b01a66cee80957321b/keystoneclient/middleware/auth_token.py#L216

What version of keystoneclient are you using?

[Sam Morrison (sorrison)]

We're using 0.2.3, yeah it's weird as it's defined as a ListOpt but it is returning as a string in that part of the code. Maybe something to do with _conf_get ?

This deployment is based on the Ubuntu Cloud Archive so this may need to be raised there as the version of python-memcache doesn't work with keystoneclient.

Although looking at your paste it would seem that setting:
memcache_servers = somehost:11211

Is not valid due to it being interpreted as a string in keystoneclient?

Thanks for your help

[Dolph Mathews (dolph)]

Passing a string with port fails in python-memcached in my paste (not keystoneclient).

What happens if you set it to a list of duplicate hosts? (I have no idea how python-memcached would behave, I'm just wondering if conf would return a list).

  memcache_servers = somehost:11211,somehost:11211

[Sam Morrison (sorrison)]

Yeah tried that but still returns a string. I've debugged this further and it looks to me like it's something to do with how auth_token get's it's configuration.

The conf dictionary gets passed in when the class in instantiated in the pipeline. This conf dictionary is just key value pairs and they are all strings. It never gets passed to oslo.config for processing.

This bit of code always returns it from self.conf

    def _conf_get(self, name):
        # try config from paste-deploy first
        if name in self.conf:
            return self.conf[name]
        else:
            return CONF.keystone_authtoken[name]

In fact CONF.keystone_authtoken is completely empty apart from the settings that have defaults.

I don't know much about oslo.config or wsgi but it would seem to me that something is pretty wrong here?
Does the conf object from paste-deploy need to be sent to oslo.config for processing?

[Ben Nemec (bnemec)]

It doesn't sound like this is a problem in Oslo, so I'm marking it invalid there. Let me know if I'm mistaken.

[Morgan Fainberg (mdrnstm)]

This should no longer be an issue and should be a duplicate of bug 113191, also python-keystoneclient development on auth_token middleware has been frozen. New development is occurring in keystonemiddleware. If this continues to be an issue please reopen/open a new bug against keystonemiddleware.

[Morgan Fainberg (mdrnstm)]

Sorry that was a duplicate of 1353315