Nova API Memcached Encoding Error

[Daneyon Hansen]

I am trying to use memcached with Nova Grizzly and I get the following error in nova-api log:

2013-03-22 19:22:03.921 ERROR nova.api.openstack [req-3b839f39-ebe5-460a-9499-736323132e43 cd9a6b253278436eb3fcbe4f92a96694 3eccdb2a9331419c96ac9ff336110b65] Caught error: Keys must be str()'s, not unicode. Convert your unicode strings using mystring.encode(charset)!
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack Traceback (most recent call last):
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 81, in __call__
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack return req.get_response(self.application)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack application, catch_exc_info=False)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 451, in __call__
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack return self.app(env, start_response)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack response = self.app(environ, start_response)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack return resp(environ, start_response)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 895, in __call__
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack content_type, body, accept)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 982, in _process_stack
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack request, action_args)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 867, in post_process_extensions
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack **action_args)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/extended_availability_zone.py", line 72, in detail
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack self._extend_server(context, server, db_instance)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/extended_availability_zone.py", line 53, in _extend_server
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack server[key] = self._get_host_az(context, instance)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/extended_availability_zone.py", line 44, in _get_host_az
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack az = self.mc.get(cache_key)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/memcache.py", line 862, in get
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack return self._get('get', key)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/memcache.py", line 813, in _get
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack self.check_key(key)
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/memcache.py", line 1014, in check_key
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack "Keys must be str()'s, not unicode. Convert your unicode "
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack MemcachedStringEncodingError: Keys must be str()'s, not unicode. Convert your unicode strings using mystring.encode(charset)!
2013-03-22 19:22:03.921 10563 TRACE nova.api.openstack

I have set the memcached_servers flag in nova.conf:

memcached_servers=192.168.220.41:11211,192.168.220.42:11211,192.168.220.43:11211

I receive the same error even when I front-end the memcached servers using a load-balancer vitual IP:

memcached_servers=192.168.220.40:11211

I have yet to test, but I recall seeing a similar problem in Folsom with consoleauth/manager.py

[Daneyon Hansen (danehans)]

Also, everything works fine when I remove the memcached_servers flag.

[Mark McLoughlin (markmc)]

If memcached_servers is set, Nova is basically using memcache.Client directly rather than any oslo code

It looks to me like Nova is passing a unicode key and memcache.Client doesn't allow it

I'd be happy enough with Oslo's memcache client using memcache.Client.check_key() to enforce the same requirements

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/25496

[Davanum Srinivas (DIMS) (dims-v)]

Mark,

For Oslo, what should be done? add a check_key() method which makes checks similar to that in memcache.Client.check_key()?

thanks,
dims

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/25496
Committed: http://github.com/openstack/nova/commit/a11c41b48dff83a91eaa65630cc6aeb0e51e6461
Submitter: Jenkins
Branch: master

commit a11c41b48dff83a91eaa65630cc6aeb0e51e6461
Author: Dan Florea <email address hidden>
Date: Tue Mar 26 23:00:32 2013 -0700

    Convert host value from unicode to a string.

    The memcached API expects string format, not unicode.
    Convert the host value to a string before creating
    the cache_key and passing it to memcached.

    Fixes: bug #1158958
    Change-Id: I26e86b7fb61b4b3380a14686bd2faa9f58017999

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/26867

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/26867
Committed: http://github.com/openstack/nova/commit/232f874d3663248f2a82eb138c0e7bd142593746
Submitter: Jenkins
Branch: stable/grizzly

commit 232f874d3663248f2a82eb138c0e7bd142593746
Author: Dan Florea <email address hidden>
Date: Tue Mar 26 23:00:32 2013 -0700

    Convert host value from unicode to a string.

    The memcached API expects string format, not unicode.
    Convert the host value to a string before creating
    the cache_key and passing it to memcached.

    Fixes: bug #1158958
    Change-Id: I26e86b7fb61b4b3380a14686bd2faa9f58017999
    (cherry picked from commit a11c41b48dff83a91eaa65630cc6aeb0e51e6461)