servers.list, filtering on metadata doesn't work. unicode error

[Sam Morrison]

I'm trying to list servers by filtering on system_metadata or metadata.

I should be able to do something like (looking into the code)

nclient.servers.list(search_opts={'system_metadata': {"some_value": "some_key"}, 'all_tenants': 1})

But this dictionary gets turned into a unicode string. I get a 500 back from nova with the below stack trace in nova-api.

The offending code is in exact_filter in the db api. It is expecting a list of dicts or a single dict when using system_metadata or metadata key when searching. It looks like this used to work but now somewhere higher up is ensuring this is a string.

2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack Traceback (most recent call last):
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/api/openstack/__init__.py", line 125, in __call__
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack return req.get_response(self.application)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack application, catch_exc_info=False)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack return resp(environ, start_response)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack return self.app(env, start_response)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack return resp(environ, start_response)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack return resp(environ, start_response)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack response = self.app(environ, start_response)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack return resp(environ, start_response)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/api/openstack/wsgi.py", line 917, in __call__
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack content_type, body, accept)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/api/openstack/wsgi.py", line 983, in _process_stack
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/api/openstack/wsgi.py", line 1070, in dispatch
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack return method(req=request, **action_args)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/api/openstack/compute/servers.py", line 520, in detail
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack servers = self._get_servers(req, is_detail=True)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/api/openstack/compute/servers.py", line 603, in _get_servers
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack want_objects=True)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/compute/api.py", line 1887, in get_all
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack expected_attrs=expected_attrs)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/compute/cells_api.py", line 224, in _get_instances_by_filters
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack limit=limit, marker=marker, expected_attrs=fields)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/objects/base.py", line 112, in wrapper
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack result = fn(cls, context, *args, **kwargs)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/objects/instance.py", line 629, in get_by_filters
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack use_slave=use_slave)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/db/api.py", line 665, in instance_get_all_by_filters
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack use_slave=use_slave)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/db/sqlalchemy/api.py", line 164, in wrapper
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack return f(*args, **kwargs)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/db/sqlalchemy/api.py", line 2008, in instance_get_all_by_filters
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack filters, exact_match_filter_names)
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack File "/opt/nova/nova/db/sqlalchemy/api.py", line 388, in exact_filter
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack for k, v in value.iteritems():
2014-09-22 11:31:28.916 20196 TRACE nova.api.openstack AttributeError: 'unicode' object has no attribute 'iteritems'

[Christopher Yeoh (cyeoh-0)]

The conversion to unicode occurs during the request to the Nova API as the filter needs to be converted to url parameter request. Nova however does not convert it back to a dict. Need to fix this in compute/api.py

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/123059

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/123059
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ba15ddbf9ba2db3bf4768a91b51d1ca10d2d2833
Submitter: Jenkins
Branch: master

commit ba15ddbf9ba2db3bf4768a91b51d1ca10d2d2833
Author: Chris Yeoh <email address hidden>
Date: Mon Sep 22 16:40:41 2014 +0930

    Fixes server list filtering on metadata

    In order to specify filtering of returned servers based on
    metadata or system metadata a key/value pair string is passed to
    the API. However, the string is never converted to a separate
    key/value dictionary. This results in the db failing to filter
    correctly and causes a stack trace.

    This change correctly converts the string values for system
    metadata and metadata to dictionaries and fixes the compute api
    unittests which previously erroneously assumed that the compute
    api layer would be passed search opts in dict format when
    required.

    Change-Id: I886d5e770fe16fc2194b33c563a131b550f13f59
    Closes-Bug: 1372218