Volumes with metadata can't be returned when admin metadata is present

[Geraint North]

Since the admin metadata changes were merged as part of read-only volumes (https://github.com/openstack/cinder/commit/900851ffe36e3f4143ec9126c5bf69dea0853114#diff-22) I cannot retrieve details of my volumes if those volumes already contain metadata, due to what I think is a bug in the code that merges the new admin metadata with existing metadata.

The error, reported in Cinder's api.log is as follows:

2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/cinder/api/middleware/fault.py", line 73, in __call__
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault return req.get_response(self.application)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/webob/request.py", line 1296, in send
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault application, catch_exc_info=False)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/webob/request.py", line 1260, in call_application
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault app_iter = application(self.environ, start_response)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault return resp(environ, start_response)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/auth_token.py", line 461, in __call__
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault return self.app(env, start_response)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault return resp(environ, start_response)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault return resp(environ, start_response)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/routes/middleware.py", line 131, in __call__
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault response = self.app(environ, start_response)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/webob/dec.py", line 144, in __call__
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault return resp(environ, start_response)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/webob/dec.py", line 130, in __call__
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault resp = self.call_func(req, *args, **self.kwargs)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/webob/dec.py", line 195, in call_func
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault return self.func(req, *args, **kwargs)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/cinder/api/openstack/wsgi.py", line 824, in __call__
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault content_type, body, accept)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/cinder/api/openstack/wsgi.py", line 872, in _process_stack
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault action_result = self.dispatch(meth, request, action_args)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/cinder/api/openstack/wsgi.py", line 948, in dispatch
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault return method(req=request, **action_args)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/cinder/api/v1/volumes.py", line 290, in index
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault return self._items(req, entity_maker=_translate_volume_summary_view)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/cinder/api/v1/volumes.py", line 317, in _items
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault self._add_visible_admin_metadata(context, volume)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib/python2.6/site-packages/cinder/api/v1/volumes.py", line 252, in _add_visible_admin_metadata
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault orig_meta.append({'key': key, 'value': value})
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/collections.py", line 1046, in append
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault item = __set(self, item, _sa_initiator)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/collections.py", line 1021, in __set
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault item = getattr(executor, 'fire_append_event')(item, _sa_initiator)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/collections.py", line 714, in fire_append_event
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault item, initiator)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/attributes.py", line 800, in fire_append_event
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault value = fn(state, value, initiator or self)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 35, in append
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault item_state = attributes.instance_state(item)
2013-09-03 08:33:46.145 15554 TRACE cinder.api.middleware.fault AttributeError: 'dict' object has no attribute '_sa_instance_state'

I think that the problem is that we are trying to add a dict - {'key': key, 'value': value} to an sqlalchemy.orm.collections.InstrumentedList that is configured to only allow Volume.volume_metadata.

I tried adding the VolumeAdminMetadata objects straight onto the list but got the following error:

DBError: Attempting to flush an item of type <class 'cinder.db.sqlalchemy.models.VolumeAdminMetadata'> as a member of collection "Volume.volume_metadata". Expected an object of type <class 'cinder.db.sqlalchemy.models.VolumeMetadata'> or a polymorphic subclass of this type.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/44906

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/44906
Committed: http://github.com/openstack/cinder/commit/fbbf79f6cadbb1d2b7ce615e9fdb9a0e0f114729
Submitter: Jenkins
Branch: master

commit fbbf79f6cadbb1d2b7ce615e9fdb9a0e0f114729
Author: Zhi Yan Liu <email address hidden>
Date: Tue Sep 3 22:40:49 2013 +0800

    Turn db model object into a primitive object to avoid error

    In volume API controller we need turn db model object into a primitive
    object to avoid error when it adding visible administrator metadata to
    volume's metadata dict.

    Fixes bug: 1220232

    Change-Id: I5f6c4fc8dd6ca3af02db577b37dcaeee92bb42e5
    Signed-off-by: Zhi Yan Liu <email address hidden>