metadata service failing on essex-2 with traceback

[Joseph Heck]

The metadata service component of the nova-api is repeatedly failing with tracebacks.

Traceback: (also at http://paste.openstack.org/show/3865/)

2011-12-16 16:32:00,716 INFO nova.api [-] 0.244149s 10.0.0.26 GET /2009-04-04/meta-data/instance-type None:None 500 [Python-urllib/2.7] t
ext/plain text/plain
2011-12-16 16:32:01,721 DEBUG nova.compute.api [1b3384a9-77ef-45d7-b830-a48eefbfb992 None None] Searching by: {'deleted': False, 'fixed_i
p': '10.0.0.26'} from (pid=3586) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:937
2011-12-16 16:32:01,722 DEBUG nova.rpc [1b3384a9-77ef-45d7-b830-a48eefbfb992 None None] Making asynchronous call on network ... from (pid
=3586) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:730
2011-12-16 16:32:01,722 DEBUG nova.rpc [1b3384a9-77ef-45d7-b830-a48eefbfb992 None None] MSG_ID is c4bf2b7029f548eda4b4545a85258bfd from (
pid=3586) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:733
2011-12-16 16:32:02,006 DEBUG nova.compute.api [1b3384a9-77ef-45d7-b830-a48eefbfb992 None None] Searching by: {'deleted': False, 'project
_id': '2'} from (pid=3586) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:937
2011-12-16 16:32:02,038 INFO nova.api [-] 0.316515s 10.0.0.26 GET /2009-04-04/meta-data/instance-type None:None 200 [Python-urllib/2.7] t
ext/plain text/html
2011-12-16 16:32:02,041 DEBUG nova.compute.api [058508f3-3cba-4bff-8fea-477a81365a48 None None] Searching by: {'deleted': False, 'fixed_i
p': '10.0.0.26'} from (pid=3586) get_all /usr/lib/python2.7/dist-packages/nova/compute/api.py:937
2011-12-16 16:32:02,041 DEBUG nova.rpc [058508f3-3cba-4bff-8fea-477a81365a48 None None] Making asynchronous call on network ... from (pid
=3586) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:730
2011-12-16 16:32:02,042 DEBUG nova.rpc [058508f3-3cba-4bff-8fea-477a81365a48 None None] MSG_ID is b0dbdd942eca4612ad451e9ef80f638e from (
pid=3586) multicall /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:733
2011-12-16 16:32:02,242 ERROR nova.api.metadata [058508f3-3cba-4bff-8fea-477a81365a48 None None] Failed to get metadata for ip: 10.0.0.26
(nova.api.metadata): TRACE: Traceback (most recent call last):
(nova.api.metadata): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/metadata/handler.py", line 248, in __call__
(nova.api.metadata): TRACE: meta_data = self.get_metadata(remote_address)
(nova.api.metadata): TRACE: File "/usr/lib/python2.7/dist-packages/nova/api/metadata/handler.py", line 145, in get_metadata
(nova.api.metadata): TRACE: search_opts=search_opts)
(nova.api.metadata): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 981, in get_all
(nova.api.metadata): TRACE: inst_models = self._get_instances_by_filters(context, filters)
(nova.api.metadata): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1015, in _get_instances_by_filters
(nova.api.metadata): TRACE: filters)
(nova.api.metadata): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 194, in get_instance_uuids_by_ip_filter
(nova.api.metadata): TRACE: 'args': args})
(nova.api.metadata): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 45, in call
(nova.api.metadata): TRACE: return get_impl().call(context, topic, msg)
(nova.api.metadata): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 747, in call
(nova.api.metadata): TRACE: rv = list(rv)
(nova.api.metadata): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 715, in __iter__
(nova.api.metadata): TRACE: raise result
(nova.api.metadata): TRACE: RemoteError: Remote error: DetachedInstanceError Parent instance <FixedIp at 0x3b786d0> is not bound to a Session; lazy load operation of attribute 'floating_ips' cannot proceed
(nova.api.metadata): TRACE: [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_carrot.py", line 284, in _process_data\n rval = node_func(context=ctxt, **node_args)\n', u' File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 584, in get_instance_uuids_by_ip_filter\n for floating_ip in fixed_ip.get(\'floating_ips\', []):\n', u' File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/models.py", line 77, in get\n return getattr(self, key, default)\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 168, in __get__\n instance_dict(instance))\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 388, in get\n value = callable_(passive=passive)\n', u' File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 599, in __call__\n (mapperutil.state_str(state), self.key)\n', u"DetachedInstanceError: Parent instance <FixedIp at 0x3b786d0> is not bound to a Session; lazy load operation of attribute 'floating_ips' cannot proceed\n"].
(nova.api.metadata): TRACE:
2011-12-16 16:32:02,244 INFO nova.api [-] 0.202969s 10.0.0.26 GET /2009-04-04/meta-data/instance-id None:None 500 [Python-urllib/2.7] text/plain text/plain

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/2431
Committed: http://github.com/openstack/nova/commit/2f76cd502fcc434b33af8963ac01c2ff6b8c723e
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 2f76cd502fcc434b33af8963ac01c2ff6b8c723e
Author: Vishvananda Ishaya <email address hidden>
Date: Fri Dec 16 13:08:56 2011 -0800

    Adds missing joinedload for vif loading.

     * fixes bug 905398

    Change-Id: I76ffea1437459942f5156d199592931afed4ae1b

[Joseph Heck (heckj)]

Thank you!!!!

[Zhongyue Luo (zyluo)]

Thank you!!!