lazy load operation of attribute 'network' cannot proceed

[Antony Messerli]

(nova.api.openstack): TRACE: Traceback (most recent call last):
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/__init__.py", line 60, in __call__
(nova.api.openstack): TRACE: return req.get_response(self.application)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
(nova.api.openstack): TRACE: application, catch_exc_info=False)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
(nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE: response = self.app(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE: response = self.app(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
(nova.api.openstack): TRACE: return self.func(req, *args, **kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/wsgi.py", line 452, in __call__
(nova.api.openstack): TRACE: action_result = self.dispatch(request, action, args)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/wsgi.py", line 481, in dispatch
(nova.api.openstack): TRACE: return controller_method(req=request, **action_args)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/servers.py", line 61, in detail
(nova.api.openstack): TRACE: servers = self._items(req, is_detail=True)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/servers.py", line 93, in _items
(nova.api.openstack): TRACE: for inst in limited_list]
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/servers.py", line 500, in _build_view
(nova.api.openstack): TRACE: return builder.build(instance, is_detail=is_detail)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/views/servers.py", line 49, in build
(nova.api.openstack): TRACE: server = self._build_detail(inst)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/views/servers.py", line 100, in _build_detail
(nova.api.openstack): TRACE: self._build_addresses(inst_dict, inst)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/views/servers.py", line 163, in _build_addresses
(nova.api.openstack): TRACE: response['addresses'] = self.addresses_builder.build(interfaces)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/views/addresses.py", line 51, in build
(nova.api.openstack): TRACE: network_label = interface['network']['label']
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/db/sqlalchemy/models.py", line 75, in __getitem__
(nova.api.openstack): TRACE: return getattr(self, key)
(nova.api.openstack): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/attributes.py", line 163, in __get__
(nova.api.openstack): TRACE: instance_dict(instance))
(nova.api.openstack): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/attributes.py", line 382, in get
(nova.api.openstack): TRACE: value = callable_(passive=passive)
(nova.api.openstack): TRACE: File "/usr/lib/python2.6/dist-packages/sqlalchemy/orm/strategies.py", line 578, in __call__
(nova.api.openstack): TRACE: (mapperutil.state_str(state), self.key)
(nova.api.openstack): TRACE: DetachedInstanceError: Parent instance <VirtualInterface at 0x5859990> is not bound to a Session; lazy load operation of attribute 'network' cannot proceed
(nova.api.openstack): TRACE:

Running rev 1304. I get this exception in the nova-api when doing a 'nova list'.

[Thierry Carrez (ttx)]

Can't reproduce that... Would you have a bit more information on your setup ? Do all calls to nova list fail that way ?

[Chris Behrens (cbehrens)]

We had a similar issue with lazy loading metadata a while back. It seemingly would happen randomly. To fix it, the instance_get* queries were modified to joinedload('metadata') at the time of the DB query, instead of having it try to be fetched later (lazy load).

So, my guess is that in nova/db/sqlalchemy/api.py, in all of the instance_get* calls, we should change joinedload('virtual_interfaces') to joinedload_all('virtual_interfaces.network').

[Chris Behrens (cbehrens)]

This happens to be fixed in my branch for adding search options to the API. Linked it to the bug...

[Trey Morris (tr3buchet)]

for future reference: the issue is that certain related objects are not being loaded by sqlalchemy api calls, for example using a joinedload() or a joinedload_all()