nova-api throws "lazy load operation of attribute 'network' cannot proceed" under stress

[Joe Gordon]

Test Environment:

Nova stable/diablo (d3f32f742c1622f2d8b4b6fc5f42b04ea5da5664) on 2 machines. Launch 20 VMs (in parallel) at a time take them down and launch another 20.

Error:

When run euca-describe-instances during and after the VM launch process it occasionally says

# euca-describe-instances
UnknownError: An unknown error has occurred. Please try your request again.

"Parent instance <FixedIp at 0x5dd6390> is not bound to a Session; lazy load operation of attribute 'network' cannot proceed"

This error is similar to Bug #785816 and Bug #815957

StackTrace (from nova-api.log)

2011-12-20 13:07:33,881 ERROR nova.api [69c94d83-24fc-403c-b550-ac528ac80414 nova project_503] Unexpected error raised: Parent instance <FixedIp at 0x5dd6390> is not bound to a Session; lazy load operation of attribute 'network' cannot proceed
(nova.api): TRACE: Traceback (most recent call last):(nova.api): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/api/ec2/__init__.py", line 348, in __call__
(nova.api): TRACE: result = api_request.invoke(context)(nova.api): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/api/ec2/apirequest.py", line 78, in invoke
(nova.api): TRACE: result = method(context, **args)
(nova.api): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/api/ec2/cloud.py", line 1152, in describe_instances
(nova.api): TRACE: instance_id=instance_id)(nova.api): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/api/ec2/cloud.py", line 1161, in _format_describe_instances
(nova.api): TRACE: return {'reservationSet': self._format_instances(context, **kwargs)}
(nova.api): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/api/ec2/cloud.py", line 1266, in _format_instances(nova.api): TRACE: if fixed['network'] and use_v6:
(nova.api): TRACE: File "/usr/local/lib/python2.7/dist-packages/nova/db/sqlalchemy/models.py", line 76, in __getitem__(nova.api): TRACE: return getattr(self, key)
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 163, in __get__
(nova.api): TRACE: instance_dict(instance))
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/attributes.py", line 383, in get
(nova.api): TRACE: value = callable_(passive=passive)
(nova.api): TRACE: File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/strategies.py", line 595, in __call__
(nova.api): TRACE: (mapperutil.state_str(state), self.key)(nova.api): TRACE: DetachedInstanceError: Parent instance <FixedIp at 0x5dd6390> is not bound to a Session; lazy load operation of attribute 'network' cannot proceed(nova.api): TRACE:

[Thierry Carrez (ttx)]

Is it Diablo-specific ?

[Vish Ishaya (vishvananda)]

This was seen by a few people in diablo. There is an odd workaround which we could put in to diablo that was discovered by HP:

diff --git a/nova/db/sqlalchemy/models.py b/nova/db/sqlalchemy/models.py
index b5f30a1..034e0b4 100644
--- a/nova/db/sqlalchemy/models.py
+++ b/nova/db/sqlalchemy/models.py
@@ -669,7 +669,7 @@ class FixedIp(BASE, NovaBase):
     id = Column(Integer, primary_key=True)
     address = Column(String(255))
     network_id = Column(Integer, ForeignKey('networks.id'), nullable=True)
- network = relationship(Network, backref=backref('fixed_ips'))
+ network = relationship(Network, backref=backref('fixed_ips'), lazy='joined')
     virtual_interface_id = Column(Integer, ForeignKey('virtual_interfaces.id'),
                                                                  nullable=True)
     virtual_interface = relationship(VirtualInterface,

Unfortunately, we could never figure out exactly why this fixes it, as it is very hard to reproduce consistently except under load. The offending relationship has been removed in essex so it no longer applies.