postgres/sqlalchemy refresh randomly fails

[Steven Hardy]

We're seeing apparently random sqlalchemy refresh errors, only on the postgres gate, in the Heat tests today (happened on two patches I've looked at):

https://review.openstack.org/#/c/33649/

And here, where it was cleared by a reverify:

https://review.openstack.org/#/c/33778/

The logs indicate an sqlalchemy error from the heat-engine, but I've never seen this before, so I guess it's possible we have a bug in either the tempest test configuration, sqlalchemy, or heat's usage of sqlalchemy - atm I don't really know which, but raising this bug in the hope that some other projects are seeing the same issue :)

Here's an example of the sqlalchemy error:

2013-06-20 14:10:20.083 17970 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/new/heat/heat/engine/api.py", line 74, in format_stack
2013-06-20 14:10:20.083 17970 TRACE heat.openstack.common.rpc.amqp api.STACK_UPDATED_TIME: timeutils.isotime(stack.updated_time),
2013-06-20 14:10:20.083 17970 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/new/heat/heat/engine/timestamp.py", line 41, in __get__
2013-06-20 14:10:20.083 17970 TRACE heat.openstack.common.rpc.amqp o.refresh(attrs=[self.attribute])
2013-06-20 14:10:20.083 17970 TRACE heat.openstack.common.rpc.amqp File "/opt/stack/new/heat/heat/db/sqlalchemy/models.py", line 83, in refresh
2013-06-20 14:10:20.083 17970 TRACE heat.openstack.common.rpc.amqp session.refresh(self, attrs)
2013-06-20 14:10:20.083 17970 TRACE heat.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1167, in refresh
2013-06-20 14:10:20.083 17970 TRACE heat.openstack.common.rpc.amqp mapperutil.instance_str(instance))
2013-06-20 14:10:20.083 17970 TRACE heat.openstack.common.rpc.amqp InvalidRequestError: Could not refresh instance '<Stack at 0x34c2490>'