sqlalchemy replies with ResourceClosedError in case of concurrent allocations of IP addresses or volume creations
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
New
|
Undecided
|
Unassigned |
Bug Description
It seems like usage of sqlalchemy's
with_
causes ResourceClosedError exception if there are several concurrent requests to update the DB.
We've seen it with volume creations, when volume_
2011-09-23 22:07:24,355 ERROR nova.api [3a5ae61d-
[u'Traceback (most recent call last):\n', u' File "/mnt/share/
(nova.api): TRACE: Traceback (most recent call last):
(nova.api): TRACE: File "/mnt/share/
(nova.api): TRACE: result = api_request.
(nova.api): TRACE: File "/mnt/share/
(nova.api): TRACE: result = method(context, **args)
(nova.api): TRACE: File "/mnt/share/
(nova.api): TRACE: public_ip = self.network_
(nova.api): TRACE: File "/mnt/share/
(nova.api): TRACE: 'args': {'project_id': context.
(nova.api): TRACE: File "/mnt/share/
(nova.api): TRACE: return get_impl(
(nova.api): TRACE: File "/mnt/share/
(nova.api): TRACE: rv = list(rv)
(nova.api): TRACE: File "/mnt/share/
(nova.api): TRACE: raise result
(nova.api): TRACE: RemoteError: ResourceClosedError This result object does not return rows. It has been closed automatically.
or
2011-09-20 04:25:20,515 WARNING nova.volume.manager [-] Volume creation failed with traceback: <traceback object at 0x3c55878>
2011-09-20 04:25:20,643 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/mnt/share/
(nova.rpc): TRACE: rval = node_func(
(nova.rpc): TRACE: File "/mnt/share/
(nova.rpc): TRACE: raise exc_info
(nova.rpc): TRACE: ResourceClosedError
(nova.rpc): TRACE:
There are several other places in db.sqlalchemy.api where with_lockmode(
The workaround for it might be to use @utils.synchronized on top of db.sqlalchemy.api methods.
Just in case, the sqlalchemy version we are using is 0.6.4. I'm not sure if newer 0.7.2 may have the same issue or not.
I don't remember seeing this before, and I'm wondering if there is a try catch that is getting missed in the db code because of the interaction with eventlet. Some things to try:
diff -r f30a2fa65f30 eventlet/ hubs/hub. py hubs/hub. py Wed Jun 08 23:47:26 2011 -0700 hubs/hub. py Sat Sep 24 04:51:22 2011 -0700
cur. parent = self.greenlet exc_info( ) sys_exc_ info() switch( )
--- a/eventlet/
+++ b/eventlet/
@@ -173,7 +173,7 @@
except ValueError:
pass # gets raised if there is a greenlet parent cycle
- clear_sys_
+ #clear_
return self.greenlet.
def squelch_ exception( self, fileno, exc_info):
And:
https:/ /bitbucket. org/which_ linden/ eventlet/ pull-request/ 1/fix-broken- expiration- timer-for
Can you see if any of those make a difference. It would also be great to have a very simple repro case for this error to aid in testing.