The db pooling code was creating up to 16 connections per worker, which maxes out connections very quickly, hence the fix to remove it.
Vish
On Sep 30, 2011, at 7:42 AM, David Kranz wrote:
> *** This bug is a duplicate of bug 838581 ***
> https://bugs.launchpad.net/bugs/838581
>
> Understood. I see that the default is 100 connections, but how does one
> decide how big that number should be? One per worker, one per worker per
> service?
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/861689
>
> Title:
> Too many connections error in nova-manage with many compute nodes
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> I was bringing up a nova cluster (using scripted automation) with one
> controller and a dozen compute nodes. After some compute nodes
> successfully register themselves and can be seen with 'nova-manage
> host list', the others get exceptions in nova-compute as shown at the
> bottom of this report. At this point any attempt to run nova-manage
> gets
>
> Exception TypeError: "'Connection' object is not iterable" in <bound
> method TpooledConnectionPool.__del__ of
> <eventlet.db_pool.TpooledConnectionPool object at 0x2794d10>> ignored
>
> But if mysql is restarted then nova-manage works again. Also, if nova-
> compute is then restarted manually on a failed compute node then it
> successfully registers its services. It seems to me that nova-compute
> should expect that there might be contention accessing the remote
> mysql and have a way to retry, at least for some time period.
>
> 2011-09-28 13:48:27,378 INFO nova.db.sqlalchemy [-] Using mysql/eventlet db_pool.
> 2011-09-28 13:48:27,402 CRITICAL nova [-] (1040, 'Too many connections')
> (nova): TRACE: Traceback (most recent call last):
> (nova): TRACE: File "/usr/bin/nova-compute", line 49, in <module>
> (nova): TRACE: service.wait()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 357, in wait
> (nova): TRACE: _launcher.wait()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 107, in wait
> (nova): TRACE: service.wait()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
> (nova): TRACE: return self._exit_event.wait()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
> (nova): TRACE: return hubs.get_hub().switch()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
> (nova): TRACE: return self.greenlet.switch()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
> (nova): TRACE: result = function(*args, **kwargs)
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 77, in run_server
> (nova): TRACE: server.start()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 137, in start
> (nova): TRACE: self.manager.init_host()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 159, in init_host
> (nova): TRACE: instances = self.db.instance_get_all_by_host(context, self.host)
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 536, in instance_get_all_by_host
> (nova): TRACE: return IMPL.instance_get_all_by_host(context, host)
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 101, in wrapper
> (nova): TRACE: return f(*args, **kwargs)
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1392, in instance_get_all_by_host
> (nova): TRACE: session = get_session()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/session.py", line 53, in get_session
> (nova): TRACE: _ENGINE = get_engine()
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/session.py", line 87, in get_engine
> (nova): TRACE: creator = eventlet.db_pool.ConnectionPool(MySQLdb, **pool_args)
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 50, in __init__
> (nova): TRACE: order_as_stack=True)
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/pools.py", line 108, in __init__
> (nova): TRACE: self.free_items.append(self.create())
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 246, in create
> (nova): TRACE: **self._kwargs)
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 253, in connect
> (nova): TRACE: conn = tpool.execute(db_module.connect, *args, **kw)
> (nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
> (nova): TRACE: rv = meth(*args,**kwargs)
> (nova): TRACE: File "/usr/lib/pymodules/python2.7/MySQLdb/__init__.py", line 81, in Connect
> (nova): TRACE: return Connection(*args, **kwargs)
> (nova): TRACE: File "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 187, in __init__
> (nova): TRACE: super(Connection, self).__init__(*args, **kwargs2)
> (nova): TRACE: OperationalError: (1040, 'Too many connections')
> (nova): TRACE:
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/861689/+subscriptions
The db pooling code was creating up to 16 connections per worker, which maxes out connections very quickly, hence the fix to remove it.
Vish
On Sep 30, 2011, at 7:42 AM, David Kranz wrote:
> *** This bug is a duplicate of bug 838581 *** /bugs.launchpad .net/bugs/ 838581 /bugs.launchpad .net/bugs/ 861689 onPool. __del__ of db_pool. TpooledConnecti onPool object at 0x2794d10>> ignored nova-compute" , line 49, in <module> python2. 7/dist- packages/ nova/service. py", line 357, in wait python2. 7/dist- packages/ nova/service. py", line 107, in wait python2. 7/dist- packages/ eventlet/ greenthread. py", line 166, in wait event.wait( ) python2. 7/dist- packages/ eventlet/ event.py" , line 116, in wait hub().switch( ) python2. 7/dist- packages/ eventlet/ hubs/hub. py", line 177, in switch switch( ) python2. 7/dist- packages/ eventlet/ greenthread. py", line 192, in main python2. 7/dist- packages/ nova/service. py", line 77, in run_server python2. 7/dist- packages/ nova/service. py", line 137, in start init_host( ) python2. 7/dist- packages/ nova/compute/ manager. py", line 159, in init_host instance_ get_all_ by_host( context, self.host) python2. 7/dist- packages/ nova/db/ api.py" , line 536, in instance_ get_all_ by_host get_all_ by_host( context, host) python2. 7/dist- packages/ nova/db/ sqlalchemy/ api.py" , line 101, in wrapper python2. 7/dist- packages/ nova/db/ sqlalchemy/ api.py" , line 1392, in instance_ get_all_ by_host python2. 7/dist- packages/ nova/db/ sqlalchemy/ session. py", line 53, in get_session python2. 7/dist- packages/ nova/db/ sqlalchemy/ session. py", line 87, in get_engine db_pool. ConnectionPool( MySQLdb, **pool_args) python2. 7/dist- packages/ eventlet/ db_pool. py", line 50, in __init__ stack=True) python2. 7/dist- packages/ eventlet/ pools.py" , line 108, in __init__ items.append( self.create( )) python2. 7/dist- packages/ eventlet/ db_pool. py", line 246, in create python2. 7/dist- packages/ eventlet/ db_pool. py", line 253, in connect db_module. connect, *args, **kw) python2. 7/dist- packages/ eventlet/ tpool.py" , line 76, in tworker **kwargs) pymodules/ python2. 7/MySQLdb/ __init_ _.py", line 81, in Connect pymodules/ python2. 7/MySQLdb/ connections. py", line 187, in __init__ _init__ (*args, **kwargs2) /bugs.launchpad .net/nova/ +bug/861689/ +subscriptions
> https:/
>
> Understood. I see that the default is 100 connections, but how does one
> decide how big that number should be? One per worker, one per worker per
> service?
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https:/
>
> Title:
> Too many connections error in nova-manage with many compute nodes
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> I was bringing up a nova cluster (using scripted automation) with one
> controller and a dozen compute nodes. After some compute nodes
> successfully register themselves and can be seen with 'nova-manage
> host list', the others get exceptions in nova-compute as shown at the
> bottom of this report. At this point any attempt to run nova-manage
> gets
>
> Exception TypeError: "'Connection' object is not iterable" in <bound
> method TpooledConnecti
> <eventlet.
>
> But if mysql is restarted then nova-manage works again. Also, if nova-
> compute is then restarted manually on a failed compute node then it
> successfully registers its services. It seems to me that nova-compute
> should expect that there might be contention accessing the remote
> mysql and have a way to retry, at least for some time period.
>
> 2011-09-28 13:48:27,378 INFO nova.db.sqlalchemy [-] Using mysql/eventlet db_pool.
> 2011-09-28 13:48:27,402 CRITICAL nova [-] (1040, 'Too many connections')
> (nova): TRACE: Traceback (most recent call last):
> (nova): TRACE: File "/usr/bin/
> (nova): TRACE: service.wait()
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: _launcher.wait()
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: service.wait()
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: return self._exit_
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: return hubs.get_
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: return self.greenlet.
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: result = function(*args, **kwargs)
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: server.start()
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: self.manager.
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: instances = self.db.
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: return IMPL.instance_
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: return f(*args, **kwargs)
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: session = get_session()
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: _ENGINE = get_engine()
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: creator = eventlet.
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: order_as_
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: self.free_
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: **self._kwargs)
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: conn = tpool.execute(
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: rv = meth(*args,
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: return Connection(*args, **kwargs)
> (nova): TRACE: File "/usr/lib/
> (nova): TRACE: super(Connection, self)._
> (nova): TRACE: OperationalError: (1040, 'Too many connections')
> (nova): TRACE:
>
> To manage notifications about this bug go to:
> https:/