The db pooling code was creating up to 16 connections per worker, which maxes out connections very quickly, hence the fix to remove it.

Vish

On Sep 30, 2011, at 7:42 AM, David Kranz wrote:

> *** This bug is a duplicate of bug 838581 ***
>    https://bugs.launchpad.net/bugs/838581
> 
> Understood. I see that the default is 100 connections, but how does one
> decide how big that number should be? One per worker, one per worker per
> service?
> 
> -- 
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/861689
> 
> Title:
>  Too many connections error in nova-manage with many compute nodes
> 
> Status in OpenStack Compute (Nova):
>  New
> 
> Bug description:
>  I was bringing up a nova cluster (using scripted automation)  with one
>  controller and a dozen compute nodes.  After some compute nodes
>  successfully register themselves and can be seen with 'nova-manage
>  host list', the others get exceptions in nova-compute as shown at the
>  bottom of this report. At this point any attempt to run nova-manage
>  gets
> 
>  Exception TypeError: "'Connection' object is not iterable" in <bound
>  method TpooledConnectionPool.__del__ of
>  <eventlet.db_pool.TpooledConnectionPool object at 0x2794d10>> ignored
> 
>  But if mysql is restarted then nova-manage works again. Also, if nova-
>  compute is then restarted manually on a failed compute node then it
>  successfully registers its services. It seems to me that nova-compute
>  should expect that there might be contention accessing the remote
>  mysql and have a way to retry, at least for some time period.
> 
>  2011-09-28 13:48:27,378 INFO nova.db.sqlalchemy [-] Using mysql/eventlet db_pool.
>  2011-09-28 13:48:27,402 CRITICAL nova [-] (1040, 'Too many connections')
>  (nova): TRACE: Traceback (most recent call last):
>  (nova): TRACE:   File "/usr/bin/nova-compute", line 49, in <module>
>  (nova): TRACE:     service.wait()
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 357, in wait
>  (nova): TRACE:     _launcher.wait()
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 107, in wait
>  (nova): TRACE:     service.wait()
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
>  (nova): TRACE:     return self._exit_event.wait()
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
>  (nova): TRACE:     return hubs.get_hub().switch()
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
>  (nova): TRACE:     return self.greenlet.switch()
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
>  (nova): TRACE:     result = function(*args, **kwargs)
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 77, in run_server
>  (nova): TRACE:     server.start()
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 137, in start
>  (nova): TRACE:     self.manager.init_host()
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 159, in init_host
>  (nova): TRACE:     instances = self.db.instance_get_all_by_host(context, self.host)
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 536, in instance_get_all_by_host
>  (nova): TRACE:     return IMPL.instance_get_all_by_host(context, host)
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 101, in wrapper
>  (nova): TRACE:     return f(*args, **kwargs)
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 1392, in instance_get_all_by_host
>  (nova): TRACE:     session = get_session()
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/session.py", line 53, in get_session
>  (nova): TRACE:     _ENGINE = get_engine()
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/session.py", line 87, in get_engine
>  (nova): TRACE:     creator = eventlet.db_pool.ConnectionPool(MySQLdb, **pool_args)
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 50, in __init__
>  (nova): TRACE:     order_as_stack=True)
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/eventlet/pools.py", line 108, in __init__
>  (nova): TRACE:     self.free_items.append(self.create())
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 246, in create
>  (nova): TRACE:     **self._kwargs)
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/eventlet/db_pool.py", line 253, in connect
>  (nova): TRACE:     conn = tpool.execute(db_module.connect, *args, **kw)
>  (nova): TRACE:   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
>  (nova): TRACE:     rv = meth(*args,**kwargs)
>  (nova): TRACE:   File "/usr/lib/pymodules/python2.7/MySQLdb/__init__.py", line 81, in Connect
>  (nova): TRACE:     return Connection(*args, **kwargs)
>  (nova): TRACE:   File "/usr/lib/pymodules/python2.7/MySQLdb/connections.py", line 187, in __init__
>  (nova): TRACE:     super(Connection, self).__init__(*args, **kwargs2)
>  (nova): TRACE: OperationalError: (1040, 'Too many connections')
>  (nova): TRACE:
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/861689/+subscriptions