quantum can not connect with rabbitmq server on the controller for over 100 compute nodes

[Neehar Athalye]

On n1kv scale setup, we have over 100 compute nodes being managed by their controller. Noticed that the quantum server initially connects to the rabbitmq server but eventually throws errors that it can not connect to it. Please see the relevant quantum server log.
========================================================================

2013-09-03 19:44:19 INFO [quantum.common.config] Logging enabled!
2013-09-03 19:44:19 DEBUG [quantum.service] ********************************************************************************
2013-09-03 19:44:19 DEBUG [quantum.service] Configuration options gathered from:
2013-09-03 19:44:19 DEBUG [quantum.service] command line args: ['--config-file', '/etc/quantum/quantum.conf', '--log-file', '/var/log/quantum/server.log', '--config-file', '/etc/quantum/plugins/cisco/cisco_plugins.ini']
2013-09-03 19:44:19 DEBUG [quantum.service] config files: ['/etc/quantum/quantum.conf', '/etc/quantum/plugins/cisco/cisco_plugins.ini']
2013-09-03 19:44:19 DEBUG [quantum.service] ================================================================================
2013-09-03 19:44:19 DEBUG [quantum.service] allow_bulk = True
2013-09-03 19:44:19 DEBUG [quantum.service] allow_overlapping_ips = False
2013-09-03 19:44:19 DEBUG [quantum.service] allow_pagination = False
2013-09-03 19:44:19 DEBUG [quantum.service] allow_sorting = False
2013-09-03 19:44:19 DEBUG [quantum.service] allowed_rpc_exception_modules = ['quantum.openstack.common.exception', 'nova.exception', 'cinder.exception', 'exceptions']
2013-09-03 19:44:19 DEBUG [quantum.service] api_extensions_path =
2013-09-03 19:44:19 DEBUG [quantum.service] api_paste_config = /etc/quantum/api-paste.ini
2013-09-03 19:44:19 DEBUG [quantum.service] auth_strategy = keystone
2013-09-03 19:44:19 DEBUG [quantum.service] backdoor_port = None
2013-09-03 19:44:19 DEBUG [quantum.service] base_mac = fa:16:3e:00:00:00
2013-09-03 19:44:19 DEBUG [quantum.service] bind_host = 0.0.0.0
2013-09-03 19:44:19 DEBUG [quantum.service] bind_port = 9696
2013-09-03 19:44:19 DEBUG [quantum.service] config_dir = None
2013-09-03 19:44:19 DEBUG [quantum.service] config_file = ['/etc/quantum/quantum.conf', '/etc/quantum/plugins/cisco/cisco_plugins.ini']
2013-09-03 19:44:19 DEBUG [quantum.service] control_exchange = quantum
2013-09-03 19:44:19 DEBUG [quantum.service] core_plugin = quantum.plugins.cisco.network_plugin.PluginV2
2013-09-03 19:44:19 DEBUG [quantum.service] debug = True
2013-09-03 19:44:19 DEBUG [quantum.service] default_log_levels = ['amqplib=WARN', 'sqlalchemy=WARN', 'boto=WARN', 'suds=INFO', 'keystone=INFO', 'eventlet.wsgi.server=WARN']
2013-09-03 19:44:19 DEBUG [quantum.service] dhcp_agent_notification = True
2013-09-03 19:44:19 DEBUG [quantum.service] dhcp_lease_duration = 120
2013-09-03 19:44:19 DEBUG [quantum.service] fake_rabbit = False
2013-09-03 19:44:19 DEBUG [quantum.service] fatal_deprecations = False
2013-09-03 19:44:19 DEBUG [quantum.service] force_gateway_on_subnet = False
2013-09-03 19:44:19 DEBUG [quantum.service] host = ctrl-node46
2013-09-03 19:44:19 DEBUG [quantum.service] instance_format = [instance: %(uuid)s]
2013-09-03 19:44:19 DEBUG [quantum.service] instance_uuid_format = [instance: %(uuid)s]
2013-09-03 19:44:19 DEBUG [quantum.service] log_config = None
2013-09-03 19:44:19 DEBUG [quantum.service] log_date_format = %Y-%m-%d %H:%M:%S
2013-09-03 19:44:19 DEBUG [quantum.service] log_dir = /var/log/quantum
2013-09-03 19:44:19 DEBUG [quantum.service] log_file = /var/log/quantum/server.log
2013-09-03 19:44:19 DEBUG [quantum.service] log_format = %(asctime)s %(levelname)8s [%(name)s] %(message)s
2013-09-03 19:44:19 DEBUG [quantum.service] logfile_mode = 0644
2013-09-03 19:44:19 DEBUG [quantum.service] logging_context_format_string = %(asctime)s.%(msecs)03d %(levelname)s %(name)s [%(request_id)s %(user)s %(tenant)s] %(instance)s%(message)s
2013-09-03 19:44:19 DEBUG [quantum.service] logging_debug_format_suffix = %(funcName)s %(pathname)s:%(lineno)d
2013-09-03 19:44:19 DEBUG [quantum.service] logging_default_format_string = %(asctime)s.%(msecs)03d %(process)d %(levelname)s %(name)s [-] %(instance)s%(message)s
2013-09-03 19:44:19 DEBUG [quantum.service] logging_exception_prefix = %(asctime)s.%(msecs)03d %(process)d TRACE %(name)s %(instance)s
2013-09-03 19:44:19 DEBUG [quantum.service] mac_generation_retries = 16
2013-09-03 19:44:19 DEBUG [quantum.service] max_dns_nameservers = 5
2013-09-03 19:44:19 DEBUG [quantum.service] max_fixed_ips_per_port = 5
2013-09-03 19:44:19 DEBUG [quantum.service] max_subnet_host_routes = 20
2013-09-03 19:44:19 DEBUG [quantum.service] pagination_max_limit = -1
2013-09-03 19:44:19 DEBUG [quantum.service] periodic_fuzzy_delay = 5
2013-09-03 19:44:19 DEBUG [quantum.service] periodic_interval = 40
2013-09-03 19:44:19 DEBUG [quantum.service] policy_file = policy.json
2013-09-03 19:44:19 DEBUG [quantum.service] publish_errors = False
2013-09-03 19:44:19 DEBUG [quantum.service] rpc_backend = quantum.openstack.common.rpc.impl_kombu
2013-09-03 19:44:19 DEBUG [quantum.service] rpc_cast_timeout = 30
2013-09-03 19:44:19 DEBUG [quantum.service] rpc_conn_pool_size = 30
2013-09-03 19:44:19 DEBUG [quantum.service] rpc_response_timeout = 60
2013-09-03 19:44:19 DEBUG [quantum.service] rpc_thread_pool_size = 64
2013-09-03 19:44:19 DEBUG [quantum.service] service_plugins = []
2013-09-03 19:44:19 DEBUG [quantum.service] state_path = /var/lib/quantum
2013-09-03 19:44:19 DEBUG [quantum.service] syslog_log_facility = LOG_USER
2013-09-03 19:44:19 DEBUG [quantum.service] use_stderr = True
2013-09-03 19:44:19 DEBUG [quantum.service] use_syslog = False
2013-09-03 19:44:19 DEBUG [quantum.service] verbose = False
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.reconnect_interval = 2
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.sql_connection = mysql://quantum:n1k12345@127.0.0.1/quantum?charset=utf8
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.sql_dbpool_enable = False
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.sql_idle_timeout = 3600
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.sql_max_pool_size = 5
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.sql_max_retries = -1
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.sql_min_pool_size = 1
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.sqlalchemy_pool_size = 5
2013-09-03 19:44:19 DEBUG [quantum.service] ********************************************************************************
2013-09-03 19:44:19 INFO [quantum.common.config] Config paste file: /etc/quantum/api-paste.ini
2013-09-03 19:44:19 DEBUG [quantum.openstack.common.lockutils] Got semaphore "qmlock" for method "_create_instance"...
2013-09-03 19:44:19 DEBUG [quantum.manager] Plugin location: quantum.plugins.cisco.network_plugin.PluginV2
2013-09-03 19:44:19 INFO [quantum.manager] Loading Plugin: quantum.plugins.cisco.network_plugin.PluginV2
2013-09-03 19:44:20 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _setup_vsm
2013-09-03 19:44:20 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:44:20 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:44:30 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:44:31 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:44:41 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:44:42 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:44:52 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:44:52 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:45:02 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:45:03 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:45:13 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:45:14 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:45:24 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:45:24 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:45:34 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:45:35 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:45:45 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:45:46 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:45:56 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:45:56 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:46:06 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 1 seconds.
2013-09-03 19:46:06 DEBUG [quantum.plugins.cisco.n1kv.n1kv_quantum_plugin] _populate_policy_profiles
2013-09-03 19:46:07 DEBUG [quantum.plugins.cisco.n1kv.n1kv_client] status_code 200
2013-09-03 19:46:07 INFO [quantum.openstack.common.rpc.common] Reconnecting to AMQP server on 127.0.0.1:5672
2013-09-03 19:48:02 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 1 seconds.
2013-09-03 19:49:48 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 3 seconds.
2013-09-03 19:51:36 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 5 seconds.
2013-09-03 19:53:25 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 7 seconds.
2013-09-03 19:55:17 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 9 seconds.
2013-09-03 19:57:11 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 11 seconds.
2013-09-03 19:59:07 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 13 seconds.
2013-09-03 20:01:04 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 15 seconds.
2013-09-03 20:03:04 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 17 seconds.
2013-09-03 20:05:05 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 19 seconds.
2013-09-03 20:07:09 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 21 seconds.
2013-09-03 20:09:14 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 23 seconds.
2013-09-03 20:11:22 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 25 seconds.
2013-09-03 20:13:31 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 27 seconds.
2013-09-03 20:15:43 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 29 seconds.
2013-09-03 20:18:45 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 1 seconds.
2013-09-03 20:19:12 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: Socket closed. Trying again in 3 seconds.
2013-09-03 20:19:15 ERROR [quantum.openstack.common.rpc.common] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 5 seconds.
2013-09-03 20:19:20 ERROR [quantum.api.extensions] Extension path 'extensions' doesn't exist!
2013-09-03 20:19:20 ERROR [quantum.api.extensions] Extension path 'quantum/plugins/cisco/extensions' doesn't exist!
2013-09-03 20:19:20 WARNING [quantum.api.extensions] Extension flavor not supported by any of loaded plugins
2013-09-03 20:19:20 WARNING [quantum.api.extensions] Extension lbaas not supported by any of loaded plugins
2013-09-03 20:19:20 WARNING [quantum.api.extensions] Extension routed-service-insertion not supported by any of loaded plugins
2013-09-03 20:19:20 WARNING [quantum.api.extensions] Extension router-service-type not supported by any of loaded plugins
2013-09-03 20:19:20 WARNING [quantum.api.extensions] Extension quotas not supported by any of loaded plugins
2013-09-03 20:19:20 WARNING [quantum.api.extensions] Extension service-type not supported by any of loaded plugins
2013-09-03 20:19:20 WARNING [quantum.api.extensions] Extension port-security not supported by any of loaded plugins
2013-09-03 20:19:20 WARNING [quantum.api.extensions] Extension extraroute not supported by any of loaded plugins
2013-09-03 20:19:20 WARNING [quantum.api.extensions] Extension security-group not supported by any of loaded plugins
2013-09-03 20:19:20 WARNING [quantum.api.extensions] Extension agent_scheduler not supported by any of loaded plugins
2013-09-03 20:19:20 ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.7/dist-packages/quantum/common/rpc.py", line 43, in dispatch
    quantum_ctxt, version, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.7/dist-packages/quantum/db/agents_db.py", line 173, in report_state
    plugin.create_or_update_agent(context, agent_state)
  File "/usr/lib/python2.7/dist-packages/quantum/db/agents_db.py", line 145, in create_or_update_agent
    context, agent['agent_type'], agent['host'])
  File "/usr/lib/python2.7/dist-packages/quantum/db/agents_db.py", line 121, in _get_agent_by_type_and_host
    Agent.host == host).one()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one
    ret = list(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances
    close_with_result=True)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2231, in _connection_from_session
    **kw)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 730, in connection
    close_with_result=close_with_result)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 734, in _connection_for_bind
    return self.transaction._connection_for_bind(engine)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 267, in _connection_for_bind
    conn = bind.contextual_connect()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2490, in contextual_connect
    self.pool.connect(),
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 224, in connect
    return _ConnectionFairy(self).checkout()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 387, in __init__
    rec = self._connection_record = pool._do_get()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 741, in _do_get
    con = self._create_connection()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 188, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 270, in __init__
    self.connection = self.__connect()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 330, in __connect
    connection = self.__pool._creator()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 80, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 281, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (OperationalError) (1040, 'Too many connections') None None
2013-09-03 20:19:20 ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.7/dist-packages/quantum/common/rpc.py", line 43, in dispatch
    quantum_ctxt, version, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.7/dist-packages/quantum/db/agents_db.py", line 173, in report_state
    plugin.create_or_update_agent(context, agent_state)
  File "/usr/lib/python2.7/dist-packages/quantum/db/agents_db.py", line 145, in create_or_update_agent
    context, agent['agent_type'], agent['host'])
  File "/usr/lib/python2.7/dist-packages/quantum/db/agents_db.py", line 121, in _get_agent_by_type_and_host
    Agent.host == host).one()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one
    ret = list(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances
    close_with_result=True)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2231, in _connection_from_session
    **kw)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 730, in connection
    close_with_result=close_with_result)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 734, in _connection_for_bind
    return self.transaction._connection_for_bind(engine)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 267, in _connection_for_bind
    conn = bind.contextual_connect()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2490, in contextual_connect
    self.pool.connect(),
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 224, in connect
    return _ConnectionFairy(self).checkout()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 387, in __init__
    rec = self._connection_record = pool._do_get()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 741, in _do_get
    con = self._create_connection()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 188, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 270, in __init__
    self.connection = self.__connect()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 330, in __connect
    connection = self.__pool._creator()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 80, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 281, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (OperationalError) (1040, 'Too many connections') None None
2013-09-03 20:19:20 ERROR [quantum.openstack.common.rpc.amqp] Exception during message handling
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/amqp.py", line 430, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/usr/lib/python2.7/dist-packages/quantum/common/rpc.py", line 43, in dispatch
    quantum_ctxt, version, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/quantum/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/usr/lib/python2.7/dist-packages/quantum/db/agents_db.py", line 173, in report_state
    plugin.create_or_update_agent(context, agent_state)
  File "/usr/lib/python2.7/dist-packages/quantum/db/agents_db.py", line 145, in create_or_update_agent
    context, agent['agent_type'], agent['host'])
  File "/usr/lib/python2.7/dist-packages/quantum/db/agents_db.py", line 121, in _get_agent_by_type_and_host
    Agent.host == host).one()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2184, in one
    ret = list(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2240, in _execute_and_instances
    close_with_result=True)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2231, in _connection_from_session
    **kw)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 730, in connection
    close_with_result=close_with_result)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 734, in _connection_for_bind
    return self.transaction._connection_for_bind(engine)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 267, in _connection_for_bind
    conn = bind.contextual_connect()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2490, in contextual_connect
    self.pool.connect(),
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 224, in connect
    return _ConnectionFairy(self).checkout()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 387, in __init__
    rec = self._connection_record = pool._do_get()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 741, in _do_get
    con = self._create_connection()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 188, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 270, in __init__
    self.connection = self.__connect()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 330, in __connect
    connection = self.__pool._creator()
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 80, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 281, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 187, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
OperationalError: (OperationalError) (1040, 'Too many connections') None None