quantum can not connect with rabbitmq server on the controller for over 100 compute nodes
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cisco Openstack |
New
|
Undecided
|
Unassigned |
Bug Description
On n1kv scale setup, we have over 100 compute nodes being managed by their controller. Noticed that the quantum server initially connects to the rabbitmq server but eventually throws errors that it can not connect to it. Please see the relevant quantum server log.
=======
2013-09-03 19:44:19 INFO [quantum.
2013-09-03 19:44:19 DEBUG [quantum.service] *******
2013-09-03 19:44:19 DEBUG [quantum.service] Configuration options gathered from:
2013-09-03 19:44:19 DEBUG [quantum.service] command line args: ['--config-file', '/etc/quantum/
2013-09-03 19:44:19 DEBUG [quantum.service] config files: ['/etc/
2013-09-03 19:44:19 DEBUG [quantum.service] =======
2013-09-03 19:44:19 DEBUG [quantum.service] allow_bulk = True
2013-09-03 19:44:19 DEBUG [quantum.service] allow_overlappi
2013-09-03 19:44:19 DEBUG [quantum.service] allow_pagination = False
2013-09-03 19:44:19 DEBUG [quantum.service] allow_sorting = False
2013-09-03 19:44:19 DEBUG [quantum.service] allowed_
2013-09-03 19:44:19 DEBUG [quantum.service] api_extensions_path =
2013-09-03 19:44:19 DEBUG [quantum.service] api_paste_config = /etc/quantum/
2013-09-03 19:44:19 DEBUG [quantum.service] auth_strategy = keystone
2013-09-03 19:44:19 DEBUG [quantum.service] backdoor_port = None
2013-09-03 19:44:19 DEBUG [quantum.service] base_mac = fa:16:3e:00:00:00
2013-09-03 19:44:19 DEBUG [quantum.service] bind_host = 0.0.0.0
2013-09-03 19:44:19 DEBUG [quantum.service] bind_port = 9696
2013-09-03 19:44:19 DEBUG [quantum.service] config_dir = None
2013-09-03 19:44:19 DEBUG [quantum.service] config_file = ['/etc/
2013-09-03 19:44:19 DEBUG [quantum.service] control_exchange = quantum
2013-09-03 19:44:19 DEBUG [quantum.service] core_plugin = quantum.
2013-09-03 19:44:19 DEBUG [quantum.service] debug = True
2013-09-03 19:44:19 DEBUG [quantum.service] default_log_levels = ['amqplib=WARN', 'sqlalchemy=WARN', 'boto=WARN', 'suds=INFO', 'keystone=INFO', 'eventlet.
2013-09-03 19:44:19 DEBUG [quantum.service] dhcp_agent_
2013-09-03 19:44:19 DEBUG [quantum.service] dhcp_lease_duration = 120
2013-09-03 19:44:19 DEBUG [quantum.service] fake_rabbit = False
2013-09-03 19:44:19 DEBUG [quantum.service] fatal_deprecations = False
2013-09-03 19:44:19 DEBUG [quantum.service] force_gateway_
2013-09-03 19:44:19 DEBUG [quantum.service] host = ctrl-node46
2013-09-03 19:44:19 DEBUG [quantum.service] instance_format = [instance: %(uuid)s]
2013-09-03 19:44:19 DEBUG [quantum.service] instance_
2013-09-03 19:44:19 DEBUG [quantum.service] log_config = None
2013-09-03 19:44:19 DEBUG [quantum.service] log_date_format = %Y-%m-%d %H:%M:%S
2013-09-03 19:44:19 DEBUG [quantum.service] log_dir = /var/log/quantum
2013-09-03 19:44:19 DEBUG [quantum.service] log_file = /var/log/
2013-09-03 19:44:19 DEBUG [quantum.service] log_format = %(asctime)s %(levelname)8s [%(name)s] %(message)s
2013-09-03 19:44:19 DEBUG [quantum.service] logfile_mode = 0644
2013-09-03 19:44:19 DEBUG [quantum.service] logging_
2013-09-03 19:44:19 DEBUG [quantum.service] logging_
2013-09-03 19:44:19 DEBUG [quantum.service] logging_
2013-09-03 19:44:19 DEBUG [quantum.service] logging_
2013-09-03 19:44:19 DEBUG [quantum.service] mac_generation_
2013-09-03 19:44:19 DEBUG [quantum.service] max_dns_nameservers = 5
2013-09-03 19:44:19 DEBUG [quantum.service] max_fixed_
2013-09-03 19:44:19 DEBUG [quantum.service] max_subnet_
2013-09-03 19:44:19 DEBUG [quantum.service] pagination_
2013-09-03 19:44:19 DEBUG [quantum.service] periodic_
2013-09-03 19:44:19 DEBUG [quantum.service] periodic_interval = 40
2013-09-03 19:44:19 DEBUG [quantum.service] policy_file = policy.json
2013-09-03 19:44:19 DEBUG [quantum.service] publish_errors = False
2013-09-03 19:44:19 DEBUG [quantum.service] rpc_backend = quantum.
2013-09-03 19:44:19 DEBUG [quantum.service] rpc_cast_timeout = 30
2013-09-03 19:44:19 DEBUG [quantum.service] rpc_conn_pool_size = 30
2013-09-03 19:44:19 DEBUG [quantum.service] rpc_response_
2013-09-03 19:44:19 DEBUG [quantum.service] rpc_thread_
2013-09-03 19:44:19 DEBUG [quantum.service] service_plugins = []
2013-09-03 19:44:19 DEBUG [quantum.service] state_path = /var/lib/quantum
2013-09-03 19:44:19 DEBUG [quantum.service] syslog_log_facility = LOG_USER
2013-09-03 19:44:19 DEBUG [quantum.service] use_stderr = True
2013-09-03 19:44:19 DEBUG [quantum.service] use_syslog = False
2013-09-03 19:44:19 DEBUG [quantum.service] verbose = False
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.
2013-09-03 19:44:19 DEBUG [quantum.service] DATABASE.
2013-09-03 19:44:19 DEBUG [quantum.service] *******
2013-09-03 19:44:19 INFO [quantum.
2013-09-03 19:44:19 DEBUG [quantum.
2013-09-03 19:44:19 DEBUG [quantum.manager] Plugin location: quantum.
2013-09-03 19:44:19 INFO [quantum.manager] Loading Plugin: quantum.
2013-09-03 19:44:20 DEBUG [quantum.
2013-09-03 19:44:20 DEBUG [quantum.
2013-09-03 19:44:20 DEBUG [quantum.
2013-09-03 19:44:30 DEBUG [quantum.
2013-09-03 19:44:31 DEBUG [quantum.
2013-09-03 19:44:41 DEBUG [quantum.
2013-09-03 19:44:42 DEBUG [quantum.
2013-09-03 19:44:52 DEBUG [quantum.
2013-09-03 19:44:52 DEBUG [quantum.
2013-09-03 19:45:02 DEBUG [quantum.
2013-09-03 19:45:03 DEBUG [quantum.
2013-09-03 19:45:13 DEBUG [quantum.
2013-09-03 19:45:14 DEBUG [quantum.
2013-09-03 19:45:24 DEBUG [quantum.
2013-09-03 19:45:24 DEBUG [quantum.
2013-09-03 19:45:34 DEBUG [quantum.
2013-09-03 19:45:35 DEBUG [quantum.
2013-09-03 19:45:45 DEBUG [quantum.
2013-09-03 19:45:46 DEBUG [quantum.
2013-09-03 19:45:56 DEBUG [quantum.
2013-09-03 19:45:56 DEBUG [quantum.
2013-09-03 19:46:06 ERROR [quantum.
2013-09-03 19:46:06 DEBUG [quantum.
2013-09-03 19:46:07 DEBUG [quantum.
2013-09-03 19:46:07 INFO [quantum.
2013-09-03 19:48:02 ERROR [quantum.
2013-09-03 19:49:48 ERROR [quantum.
2013-09-03 19:51:36 ERROR [quantum.
2013-09-03 19:53:25 ERROR [quantum.
2013-09-03 19:55:17 ERROR [quantum.
2013-09-03 19:57:11 ERROR [quantum.
2013-09-03 19:59:07 ERROR [quantum.
2013-09-03 20:01:04 ERROR [quantum.
2013-09-03 20:03:04 ERROR [quantum.
2013-09-03 20:05:05 ERROR [quantum.
2013-09-03 20:07:09 ERROR [quantum.
2013-09-03 20:09:14 ERROR [quantum.
2013-09-03 20:11:22 ERROR [quantum.
2013-09-03 20:13:31 ERROR [quantum.
2013-09-03 20:15:43 ERROR [quantum.
2013-09-03 20:18:45 ERROR [quantum.
2013-09-03 20:19:12 ERROR [quantum.
2013-09-03 20:19:15 ERROR [quantum.
2013-09-03 20:19:20 ERROR [quantum.
2013-09-03 20:19:20 ERROR [quantum.
2013-09-03 20:19:20 WARNING [quantum.
2013-09-03 20:19:20 WARNING [quantum.
2013-09-03 20:19:20 WARNING [quantum.
2013-09-03 20:19:20 WARNING [quantum.
2013-09-03 20:19:20 WARNING [quantum.
2013-09-03 20:19:20 WARNING [quantum.
2013-09-03 20:19:20 WARNING [quantum.
2013-09-03 20:19:20 WARNING [quantum.
2013-09-03 20:19:20 WARNING [quantum.
2013-09-03 20:19:20 WARNING [quantum.
2013-09-03 20:19:20 ERROR [quantum.
Traceback (most recent call last):
File "/usr/lib/
rval = self.proxy.
File "/usr/lib/
quantum_ctxt, version, method, **kwargs)
File "/usr/lib/
return getattr(proxyobj, method)(ctxt, **kwargs)
File "/usr/lib/
plugin.
File "/usr/lib/
context, agent['
File "/usr/lib/
Agent.host == host).one()
File "/usr/lib/
ret = list(self)
File "/usr/lib/
return self._execute_
File "/usr/lib/
close_
File "/usr/lib/
**kw)
File "/usr/lib/
close_
File "/usr/lib/
return self.transactio
File "/usr/lib/
conn = bind.contextual
File "/usr/lib/
self.
File "/usr/lib/
return _ConnectionFair
File "/usr/lib/
rec = self._connectio
File "/usr/lib/
con = self._create_
File "/usr/lib/
return _ConnectionReco
File "/usr/lib/
self.connection = self.__connect()
File "/usr/lib/
connection = self.__
File "/usr/lib/
return dialect.
File "/usr/lib/
return self.dbapi.
File "/usr/lib/
return Connection(*args, **kwargs)
File "/usr/lib/
super(
OperationalError: (OperationalError) (1040, 'Too many connections') None None
2013-09-03 20:19:20 ERROR [quantum.
Traceback (most recent call last):
File "/usr/lib/
rval = self.proxy.
File "/usr/lib/
quantum_ctxt, version, method, **kwargs)
File "/usr/lib/
return getattr(proxyobj, method)(ctxt, **kwargs)
File "/usr/lib/
plugin.
File "/usr/lib/
context, agent['
File "/usr/lib/
Agent.host == host).one()
File "/usr/lib/
ret = list(self)
File "/usr/lib/
return self._execute_
File "/usr/lib/
close_
File "/usr/lib/
**kw)
File "/usr/lib/
close_
File "/usr/lib/
return self.transactio
File "/usr/lib/
conn = bind.contextual
File "/usr/lib/
self.
File "/usr/lib/
return _ConnectionFair
File "/usr/lib/
rec = self._connectio
File "/usr/lib/
con = self._create_
File "/usr/lib/
return _ConnectionReco
File "/usr/lib/
self.connection = self.__connect()
File "/usr/lib/
connection = self.__
File "/usr/lib/
return dialect.
File "/usr/lib/
return self.dbapi.
File "/usr/lib/
return Connection(*args, **kwargs)
File "/usr/lib/
super(
OperationalError: (OperationalError) (1040, 'Too many connections') None None
2013-09-03 20:19:20 ERROR [quantum.
Traceback (most recent call last):
File "/usr/lib/
rval = self.proxy.
File "/usr/lib/
quantum_ctxt, version, method, **kwargs)
File "/usr/lib/
return getattr(proxyobj, method)(ctxt, **kwargs)
File "/usr/lib/
plugin.
File "/usr/lib/
context, agent['
File "/usr/lib/
Agent.host == host).one()
File "/usr/lib/
ret = list(self)
File "/usr/lib/
return self._execute_
File "/usr/lib/
close_
File "/usr/lib/
**kw)
File "/usr/lib/
close_
File "/usr/lib/
return self.transactio
File "/usr/lib/
conn = bind.contextual
File "/usr/lib/
self.
File "/usr/lib/
return _ConnectionFair
File "/usr/lib/
rec = self._connectio
File "/usr/lib/
con = self._create_
File "/usr/lib/
return _ConnectionReco
File "/usr/lib/
self.connection = self.__connect()
File "/usr/lib/
connection = self.__
File "/usr/lib/
return dialect.
File "/usr/lib/
return self.dbapi.
File "/usr/lib/
return Connection(*args, **kwargs)
File "/usr/lib/
super(
OperationalError: (OperationalError) (1040, 'Too many connections') None None