OpenStack stable/kilo release, Ubuntu 14.04 devstack setup .while multiple calls are generated during tempest run(tempest.api.networks) in HP networking CI setup we see queuePool limit related exception which results in tempest failure.
tempest tests that are run include
cd /opt/stack/tempest; testr init ; ./tools/pretty_tox_serial.sh \" tempest.api.network.test_networks tempest.api.network.test_extensions tempest.api.network.test_allowed_address_pair tempest.api.network.test_extra_dhcp_options tempest.api.network.test_floating_ips_negative tempest.api.network.test_floating_ips tempest.api.network.test_networks_negative tempest.api.network.test_networks tempest.api.network.test_ports tempest.api.network.test_routers_negative tempest.api.network.test_routers tempest.api.network.test_security_groups_negative tempest.api.network.test_security_groups tempest.api.network.test_service_type_management tempest.scenario.test_network_basic_ops|^test_security_groups_basic_ops|^test_server_basic_ops|^test_network_advanced_server_ops|^test_server_advanced_ops\" "
detailed logs can be found at
http://15.126.220.115/OVSVAPP-VLAN-KILO/602/203586/9215e7309301fd72961e906a1ae5afe1d5c8f909/
might have been deleted concurrently.
2015-07-21 12:27:55.595 WARNING neutron.db.agentschedulers_db [req-2585e7c4-8f33-439b-8b64-afc1522a686a None None] Removing network 17d6770a-c478-46d4-9556-ee77bbaf2d6b from agent 04fa72de-c1f9-4b51-8369-3b027aaf42d4 because the agent did not report to the server in the last 150 seconds.
2015-07-21 12:28:15.822 ERROR oslo_messaging.rpc.dispatcher [req-99ec4c7c-3f9d-4dd7-b920-c57606a07df7 None None] Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/neutron/neutron/db/agents_db.py", line 281, in report_state
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher self.plugin.create_or_update_agent(context, agent_state)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/neutron/neutron/db/agents_db.py", line 238, in create_or_update_agent
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher return self._create_or_update_agent(context, agent)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/neutron/neutron/db/agents_db.py", line 217, in _create_or_update_agent
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher context, agent['agent_type'], agent['host'])
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/neutron/neutron/db/agents_db.py", line 193, in _get_agent_by_type_and_host
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher Agent.host == host).one()
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2395, in one
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher ret = list(self)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher return self._execute_and_instances(context)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2451, in _execute_and_instances
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher close_with_result=True)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2442, in _connection_from_session
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher **kw)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 854, in connection
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher close_with_result=close_with_result)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 858, in _connection_for_bind
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher return self.transaction._connection_for_bind(engine)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 322, in _connection_for_bind
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher conn = bind.contextual_connect()
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 208, in contextual_connect
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher self._wrap_pool_connect(self.pool.connect, None),
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/compat/handle_error.py", line 215, in _wrap_pool_connect
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher return fn()
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 338, in connect
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher return _ConnectionFairy._checkout(self)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 644, in _checkout
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher fairy = _ConnectionRecord.checkout(pool)
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 440, in checkout
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher rec = pool._do_get()
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 959, in _do_get
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher (self.size(), self.overflow(), self._timeout))
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher TimeoutError: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
2015-07-21 12:28:15.822 TRACE oslo_messaging.rpc.dispatcher
2015-07-21 12:28:15.885 ERROR oslo_messaging.rpc.dispatcher [req-6ee592e3-e5aa-4d6d-a422-3e7ca4734a25 None None] Exception during message handling: QueuePool limit of size 10 overflow 20 reached, connection timed out, timeout 10
This bug is > 172 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.
If the bug is still valid, then update the bug status.