For the second run: -------------------
I have set the below parameters in /etc/neutron/neutron.conf.
max_pool_size = 50 retry_interval = 10 max_overflow = 50 pool_max_size = 50 pool_max_overflow = 50 pool_timeout = 30
And also I have set the below parameters in nova.conf.
max_pool_size = 50 max_overflow = 50
For the second run, I didn't get any ERROR messages in the neutron log. Only traces are found in nova log.
Nova Logs:
2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Unexpected exception in API method 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions Traceback (most recent call last): 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 914, in delete 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions self._delete(req.environ['nova.context'], req, id) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 755, in _delete 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions instance = self._get_server(context, req, instance_uuid) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 382, in _get_server 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/common.py", line 496, in get_instance 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions expected_attrs=expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2215, in get 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 2198, in _get_instance 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions context, instance_uuid, expected_attrs=expected_attrs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions result = fn(cls, context, *args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 444, in get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions use_slave=use_slave) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 225, in wrapper 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/instance.py", line 436, in _db_instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/api.py", line 725, in instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return IMPL.instance_get_by_uuid(context, uuid, columns_to_join) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 169, in wrapper 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(*args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 270, in wrapped 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return f(context, *args, **kwargs) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1867, in instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions columns_to_join=columns_to_join) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1872, in _instance_get_by_uuid 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions filter_by(uuid=uuid).\ 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2659, in first 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions ret = list(self[0:1]) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2457, in __getitem__ 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return list(res) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2761, in __iter__ 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return self._execute_and_instances(context) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions close_with_result=True) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions **kw) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 893, in connection 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions execution_options=execution_options) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions engine, execution_options) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions conn = bind.contextual_connect() 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions self._wrap_pool_connect(self.pool.connect, None), 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return fn() 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions return _ConnectionFairy._checkout(self) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions fairy = _ConnectionRecord.checkout(pool) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions rec = pool._do_get() 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1053, in _do_get 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions (self.size(), self.overflow(), self._timeout)) 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30 2016-11-03 17:27:34.146 2399 ERROR nova.api.openstack.extensions 2016-11-03 17:27:34.148 2399 INFO nova.api.openstack.wsgi [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.TimeoutError'> 2016-11-03 17:27:34.148 2399 DEBUG nova.api.openstack.wsgi [req-db3d6d66-9508-4eb8-be65-964f05ff50f8 admin admin] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'sqlalchemy.exc.TimeoutError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1044
For the second run:
-------------------
I have set the below parameters in /etc/neutron/ neutron. conf.
max_pool_size = 50
retry_interval = 10
max_overflow = 50
pool_max_size = 50
pool_max_overflow = 50
pool_timeout = 30
And also I have set the below parameters in nova.conf.
max_pool_size = 50
max_overflow = 50
For the second run, I didn't get any ERROR messages in the neutron log. Only traces are found in nova log.
Nova Logs:
2016-11-03 17:27:34.146 2399 ERROR nova.api. openstack. extensions [req-db3d6d66- 9508-4eb8- be65-964f05ff50 f8 admin admin] Unexpected exception in API method openstack. extensions Traceback (most recent call last): openstack. extensions File "/opt/stack/ nova/nova/ api/openstack/ extensions. py", line 338, in wrapped openstack. extensions return f(*args, **kwargs) openstack. extensions File "/opt/stack/ nova/nova/ api/openstack/ compute/ servers. py", line 914, in delete openstack. extensions self._delete( req.environ[ 'nova.context' ], req, id) openstack. extensions File "/opt/stack/ nova/nova/ api/openstack/ compute/ servers. py", line 755, in _delete openstack. extensions instance = self._get_ server( context, req, instance_uuid) openstack. extensions File "/opt/stack/ nova/nova/ api/openstack/ compute/ servers. py", line 382, in _get_server openstack. extensions expected_ attrs=expected_ attrs) openstack. extensions File "/opt/stack/ nova/nova/ api/openstack/ common. py", line 496, in get_instance openstack. extensions expected_ attrs=expected_ attrs) openstack. extensions File "/opt/stack/ nova/nova/ compute/ api.py" , line 2215, in get openstack. extensions expected_attrs) openstack. extensions File "/opt/stack/ nova/nova/ compute/ api.py" , line 2198, in _get_instance openstack. extensions context, instance_uuid, expected_ attrs=expected_ attrs) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ oslo_versionedo bjects/ base.py" , line 184, in wrapper openstack. extensions result = fn(cls, context, *args, **kwargs) openstack. extensions File "/opt/stack/ nova/nova/ objects/ instance. py", line 444, in get_by_uuid openstack. extensions use_slave= use_slave) openstack. extensions File "/opt/stack/ nova/nova/ db/sqlalchemy/ api.py" , line 225, in wrapper openstack. extensions return f(*args, **kwargs) openstack. extensions File "/opt/stack/ nova/nova/ objects/ instance. py", line 436, in _db_instance_ get_by_ uuid openstack. extensions columns_ to_join= columns_ to_join) openstack. extensions File "/opt/stack/ nova/nova/ db/api. py", line 725, in instance_ get_by_ uuid openstack. extensions return IMPL.instance_ get_by_ uuid(context, uuid, columns_to_join) openstack. extensions File "/opt/stack/ nova/nova/ db/sqlalchemy/ api.py" , line 169, in wrapper openstack. extensions return f(*args, **kwargs) openstack. extensions File "/opt/stack/ nova/nova/ db/sqlalchemy/ api.py" , line 270, in wrapped openstack. extensions return f(context, *args, **kwargs) openstack. extensions File "/opt/stack/ nova/nova/ db/sqlalchemy/ api.py" , line 1867, in instance_ get_by_ uuid openstack. extensions columns_ to_join= columns_ to_join) openstack. extensions File "/opt/stack/ nova/nova/ db/sqlalchemy/ api.py" , line 1872, in _instance_ get_by_ uuid openstack. extensions filter_ by(uuid= uuid).\ openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2659, in first openstack. extensions ret = list(self[0:1]) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2457, in __getitem__ openstack. extensions return list(res) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2761, in __iter__ openstack. extensions return self._execute_ and_instances( context) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2774, in _execute_ and_instances openstack. extensions close_with_ result= True) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ orm/query. py", line 2765, in _connection_ from_session openstack. extensions **kw) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ orm/session. py", line 893, in connection openstack. extensions execution_ options= execution_ options) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ orm/session. py", line 898, in _connection_ for_bind openstack. extensions engine, execution_options) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ orm/session. py", line 334, in _connection_ for_bind openstack. extensions conn = bind.contextual _connect( ) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 2039, in contextual_connect openstack. extensions self._wrap_ pool_connect( self.pool. connect, None), openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ engine/ base.py" , line 2074, in _wrap_pool_connect openstack. extensions return fn() openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ pool.py" , line 376, in connect openstack. extensions return _ConnectionFair y._checkout( self) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ pool.py" , line 713, in _checkout openstack. extensions fairy = _ConnectionReco rd.checkout( pool) openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ pool.py" , line 480, in checkout openstack. extensions rec = pool._do_get() openstack. extensions File "/usr/local/ lib/python2. 7/dist- packages/ sqlalchemy/ pool.py" , line 1053, in _do_get openstack. extensions (self.size(), self.overflow(), self._timeout)) openstack. extensions TimeoutError: QueuePool limit of size 50 overflow 50 reached, connection timed out, timeout 30 openstack. extensions openstack. wsgi [req-db3d6d66- 9508-4eb8- be65-964f05ff50 f8 admin admin] HTTP exception thrown: Unexpected API Error. Please report this at http:// bugs.launchpad. net/nova/ and attach the Nova API log if possible. exc.TimeoutErro r'> openstack. wsgi [req-db3d6d66- 9508-4eb8- be65-964f05ff50 f8 admin admin] Returning 500 to user: Unexpected API Error. Please report this at http:// bugs.launchpad. net/nova/ and attach the Nova API log if possible. exc.TimeoutErro r'> __call__ /opt/stack/ nova/nova/ api/openstack/ wsgi.py: 1044
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.146 2399 ERROR nova.api.
2016-11-03 17:27:34.148 2399 INFO nova.api.
<class 'sqlalchemy.
2016-11-03 17:27:34.148 2399 DEBUG nova.api.
<class 'sqlalchemy.