| 2016-05-17 09:58:39 |
venkata anil |
description |
With wsgi_default_pool_size=100[1], if the keepalived state change server gets 100 requests at the same time, while processing the requests heavy load is seen on cpu, making the network node unresponsive. For each request, keepalived state change server spawns a new meta data proxy process(i.e neutron-ns-metadata-proxy). During heavy cpu load, with "top" command, I can see many metadata proxy processes in "running" state at same time(see the attachment).
When wsgi_default_pool_size=8, I see state change server spawning 8 metadata proxy processes at a time("top" command shows 8 meta data proxy processes in "running" state at a time), cpu load is less and metadata proxy processes(for example, 100) spawned for all requests without failures.
We can keep wsgi_default_pool_size=100 for neutron API server, and use seperate configuration for UnixDomainWSGIServer(for example CONF.unix_domain_wsgi_default_pool_size).
neutron/agent/linux/utils.py
class UnixDomainWSGIServer(wsgi.Server):
def _run(self, application, socket):
"""Start a WSGI service in a new green thread."""
logger = logging.getLogger('eventlet.wsgi.server')
eventlet.wsgi.server(socket,
application,
max_size=CONF.unix_domain_wsgi_default_pool_size,
protocol=UnixDomainHttpProtocol,
log=logger)
[1] https://github.com/openstack/neutron/commit/9d573387f1e33ce85269d3ed9be501717eed4807 |
With wsgi_default_pool_size=100[1], if the keepalived state change server gets 100 requests at the same time, while processing the requests heavy load is seen on cpu, making the network node unresponsive. For each request, keepalived state change server spawns a new meta data proxy process(i.e neutron-ns-metadata-proxy). During heavy cpu load, with "top" command, I can see many metadata proxy processes in "running" state at same time(see the attachment).
When wsgi_default_pool_size=8, I see state change server spawning 8 metadata proxy processes at a time("top" command shows 8 meta data proxy processes in "running" state at a time), cpu load is less and metadata proxy processes(for example, 100) spawned for all requests without failures(as backlog=KEEPALIVED_STATE_CHANGE_SERVER_BACKLOG i.e 4096).
We can keep wsgi_default_pool_size=100 for neutron API server, and use seperate configuration for UnixDomainWSGIServer(for example CONF.unix_domain_wsgi_default_pool_size).
neutron/agent/linux/utils.py
class UnixDomainWSGIServer(wsgi.Server):
def _run(self, application, socket):
"""Start a WSGI service in a new green thread."""
logger = logging.getLogger('eventlet.wsgi.server')
eventlet.wsgi.server(socket,
application,
max_size=CONF.unix_domain_wsgi_default_pool_size,
protocol=UnixDomainHttpProtocol,
log=logger)
[1] https://github.com/openstack/neutron/commit/9d573387f1e33ce85269d3ed9be501717eed4807 |
|
