Activity log for bug #1581580

Date Who What changed Old value New value Message
2016-05-13 15:36:52 venkata anil bug added bug
2016-05-13 15:36:52 venkata anil attachment added top.gif https://bugs.launchpad.net/bugs/1581580/+attachment/4662237/+files/top.gif
2016-05-13 15:38:06 venkata anil neutron: assignee venkata anil (anil-venkata)
2016-05-13 16:05:17 Assaf Muller tags l3-ha l3-ha loadimpact
2016-05-13 16:05:39 Assaf Muller neutron: status New Confirmed
2016-05-13 16:06:03 Assaf Muller neutron: importance Undecided Medium
2016-05-17 09:58:39 venkata anil description With wsgi_default_pool_size=100[1], if the keepalived state change server gets 100 requests at the same time, while processing the requests heavy load is seen on cpu, making the network node unresponsive. For each request, keepalived state change server spawns a new meta data proxy process(i.e neutron-ns-metadata-proxy). During heavy cpu load, with "top" command, I can see many metadata proxy processes in "running" state at same time(see the attachment). When wsgi_default_pool_size=8, I see state change server spawning 8 metadata proxy processes at a time("top" command shows 8 meta data proxy processes in "running" state at a time), cpu load is less and metadata proxy processes(for example, 100) spawned for all requests without failures. We can keep wsgi_default_pool_size=100 for neutron API server, and use seperate configuration for UnixDomainWSGIServer(for example CONF.unix_domain_wsgi_default_pool_size). neutron/agent/linux/utils.py class UnixDomainWSGIServer(wsgi.Server): def _run(self, application, socket): """Start a WSGI service in a new green thread.""" logger = logging.getLogger('eventlet.wsgi.server') eventlet.wsgi.server(socket, application, max_size=CONF.unix_domain_wsgi_default_pool_size, protocol=UnixDomainHttpProtocol, log=logger) [1] https://github.com/openstack/neutron/commit/9d573387f1e33ce85269d3ed9be501717eed4807 With wsgi_default_pool_size=100[1], if the keepalived state change server gets 100 requests at the same time, while processing the requests heavy load is seen on cpu, making the network node unresponsive. For each request, keepalived state change server spawns a new meta data proxy process(i.e neutron-ns-metadata-proxy). During heavy cpu load, with "top" command, I can see many metadata proxy processes in "running" state at same time(see the attachment). When wsgi_default_pool_size=8, I see state change server spawning 8 metadata proxy processes at a time("top" command shows 8 meta data proxy processes in "running" state at a time), cpu load is less and metadata proxy processes(for example, 100) spawned for all requests without failures(as backlog=KEEPALIVED_STATE_CHANGE_SERVER_BACKLOG i.e 4096). We can keep wsgi_default_pool_size=100 for neutron API server, and use seperate configuration for UnixDomainWSGIServer(for example CONF.unix_domain_wsgi_default_pool_size). neutron/agent/linux/utils.py class UnixDomainWSGIServer(wsgi.Server):     def _run(self, application, socket):         """Start a WSGI service in a new green thread."""         logger = logging.getLogger('eventlet.wsgi.server')         eventlet.wsgi.server(socket,                              application,                              max_size=CONF.unix_domain_wsgi_default_pool_size,                              protocol=UnixDomainHttpProtocol,                              log=logger) [1] https://github.com/openstack/neutron/commit/9d573387f1e33ce85269d3ed9be501717eed4807
2016-05-17 16:33:02 OpenStack Infra neutron: status Confirmed In Progress
2016-09-28 13:32:56 Henry Gessau summary Heavy cpu load seen when keepalived state change server gets wsi_default_pool_size requests at same time Heavy cpu load seen when keepalived state change server gets wsgi_default_pool_size requests at same time
2016-09-28 14:59:59 OpenStack Infra neutron: status In Progress Fix Released
2016-10-07 13:37:36 OpenStack Infra tags l3-ha loadimpact in-stable-liberty l3-ha loadimpact
2016-10-07 13:37:50 OpenStack Infra tags in-stable-liberty l3-ha loadimpact in-stable-liberty in-stable-mitaka l3-ha loadimpact
2016-10-07 13:51:58 OpenStack Infra tags in-stable-liberty in-stable-mitaka l3-ha loadimpact in-stable-liberty in-stable-mitaka in-stable-newton l3-ha loadimpact