Currently, puppet configures neutron to have
1 master
2 api_workers
2 rpc_workers
1 rpc_state_report_workers
so we have 6 processes in total. Due to its nature neutron-server utilizes a lot of memory ~400Mb for every process. So we have 6* 400 ~ 2.3Gb only for neutron.
root@node-2:/etc/neutron# ps -C neutron-server -O rss | awk '{ count ++; sum += $2 }; END {count --; print "Number of processes =",count; print "Memory usage per process =",sum/1024/count, "MB"; print "Total memory usage =", sum/1024, "MB" ;};'
Number of processes = 6
Memory usage per process = 63.8444 MB
Total memory usage = 383.066 MB
That's pretty fine when MySQL or Rabbit was up before neutron. However, when they die they will not be able to allocate memory. For instance MySQL dies on start action with the following message
InnoDB: Initializing buffer pool, size = 512.0M InnoDB: mmap(549453824 bytes) failed; errno 12 2016-04-13 09:45:04 1201 [ERROR] InnoDB: Cannot allocate memory for the buffer pool 2016-04-13 09:45:04 1201 [ERROR] Plugin 'InnoDB' init function returned error. 2016-04-13 09:45:04 1201 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed. 2016-04-13 09:45:04 1201 [ERROR] Unknown/unsupported storage engine: innodb 2016-04-13 09:45:04 1201 [ERROR] Aborting 2016-04-13 09:45:04 1201 [Note] Binlog end 2016-04-13 09:45:04 1201
Many of sporadic SWARM issues are based on this fact as alloc(), malloc() don't allocate RAM for services which have a great respect for allocating and releasing RAM.
From Fuel side the only thing we can do is to use cgroups