Seems like all 32 possible apache "preforks" (and ServerLimit is 32) are busy with request GET /v1/capabilities/ HTTP/1.1 which is initiated to gnocchi_wsgi:
/var/log/httpd/gnocchi_wsgi_access.log:192.168.24.11 - - [21/Feb/2017:18:27:26 +0000] "GET /v1/capabilities/ HTTP/1.1" 500 531 "-" "ceilometer-agent-notification keystoneauth1/2.18.0 python-requests/2.11.1 CPython/2.7.5"
Processing them in parallel, apache can't fork another preforks (because it reached the limit) and no new connection is accepted, including those of nova, neutron, etc - of pingtest. Restarting apache usually fixes it.
I tried to run the same jobs without gnocchi enabled and they passed: https://review.openstack.org/#/c/436497/
After an investigation, I found some problems with apache threads on controller: http:// paste.openstack .org/show/ 599888/ (the same file is attached)
Seems like all 32 possible apache "preforks" (and ServerLimit is 32) are busy with request GET /v1/capabilities/ HTTP/1.1 which is initiated to gnocchi_wsgi: httpd/gnocchi_ wsgi_access. log:192. 168.24. 11 - - [21/Feb/ 2017:18: 27:26 +0000] "GET /v1/capabilities/ HTTP/1.1" 500 531 "-" "ceilometer- agent-notificat ion keystoneauth1/ 2.18.0 python- requests/ 2.11.1 CPython/2.7.5"
/var/log/
Processing them in parallel, apache can't fork another preforks (because it reached the limit) and no new connection is accepted, including those of nova, neutron, etc - of pingtest. Restarting apache usually fixes it. /review. openstack. org/#/c/ 436497/
I tried to run the same jobs without gnocchi enabled and they passed:
https:/
So problem is most likely in gnocchi or its interconnection with ceilometer. /github. com/openstack/ tripleo- heat-templates/ blob/c99c48b84e 20925b4f4b728e9 b103d6c8bcb3d11 /environments/ low-memory- usage.yaml# L15-L15
So I see a few solutions:
1) Investigate and solve it in gnocchi/ceilometer projects
2) Increase apache ServerLimit to something more than 32 in low-memory-usage template: https:/
3) To set up apache to work with mpm_event instead of mpm_prefork.