wait_child() without greenthread.sleep() uses excessive CPU
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Fix Released
|
Undecided
|
Scott DAngelo |
Bug Description
https:/
However, the wait() function will also loop in _wait_child() when the parent catched SIGTERM and the parent is waiting to reap the children. This also causes excessive CPU usage if the child does not die quickly for some reason.
I've an instance of cinder-volume that has been running for several days in this wait state. An strace of the parent shows:
wait4(0, 0x7fff081b7780, WNOHANG, NULL) = 0
wait4(0, 0x7fff081b7780, WNOHANG, NULL) = 0
wait4(0, 0x7fff081b7780, WNOHANG, NULL) = 0
......forever....
this is because the parent has caught SIGTERM is and is looping on wait_child without the eventlet.
during the normal running state of cinder-volume the parent does an epoll for the eventlet.
wait4(0, 0x7fff25185430, WNOHANG, NULL) = 0
select(0, NULL, NULL, NULL, {0, 9984}) = 0 (Timeout)
wait4(0, 0x7fff25185430, WNOHANG, NULL) = 0
select(0, NULL, NULL, NULL, {0, 9984}) = 0 (Timeout)
My CPU is pegged at 100% with this process (shown in top):
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27313 cinder 20 0 103m 25m 4080 R 100 0.0 56343:11 cinder-volume
To fix this, the second call to _wait_child() needs the same sleep as the first:
def wait(self):
"""Loop waiting on children to die and respawning as necessary."""
while self.running:
wrap = self._wait_child()
if not wrap:
# Yield to other threads if no children have exited
# Sleep for a short time to avoid excessive CPU usage
# (see bug #1095346)
while (self.running and len(wrap.children) < wrap.workers
if self.sigcaught:
signame = {signal.SIGTERM: 'SIGTERM',
for pid in self.children:
try:
except OSError as exc:
if exc.errno != errno.ESRCH:
# Wait for children to die
if self.children:
while self.children:
Patch is :
--- a/cinder/service.py
+++ b/cinder/service.py
@@ -323,7 +323,10 @@ class ProcessLauncher
if self.children:
while self.children:
- self._wait_child()
+ wrap = self._wait_child()
+ if not wrap:
+ eventlet.
+ continue
Changed in cinder: | |
assignee: | nobody → Scott DAngelo (scott-dangelo) |
Changed in cinder: | |
milestone: | none → icehouse-1 |
status: | Fix Committed → Fix Released |
Changed in cinder: | |
milestone: | icehouse-1 → 2014.1 |
Fix proposed to branch: master /review. openstack. org/57971
Review: https:/