glance re-spawns a child when terminating
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Glance |
Fix Released
|
Medium
|
Bernhard M. Wiedemann |
Bug Description
When sending a SIGTERM to the main glance-api process,
api.log shows
2017-08-31 13:10:30.996 10618 INFO glance.common.wsgi [-] Removed dead child 10628
2017-08-31 13:10:31.004 10618 INFO glance.common.wsgi [-] Started child 10642
2017-08-31 13:10:31.006 10642 INFO eventlet.
2017-08-31 13:10:31.008 10642 INFO eventlet.
2017-08-31 13:10:31.009 10642 INFO glance.common.wsgi [-] Child 10642 exiting normally
This is because kill_children sends a SIGTERM to all children
and wait_on_children restarts one, when it notices a dead child
We noticed this, because this triggered a fencing in our cloud's pacemaker setup because systemd seems to have a race condition in the cgroup code that should detect that all related services have terminated.
# systemctl status openstack-
● openstack-
Loaded: loaded (/usr/lib/
Active: deactivating (final-sigterm) since Thu 2017-08-31 10:13:48 UTC; 1min 14s ago
Main PID: 25077 (code=exited, status=0/SUCCESS)
Tasks: 0 (limit: 512)
CGroup: /system.
Aug 31 10:13:48 d08-9e-01-b4-9e-42 systemd[1]: Stopping OpenStack Image Service API server...
Aug 31 10:15:21 d08-9e-01-b4-9e-42 systemd[1]: openstack-
Aug 31 10:15:21 d08-9e-01-b4-9e-42 systemd[1]: Stopped OpenStack Image Service API server.
Aug 31 10:15:21 d08-9e-01-b4-9e-42 systemd[1]: openstack-
Aug 31 10:15:21 d08-9e-01-b4-9e-42 systemd[1]: openstack-
description: | updated |
Changed in glance: | |
assignee: | nobody → Bernhard M. Wiedemann (ubuntubmw) |
status: | New → In Progress |
importance: | Undecided → Medium |
Changed in glance: | |
milestone: | none → queens-1 |
Reviewed: https:/ /review. openstack. org/499592 /git.openstack. org/cgit/ openstack/ glance/ commit/ ?id=877cd166b56 ec4b7f5530ea9bf 1587077692275b
Committed: https:/
Submitter: Jenkins
Branch: master
commit 877cd166b56ec4b 7f5530ea9bf1587 077692275b
Author: Bernhard M. Wiedemann <email address hidden>
Date: Thu Aug 31 15:11:41 2017 +0200
Avoid restarting a child when terminating
When sending a SIGTERM to the main glance-api process,
it was sending a SIGTERM to its children
but then also re-spawning its first dead child.
Closes-bug: #1714240
Change-Id: Ibef426c198d287 bbdac4e764fd654 edba4b7c2d6