heat-engine doesn't do shutdown gracefully when SIGTERM received
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Heat |
Fix Released
|
Medium
|
Mitsuru Kanabuchi | ||
oslo-incubator |
Fix Released
|
Medium
|
Mitsuru Kanabuchi |
Bug Description
[Issue]
The heat-engine does shutdown immediately when it received SIGTERM in middle of stack creation.
My expectation is the heat-engine would exit process after stack creation finished when SIGTERM received.
Because graceful shutdown functionality already merged from oslo.
In my understanding, this functionality prevent process exit in middle of processing.
https:/
I'm not sure my expectation is right or not.
However, this behavior is different from nova-compute.
[How to reproduce]
commit id: 043590a62fa576b
1) configure heat-engine as daemon process.
$ cat /etc/init/
description "heat-engine"
author "openstack"
start on (local-filesystems and net-device-up IFACE!=lo)
stop on runlevel [016]
exec su -s /bin/sh -c "exec /opt/stack/
2) start heat-engine as daemon.
$ sudo service heat-engine start
heat-engine start/running, process 31555
3) create something stack it takes a lot of time
$ heat stack-create -f a-lot-of-vm a-lot-of-vm
+------
| id | stack_name | stack_status | creation_time |
+------
| b1ada312-
+------
4) send SIGTERM in middle of stack creation
$ ps aux|grep heat-engine|grep -v grep
root 31555 0.0 0.0 4052 1548 ? Ss 16:45 0:00 su -s /bin/sh -c exec /opt/stack/
devstack 31557 1.5 1.9 64868 40480 ? Ss 16:45 0:01 python /opt/stack/
$ kill -SIGTERM 31557
5) check heat-engine.log, so we can see heat-engine does shutdown immediately
:
2014-04-08 16:47:26.611 DEBUG heat.engine.
2014-04-08 16:47:27.612 DEBUG heat.engine.
2014-04-08 16:47:27.612 DEBUG heat.engine.
2014-04-08 16:47:27.613 INFO urllib3.
2014-04-08 16:47:27.718 INFO heat.openstack.
2014-04-08 16:47:27.721 DEBUG amqp [-] Closed channel #1 from (pid=31557) _do_close /usr/lib/
2014-04-08 16:47:27.722 DEBUG amqp [-] Closed channel #1 from (pid=31557) _do_close /usr/lib/
6) The stack status is "CREATE_
$ sudo service heat-engine start
heat-engine start/running, process 32161
$ heat stack-list
+------
| id | stack_name | stack_status | creation_time |
+------
| b1ada312-
+------
Changed in oslo: | |
assignee: | nobody → Mitsuru Kanabuchi (kanabuchi) |
Changed in oslo: | |
importance: | Undecided → Medium |
Changed in heat: | |
status: | Fix Committed → Fix Released |
Changed in oslo: | |
milestone: | none → juno-1 |
status: | Fix Committed → Fix Released |
Changed in heat: | |
milestone: | juno-1 → 2014.2 |
heat-engine would need to stop processing new requests and remain running until all current IN_PROGRESS jobs were complete.