IMO this isn't really fixable until Heat records all of its state in
the db and doesn't need to keep a stack action in RAM. That seems like
something we'll need for reliable update failure recovery anyway, so
perhaps we should just make sure this works once that is in place.

Excerpts from Steve Baker's message of 2014-04-08 22:13:20 UTC:
> heat-engine would need to stop processing new requests and remain
> running until all current IN_PROGRESS jobs were complete.
> 
> ** Changed in: heat
>        Status: New => Triaged
> 
> ** Changed in: heat
>    Importance: Undecided => Medium
> 
> ** Changed in: heat
>     Milestone: None => juno-1
> 
> -- 
> You received this bug notification because you are subscribed to heat.
> https://bugs.launchpad.net/bugs/1304244
> 
> Title:
>   heat-engine doesn't do shutdown gracefully when SIGTERM received
> 
> Status in Orchestration API (Heat):
>   Triaged
> 
> Bug description:
>   [Issue]
> 
>   The heat-engine does shutdown immediately when it received SIGTERM in
>   middle of stack creation.
> 
>   My expectation is the heat-engine would exit process after stack creation finished when SIGTERM received.
>   Because graceful shutdown functionality already merged from oslo.
>   In my understanding, this functionality prevent process exit in middle of processing.
> 
>     https://blueprints.launchpad.net/oslo/+spec/graceful-shutdown
> 
>   I'm not sure my expectation is right or not.
>   However, this behavior is different from nova-compute.
> 
>   [How to reproduce]
> 
>   commit id: 043590a62fa576ba339cdad3047a37c4ad583f6c
> 
>   1) configure heat-engine as daemon process.
> 
>   $ cat /etc/init/heat-engine.conf 
>   description "heat-engine"
>   author "openstack"
> 
>   start on (local-filesystems and net-device-up IFACE!=lo)
>   stop on runlevel [016]
> 
>   exec su -s /bin/sh -c "exec /opt/stack/heat/bin/heat-engine --config-
>   file=/etc/heat/heat.conf --log-file /home/devstack/log/heat-engine.log
>   > /dev/null 2>&1" devstack
> 
>   2) start heat-engine as daemon.
> 
>   $ sudo service heat-engine start
>   heat-engine start/running, process 31555
> 
>   3) create something stack it takes a lot of time
> 
>   $ heat stack-create -f a-lot-of-vm a-lot-of-vm
>   +--------------------------------------+-------------+--------------------+----------------------+
>   | id                                   | stack_name  | stack_status       | creation_time        |
>   +--------------------------------------+-------------+--------------------+----------------------+
>   | b1ada312-f8d5-4afc-af09-2ed9a1fbe4b3 | a-lot-of-vm | CREATE_IN_PROGRESS | 2014-04-08T07:47:02Z |
>   +--------------------------------------+-------------+--------------------+----------------------+
> 
>   4) send SIGTERM in middle of stack creation
> 
>   $ ps aux|grep heat-engine|grep -v grep
>   root     31555  0.0  0.0   4052  1548 ?        Ss   16:45   0:00 su -s /bin/sh -c exec /opt/stack/heat/bin/heat-engine --config-file=/etc/heat/heat.conf --log-file /home/devstack/log/heat-engine.log > /dev/null 2>&1 devstack
>   devstack 31557  1.5  1.9  64868 40480 ?        Ss   16:45   0:01 python /opt/stack/heat/bin/heat-engine --config-file=/etc/heat/heat.conf --log-file /home/devstack/log/heat-engine.log
>   $ kill -SIGTERM 31557
> 
>   5) check heat-engine.log, so we can see heat-engine does shutdown
>   immediately
> 
>       :
>   2014-04-08 16:47:26.611 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "a-lot-of-vm" [b1ada312-f8d5-4afc-af09-2ed9a1fbe4b3] sleeping from (pid=31557) _sleep /opt/stack/heat/heat/engine/scheduler.py:130
>   2014-04-08 16:47:27.612 DEBUG heat.engine.scheduler [-] Task stack_task from Stack "a-lot-of-vm" [b1ada312-f8d5-4afc-af09-2ed9a1fbe4b3] running from (pid=31557) step /opt/stack/heat/heat/engine/scheduler.py:186
>   2014-04-08 16:47:27.612 DEBUG heat.engine.scheduler [-] Task resource_action running from (pid=31557) step /opt/stack/heat/heat/engine/scheduler.py:186
>   2014-04-08 16:47:27.613 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 127.0.0.1
>   2014-04-08 16:47:27.718 INFO heat.openstack.common.service [-] Caught SIGTERM, exiting
>   2014-04-08 16:47:27.721 DEBUG amqp [-] Closed channel #1 from (pid=31557) _do_close /usr/lib/python2.7/dist-packages/amqp/channel.py:88
>   2014-04-08 16:47:27.722 DEBUG amqp [-] Closed channel #1 from (pid=31557) _do_close /usr/lib/python2.7/dist-packages/amqp/channel.py:88
> 
>   6) The stack status is "CREATE_IN_PROGRESS" after rebooted heat-
>   engine.
> 
>   $ sudo service heat-engine start
>   heat-engine start/running, process 32161
>   $ heat stack-list
>   +--------------------------------------+-------------+--------------------+----------------------+
>   | id                                   | stack_name  | stack_status       | creation_time        |
>   +--------------------------------------+-------------+--------------------+----------------------+
>   | b1ada312-f8d5-4afc-af09-2ed9a1fbe4b3 | a-lot-of-vm | CREATE_IN_PROGRESS | 2014-04-08T07:47:02Z |
>   +--------------------------------------+-------------+--------------------+----------------------+
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/heat/+bug/1304244/+subscriptions