nova-api should shutdown gracefully

[Tiantian Gao]

In IceHouse, An awesome feature got implemented: graceful shutdown nova service, which can make sure in-process RPC request got done before kill the process.

But nova-api not support graceful shutdown now, which can cause problem when do upgrading. For example, when a request to create an instance was in-progress, kill the nova-api may lead to quota not sync or odd database records. Especially in large-scale development, there are hundreds of request in a second, kill the nova-api will interrupt lots in-process greenlet.

In nova/wsgi.py, when stoping WSGI service, we first shrink the greenlet pool size to 0, then kill the eventlet wsgi server. The work around is quick and easy: wait for all greenlets in the pool to finish, then kill wsgi server. The code looks like below:

diff --git a/nova/wsgi.py b/nova/wsgi.py
index ba52872..3c89297 100644
--- a/nova/wsgi.py
+++ b/nova/wsgi.py
@@ -212,6 +212,9 @@ class Server(object):
         if self._server is not None:
             # Resize pool to stop new requests from being processed
             self._pool.resize(0)
+ num = self._pool.running()
+ LOG.info(_("Waiting WSGI server to finish %d requests." % num))
+ self._pool.waitall()
             self._server.kill()

     def wait(self):

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/129167

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sean Dague (<email address hidden>) on branch: master
Review: https://review.openstack.org/129167
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Davanum Srinivas (DIMS) (dims-v)]

Removing "In Progress" status and assignee as change is abandoned.

[Davanum Srinivas (DIMS) (dims-v)]

We should be adding support for this in oslo.service

[Joe Gordon (jogo)]

nova-compute already does something like this, I just don't think nova-api does yet.

[Marian Horban (mhorban)]

One more test for graceful shutdown. Can be used with any of openstack services.
Usage:
python graceful_shutdown.py openstack-service api-port

For example
python graceful_shutdown.py 'neutron-server' 9696

Must be run on the same host with service's proces.

[Mitsuhiro Tanino (mitsuhiro-tanino)]

Fix proposed to oslo.service : master
https://review.openstack.org/#/c/190175/

[Mitsuhiro Tanino (mitsuhiro-tanino)]

I made my bug report for duplicate of this bug.

https://bugs.launchpad.net/oslo.service/+bug/1463132

[Elena Ezhova (eezhova)]

Related fix for oslo.service: https://review.openstack.org/#/c/190175/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Michael Still (<email address hidden>) on branch: master
Review: https://review.openstack.org/129167
Reason: This patch as been stalled for a very long time, so I am going to abandon it to keep the review queue sane. Please restore the change when its ready for review.