Graceful shutdown time is far more than configuration time

[Sun Mengyun]

Description
===========
When we set shutdown_timeout in /etc/nova/nova.conf and then shutdown the instance, the shutdown time is much longer than the value we configure.

This is due to the fact: shutdown_timeout is used as the number of loops, not the loop time. In a loop, the code calls libvirt constantly to query the virtual machine status. So, loop time is much longer than configuration.

The code looks like this：

for sec in range(shutdown_timeout):

    query instance status

    sleep(1)

Steps to reproduce
==================
1. deploy a instance in your openstack
2. configure shutdown_timeout of the comopute node where the instance was deployed at
3. restart compute service
4. shutdown the instance

Expected result
===============
Shutdown takes about the same amount of time as configuration.

Actual result
=============
Shutdown takes much more time than configuration time. For example, shutdowm_timeout=180s, but it actually cost 220-240s.

Environment
===========
[root@nail1 ~]# rpm -qa | grep nova
openstack-nova-api-18.0.2-1.el7.noarch
openstack-nova-common-18.0.2-1.el7.noarch
python2-novaclient-11.0.0-1.el7.noarch
openstack-nova-placement-api-18.0.2-1.el7.noarch
openstack-nova-scheduler-18.0.2-1.el7.noarch
openstack-nova-conductor-18.0.2-1.el7.noarch
openstack-nova-novncproxy-18.0.2-1.el7.noarch
python-nova-18.0.2-1.el7.noarch
openstack-nova-compute-18.0.2-1.el7.noarch
openstack-nova-console-18.0.2-1.el7.noarch

[Matt Riedemann (mriedem)]

Yup code is here:

https://github.com/openstack/nova/blob/926e584136e7dce59f32065292aa4eb8120f628c/nova/virt/libvirt/driver.py#L2914-L2947

This code could probably be cleaned up by using FixedIntervalWithTimeoutLoopingCall from the oslo.service library:

https://github.com/openstack/oslo.service/blob/1.38.0/oslo_service/loopingcall.py#L197

[Matt Riedemann (mriedem)]

Although it might get tricky because shutdown_retry_interval is not the amount of time to sleep between checks, it's the amount of time in between trying to shutdown the guest.