current_workload in nova hypervisor-show not recover after nova suspend/resume

[Dao Tao]

with nova suspend/resume action, nova hypervisor-show <hypervisor-id> does not recover the current_workload value.

the observation is based on Grizzly, nova version is 2.10.0.

Below is the console log: (issue also reproduced on kvm hypervisor)
Steps:
[root@ci-525400E87B4A ~]# nova list
+--------------------------------------+---------+--------+--------------------------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+---------+--------+--------------------------------------------+
| cc2bbd51-6b83-40e7-b83a-54f8b1e1f6ce | esx_vm1 | BUILD | Private network=10.10.0.27, 16.190.248.157 |
+--------------------------------------+---------+--------+--------------------------------------------+

[root@ci-525400E87B4A ~]# nova hypervisor-show 9
+----------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| cpu_info | {"model": "Intel(R) Xeon(R) CPU E5540 @ 2.53GHz", "vendor": "HP", "topology": {"cores": 8, "threads": 16, "sockets": 2}} |
| current_workload | 1 |
| disk_available_least | None |
| free_disk_gb | 164 |
| free_ram_mb | 38556 |
| hypervisor_hostname | domain-c45(Cluster-3PAR-FC-ISCSI) |
| hypervisor_type | VMware ESXi |
| hypervisor_version | 500 |
| id | 9 |
| local_gb | 224 |
| local_gb_used | 60 |
| memory_mb | 41116 |
| memory_mb_used | 2560 |
| running_vms | 2 |
| service_host | mslsh104.mgmt.local |
| service_id | 13 |
| vcpus | 200 |
| vcpus_used | 2 |
+----------------------+-------------------------------------------------------------------------------------------------------------------------------------+

[root@ci-525400E87B4A ~]# nova list
+--------------------------------------+---------+--------+--------------------------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+---------+--------+--------------------------------------------+
| cc2bbd51-6b83-40e7-b83a-54f8b1e1f6ce | esx_vm1 | ACTIVE | Private network=10.10.0.27, 16.190.248.157 |
+--------------------------------------+---------+--------+--------------------------------------------+

[root@ci-525400E87B4A ~]# nova suspend esx_vm1

[root@ci-525400E87B4A ~]# nova hypervisor-show 9
+----------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| cpu_info | {"model": "Intel(R) Xeon(R) CPU E5540 @ 2.53GHz", "vendor": "HP", "topology": {"cores": 8, "threads": 16, "sockets": 2}} |
| current_workload | 0 |
| disk_available_least | None |
| free_disk_gb | 164 |
| free_ram_mb | 38555 |
| hypervisor_hostname | domain-c45(Cluster-3PAR-FC-ISCSI) |
| hypervisor_type | VMware ESXi |
| hypervisor_version | 500 |
| id | 9 |
| local_gb | 224 |
| local_gb_used | 60 |
| memory_mb | 41115 |
| memory_mb_used | 2560 |
| running_vms | 2 |
| service_host | mslsh104.mgmt.local |
| service_id | 13 |
| vcpus | 200 |
| vcpus_used | 2 |
+----------------------+-------------------------------------------------------------------------------------------------------------------------------------+

[root@ci-525400E87B4A ~]# nova resume esx_vm1

[root@ci-525400E87B4A ~]# nova list
+--------------------------------------+---------+--------+--------------------------------------------+
| ID | Name | Status | Networks |
+--------------------------------------+---------+--------+--------------------------------------------+
| cc2bbd51-6b83-40e7-b83a-54f8b1e1f6ce | esx_vm1 | ACTIVE | Private network=10.10.0.27, 16.190.248.157 |
+--------------------------------------+---------+--------+--------------------------------------------+

[root@ci-525400E87B4A ~]# nova hypervisor-show 9
+----------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+----------------------+-------------------------------------------------------------------------------------------------------------------------------------+
| cpu_info | {"model": "Intel(R) Xeon(R) CPU E5540 @ 2.53GHz", "vendor": "HP", "topology": {"cores": 8, "threads": 16, "sockets": 2}} |
| current_workload | 0 |
| disk_available_least | None |
| free_disk_gb | 164 |
| free_ram_mb | 38555 |
| hypervisor_hostname | domain-c45(Cluster-3PAR-FC-ISCSI) |
| hypervisor_type | VMware ESXi |
| hypervisor_version | 500 |
| id | 9 |
| local_gb | 224 |
| local_gb_used | 60 |
| memory_mb | 41115 |
| memory_mb_used | 2560 |
| running_vms | 2 |
| service_host | mslsh104.mgmt.local |
| service_id | 13 |
| vcpus | 200 |
| vcpus_used | 2 |
+----------------------+-------------------------------------------------------------------------------------------------------------------------------------+

[Shawn Hartsock (hartsock)]

One potential work-around is to place the VM in the "automatic boot" list under "Virtual Machine - Start up and Shut down" in vCenter. This should be part of the VM spawn procedure.

[Shawn Hartsock (hartsock)]

whoops, comment on wrong bug. Can't fix it now. Disregard.

[Gary Kotton (garyk)]

Hi,
I am not sure what the problem is an I have been unable to reproduce this. I have done the following (on 2 different setups - the first with KVM and the second with ESX):
1. Started 3 instances
2. Suspended one of the instances
3. Resumed the suspended instance
When the instances were booting the 'current_workload' was 3. This was the same on both setups. When the instance was in suspend the 'current_workload' is 0.
From the code the it seems like the task_state is not set when the VM is in mode suspend. (https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2907).
To be honest I am not sure that this is a bug. It would be interesting to know what the intended functionality is.
Thanks
Gary

[Dao Tao (dao-tao)]

If you resume the suspended instance, the current workload should return to 3. But actually from my observation, it still keep 0 after resume.

Thanks,
-David

-----Original Message-----
From: <email address hidden> [mailto:<email address hidden>] On Behalf Of Gary Kotton
Sent: 2013年7月9日 3:26 PM
To: Tao, Dao (David, ISS-MSL-SH)
Subject: [Bug 1194076] Re: current_workload in nova hypervisor-show not recover after nova suspend/resume

Hi,
I am not sure what the problem is an I have been unable to reproduce this. I have done the following (on 2 different setups - the first with KVM and the second with ESX):
1. Started 3 instances
2. Suspended one of the instances
3. Resumed the suspended instance
When the instances were booting the 'current_workload' was 3. This was the same on both setups. When the instance was in suspend the 'current_workload' is 0.
>From the code the it seems like the task_state is not set when the VM is in mode suspend. (https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2907).
To be honest I am not sure that this is a bug. It would be interesting to know what the intended functionality is.
Thanks
Gary

--
You received this bug notification because you are subscribed to the bug report.
https://bugs.launchpad.net/bugs/1194076

Title:
  current_workload in nova hypervisor-show not recover after nova
  suspend/resume

Status in OpenStack Compute (Nova):
  Incomplete

Bug description:
  with nova suspend/resume action, nova hypervisor-show <hypervisor-id>
  does not recover the current_workload value.

  the observation is based on Grizzly, nova version is 2.10.0.

  Below is the console log: (issue also reproduced on kvm hypervisor)
  Steps:
  [root@ci-525400E87B4A ~]# nova list
  +--------------------------------------+---------+--------+--------------------------------------------+
  | ID | Name | Status | Networks |
  +--------------------------------------+---------+--------+--------------------------------------------+
  | cc2bbd51-6b83-40e7-b83a-54f8b1e1f6ce | esx_vm1 | BUILD | Private network=10.10.0.27, 16.190.248.157 |
  +--------------------------------------+---------+--------+--------------------------------------------+

  [root@ci-525400E87B4A ~]# nova hypervisor-show 9
  +----------------------+-------------------------------------------------------------------------------------------------------------------------------------+
  | Property | Value |
  +----------------------+-------------------------------------------------------------------------------------------------------------------------------------+
  | cpu_info | {"model": "Intel(R) Xeon(R) CPU E5540 @ 2.53GHz", "vendor": "HP", "topology": {"cores": 8, "threads": 16, "sockets": 2}} |
  | current_workload | 1 |
  | d...

[Gary Kotton (garyk)]

Hi,
Please see the trace from below:

ankur@ankur-virtual-machine:~$ nova suspend 11bd61b2-87aa-4656-9ed6-b11b5734ae95
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion `g_type_from_name (name) == 0' failed
  import gobject._gobject
ankur@ankur-virtual-machine:~$ nova list
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion `g_type_from_name (name) == 0' failed
  import gobject._gobject
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------+
| 11bd61b2-87aa-4656-9ed6-b11b5734ae95 | X | ACTIVE | suspending | Running | private=10.0.0.5 |
| 9355de52-1cbd-40a0-bea8-af21eba16fbe | X | ACTIVE | None | Running | private=10.0.0.4 |
+--------------------------------------+------+--------+------------+-------------+------------------+
ankur@ankur-virtual-machine:~$ nova list
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion `g_type_from_name (name) == 0' failed
  import gobject._gobject
+--------------------------------------+------+-----------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+-----------+------------+-------------+------------------+
| 11bd61b2-87aa-4656-9ed6-b11b5734ae95 | X | SUSPENDED | None | Suspended | private=10.0.0.5 |
| 9355de52-1cbd-40a0-bea8-af21eba16fbe | X | ACTIVE | None | Running | private=10.0.0.4 |
+--------------------------------------+------+-----------+------------+-------------+------------------+
ankur@ankur-virtual-machine:~$ nova resume 11bd61b2-87aa-4656-9ed6-b11b5734ae95
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion `g_type_from_name (name) == 0' failed
  import gobject._gobject
ankur@ankur-virtual-machine:~$ nova list
/usr/lib/python2.7/dist-packages/gobject/constants.py:24: Warning: g_boxed_type_register_static: assertion `g_type_from_name (name) == 0' failed
  import gobject._gobject
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------+
| 11bd61b2-87aa-4656-9ed6-b11b5734ae95 | X | ACTIVE | None | Running | private=10.0.0.5 |
| 9355de52-1cbd-40a0-bea8-af21eba16fbe | X | ACTIVE | None | Running | private=10.0.0.4 |
+--------------------------------------+------+--------+------------+-------------+------------------+
ankur@ankur-virtual-machine:~$

[Gary Kotton (garyk)]

Please note that the above code was done with Hanava.
Thanks
Gary

[Maithem (maithem)]

This isn't a bug, if you look at this https://github.com/openstack/nova/blob/master/nova/compute/stats.py#L52
the workload is being computed by the number of tasks in progress. In your case, I believe that once you suspend and show the hypervisor stats the VM is still suspending, but when you resume, by the time you show hypervisor stats the task is no longer a task in progress, thats why you see 0.