Nova VM can't reach verify_resize state after resize

[Yvonne Ding]

Brief Description
-----------------
VM can't reach to VERIFY_RESIZE state after resize.

Severity
--------
Major

Steps to Reproduce
------------------
1. Create origin flavor
openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne flavor create --ephemeral=1 --swap=512 --disk=4 --ram=1024 --vcpus=1 flavor
2. Boot a vm with given flavor
nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --flavor=af508fb0-da99-489a-a568-3ad877cfbced --image=d49cf512-91fe-4b05-9fea-55f4a7352b60 --key-name=keypair-tenant1 --availability-zone=cgcsauto:compute-1 --poll --nic net-id=124197db-317e-4913-acd5-9a74ec5709b2 --nic net-id=0d0dd7dc-ea0a-4b44-a5ca-0749541604cc tenant1-vm-2
3. Auto mount non-root disk(s)
sudo mount | grep --color=never /dev/vdb
sudo mount | grep --color=never /dev/vdc
4. Create files under vm disks: ['/', '/mnt']
sudo mkdir -p /; touch /test_file.txt
sudo echo "I'm a test file" >> /test_file.txt
sudo mount | grep vd
5. Check initial disk usage
6. Create destination flavor
openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne flavor create --ephemeral=2 --swap=1024 --disk=5 --ram=1024 --vcpus=1 flavor-2

openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne flavor set --property hw:mem_page_size=large 94ceb824-d2eb-4b9a-851f-04cbc99a67cd
7. Resize vm to dest flavor and revert
openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server resize --wait --flavor 94ceb824-d2eb-4b9a-851f-04cbc99a67cd 3ee2e240-d033-4382-a774-44cccd0ebdc6

TC-name:
TestResizeSameHost::test_resize_vm_positive[local_image-4_0_0-5_1_512-image]
TestResizeSameHost::test_resize_vm_positive[local_image-4_1_512-5_2_1024-image]

Expected Behavior
------------------
VM reach to VERIFY_RESIZE state

Actual Behavior
----------------
VM keep at Active state

Reproducibility
---------------
reproducible

System Configuration
--------------------
Regular standard 2+2

Branch/Pull Time/Commit
-----------------------
OS="centos"
SW_VERSION="21.05"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20210327T013338Z"

<email address hidden>"
BUILD_NUMBER="438"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2021-03-27 01:33:38 +0000"

FLOCK_OS="centos"
FLOCK_JOB="STX_build_layer_flock_master_master"
<email address hidden>"
FLOCK_BUILD_NUMBER="438"
FLOCK_BUILD_HOST="starlingx_mirror"
FLOCK_BUILD_DATE="2021-03-27 01:33:38 +0000"

DISTRO_OS="centos"
DISTRO_JOB="STX_build_layer_distro_master_master"
<email address hidden>"
DISTRO_BUILD_NUMBER="443"
DISTRO_BUILD_HOST="starlingx_mirror"
DISTRO_BUILD_DATE="2021-03-24 13:58:38 +0000"

COMPILER_OS="centos"
COMPILER_JOB="STX_build_layer_compiler_master_master"
<email address hidden>"
COMPILER_BUILD_NUMBER="487"
COMPILER_BUILD_HOST="starlingx_mirror"
COMPILER_BUILD_DATE="2021-03-12 18:13:44 +0000"

Timestamp/Logs
--------------
[2021-03-30 14:46:47,343] 65 DEBUG MainThread conftest.update_results:: ***Failure at test call: /home/yding/cgcsAuto/cgcs/CGCSAuto/keywords/vm_helper.py:1821: utils.exceptions.VMTimeout: VM operation timed out.
***Details: self = <CGCSAuto.testcases.functional.nova.test_resize_vm.TestResizeSameHost object at 0x7f1b2075ea58>, add_hosts_to_zone = {'local_image': 'compute-1'}
check_availability_zone = None, storage_backing = 'local_image', origin_flavor = (4, 1, 512), dest_flavor = (5, 2, 1024), boot_source = 'image'
.....
> raise exceptions.VMTimeout(err_msg)
E utils.exceptions.VMTimeout: VM operation timed out.
E Details: Timed out waiting for vm status: ['VERIFY_RESIZE', 'ERROR']. Actual vm status: ACTIVE

Test Activity
-------------
STX Nova regression

[zhipeng liu (zhipengs)]

Is it a regression? which version we pass last time.

Thanks!
Zhipeng

[Yvonne Ding (yding)]

Yes, it's nova regression.

Sorry, I have no idea when it was passed last time because I did
StarlingX regression first time.

Best,

Yvonne

On 2021-04-27 11:32 p.m., zhipeng liu wrote:
> [Please note: This e-mail is from an EXTERNAL e-mail address]
>
> Is it a regression? which version we pass last time.
>
> Thanks!
> Zhipeng
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1921934
>
> Title:
> Nova VM can't reach verify_resize state after resize
>
> Status in StarlingX:
> New
>
> Bug description:
> Brief Description
> -----------------
> VM can't reach to VERIFY_RESIZE state after resize.
>
> Severity
> --------
> Major
>
> Steps to Reproduce
> ------------------
> 1. Create origin flavor
> openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne flavor create --ephemeral=1 --swap=512 --disk=4 --ram=1024 --vcpus=1 flavor
> 2. Boot a vm with given flavor
> nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --flavor=af508fb0-da99-489a-a568-3ad877cfbced --image=d49cf512-91fe-4b05-9fea-55f4a7352b60 --key-name=keypair-tenant1 --availability-zone=cgcsauto:compute-1 --poll --nic net-id=124197db-317e-4913-acd5-9a74ec5709b2 --nic net-id=0d0dd7dc-ea0a-4b44-a5ca-0749541604cc tenant1-vm-2
> 3. Auto mount non-root disk(s)
> sudo mount | grep --color=never /dev/vdb
> sudo mount | grep --color=never /dev/vdc
> 4. Create files under vm disks: ['/', '/mnt']
> sudo mkdir -p /; touch /test_file.txt
> sudo echo "I'm a test file" >> /test_file.txt
> sudo mount | grep vd
> 5. Check initial disk usage
> 6. Create destination flavor
> openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne flavor create --ephemeral=2 --swap=1024 --disk=5 --ram=1024 --vcpus=1 flavor-2
>
> openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne flavor set --property hw:mem_page_size=large 94ceb824-d2eb-4b9a-851f-04cbc99a67cd
> 7. Resize vm to dest flavor and revert
> openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name Regio...

[zhipeng liu (zhipengs)]

Hi,

From the log, it seems nova schedule not filter a suitable host (AvailabilityZoneFilter not pass) to meet the resize requirement, then resize could not be done!
Please double check your test set up again. No other issue found.

===============================================================================================
2021-03-30T14:41:39.527463748Z stdout F 2021-03-30 14:41:39.526 1 WARNING nova.scheduler.utils [req-3a563ed4-326d-4749-b5c3-723a67bb5629 10e13e496c894457acefa5a56809ab24 acc51db1f44d42dcb18313535a45f97f - default default] Failed to compute_task_migrate_server: No valid host was found. There are not enough hosts available.

2021-03-30T14:41:39.051276934Z stdout F 2021-03-30 14:41:39.051 1 INFO nova.filters [req-3a563ed4-326d-4749-b5c3-723a67bb5629 10e13e496c894457acefa5a56809ab24 acc51db1f44d42dcb18313535a45f97f - default default] Filtering removed all hosts for the request with instance ID '3ee2e240-d033-4382-a774-44cccd0ebdc6'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 0)']
2021-03-30T14:41:39.05143533Z stdout F 2021-03-30 14:41:39.051 1 INFO nova.filters [req-3a563ed4-326d-4749-b5c3-723a67bb5629 10e13e496c894457acefa5a56809ab24 acc51db1f44d42dcb18313535a45f97f - default default] Filtering removed all hosts for the request with instance ID '3ee2e240-d033-4382-a774-44cccd0ebdc6'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 0)']
  fm.py.254 Raised alarm, uuid=55f29b20-61b7-4b86-80d1-510cde35b5ca, fm_uuid=f025e3a0-01bd-4d6d-bb43-d5deb672b6e1.

Thanks!
Zhipeng

[Austin Sun (sunausti)]

change to invalid as no feedback for longtime.