VM resize failed by "No valid host was found"

[Peng Peng]

Brief Description
-----------------
Resizing VM to a different flavor failed by RROR (BadRequest): No valid host was found. No valid host found for resize (HTTP 400)

Severity
--------
Major

Steps to Reproduce
------------------
1. boot up VM with original flavor (--ephemeral 1 --swap 512 flavor auto 1024 4 1)
2. touch a file on VM
3. Create a new flavor (--ephemeral 2 --swap 1024 flavor-1 auto 1024 5 1)
....

TC-name: /nova/test_resize_vm.py::TestResizeSameHost::()::test_resize_vm_positive[local_image-4_1_512-5_2_1024-image]

Expected Behavior
------------------
resize success

Actual Behavior
----------------
resize failed

Reproducibility
---------------
Reproducible
100%

System Configuration
--------------------
One node system

Lab-name: SM_2

Branch/Pull Time/Commit
-----------------------
stx master as of 20190411T013000Z

Last Pass
---------
20190410T013000Z

Timestamp/Logs
--------------
[2019-04-11 18:23:07,139] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne flavor-create --ephemeral 1 --swap 512 flavor auto 1024 4 1'
[2019-04-11 18:23:08,663] 387 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+--------+------------+------+-----------+------+-------+-------------+-----------+-------------+
| ID | Name | Memory_MiB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | Description |
+--------------------------------------+--------+------------+------+-----------+------+-------+-------------+-----------+-------------+
| 7f736025-2d1e-4686-a88e-32c7e8c9590f | flavor | 1024 | 4 | 1 | 512 | 1 | 1.0 | True | - |
+--------------------------------------+--------+------------+------+-----------+------+-------+-------------+-----------+-------------+

[2019-04-11 18:24:23,850] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne flavor-create --ephemeral 2 --swap 1024 flavor-1 auto 1024 5 1'

[2019-04-11 18:24:32,144] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne show 71b1bcc9-4b86-4030-888a-eaf91a71847e'
[2019-04-11 18:24:33,822] 387 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | cgcsauto |
| OS-EXT-SRV-ATTR:host | controller-0 |
| OS-EXT-SRV-ATTR:hostname | tenant2-vm-8 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-0 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000014 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-6cyvrzj3 |
| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2019-04-11T18:23:36.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2019-04-11T18:23:25Z |
| description | - |
| flavor:disk | 4 |
| flavor:ephemeral | 1 |
| flavor:extra_specs | {"hw:mem_page_size": "2048"} |
| flavor:original_name | flavor |
| flavor:ram | 1024 |
| flavor:swap | 512 |
| flavor:vcpus | 1 |
| hostId | 40faf9a30b1a46eeeb3217a9e15aba177e49641530bdfd2cc562e7fc |
| host_status | UP |
| id | 71b1bcc9-4b86-4030-888a-eaf91a71847e |
| image | tis-centos-guest (85620fd5-c9fc-45f0-be14-4052014b0157) |
| key_name | keypair-tenant2 |
| locked | False |
| metadata | {} |
| name | tenant2-vm-8 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | [] |
| status | ACTIVE |
| tags | [] |
| tenant2-mgmt-net network | 192.168.251.85 |
| tenant2-net2 network | 172.18.2.173 |
| tenant_id | 0ba9533d53d34dac98d0926c13c4bcce |
| trusted_image_certificates | - |
| updated | 2019-04-11T18:23:36Z |
| user_id | f026d657c2124fb2bf8bb3f8d330266e |
+--------------------------------------+----------------------------------------------------------+
controller-0:~$
[2019-04-11 18:24:33,823] 262 DEBUG MainThread ssh.send :: Send 'echo $?'
[2019-04-11 18:24:33,928] 387 DEBUG MainThread ssh.expect :: Output:
0
controller-0:~$
[2019-04-11 18:24:33,928] 1313 INFO MainThread vm_helper.resize_vm:: Resizing VM 71b1bcc9-4b86-4030-888a-eaf91a71847e to flavor 571f5017-926c-4dfe-98a8-ee54c7583af5...
[2019-04-11 18:24:33,929] 1335 DEBUG MainThread ssh.get_active_controller:: Getting active controller client for sm_2
[2019-04-11 18:24:33,929] 423 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2019-04-11 18:24:33,929] 262 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne resize --poll 71b1bcc9-4b86-4030-888a-eaf91a71847e 571f5017-926c-4dfe-98a8-ee54c7583af5'
[2019-04-11 18:24:38,303] 387 DEBUG MainThread ssh.expect :: Output:
ERROR (BadRequest): No valid host was found. No valid host found for resize (HTTP 400) (Request-ID: req-6f3c3a0f-deef-4bdc-ace0-34b1808232d7)
controller-0:~$

Test Activity
-------------
Regression Testing

[Ghada Khalil (gkhalil)]

Marking as release gating until further investigation. This maybe a nova issue and would require follow-up with upstream.

[Juan Pablo Gomez (jpgomez)]

Bug can be reproduced: http://paste.openstack.org/show/749679/

ISO: 20190415T233001Z
Instance Image: Cirros

VM can't be resized

[zhipeng liu (zhipengs)]

Hi Peng,

Could you help to resize from second flavor to first one? (Big one to small one)
If it still fail, please enable debug log for nova and catch a log for further analysis.

Thanks!
Zhipeng

[Gerry Kopec (gerry-kopec)]

Looks like in the latest openstack-helm rebase (https://review.opendev.org/#/c/645806/) we picked up an upstream commit that removed the setting of nova config option allow_resize_to_same_host to true (see https://review.opendev.org/#/c/615707/). Based on that, I'd expect this testcase to fail. We'll now need to set it to true in our nova overrides.

Note that this would normally only account for cases on AIO-Simplex (which aligns with the above description), but I've also seen this tagged against AIO-Duplex and standard config. I'm not sure how the testcase is written. If it reduces down to one hypervisor, then that would also explain it. If there are other hypervisors available, it would be a different problem in those cases.

[zhipeng liu (zhipengs)]

"This parameter should only be set to true for resize testing in
single-host environments."

Can we manually set it to true and retest this case in AIO again?

[Gerry Kopec (gerry-kopec)]

You should be able to change the config option and retest.
system helm-override-update nova openstack --set conf.nova.DEFAULT.allow_resize_to_same_host=true
system application-apply stx-openstack

Once the nova pods are restarted you should be able to see the conf option set inside one of the nova pods and can retry the test:
kubectl exec -it -n openstack <nova-api-osapi pod name> cat /etc/nova/nova.conf

I think overall we'd want to turn this to true in all environments as that was the behaviour before in the non-containerized setup.

[zhipeng liu (zhipengs)]

Hi Kopec,

I have verified it as the method you mentioned, it can work now in AIO.
Since upstream already disable it by default, I propose to keep it now and enable it when do AIO test
according to nova comment.

Zhipeng

[Maria Guadalupe Perez Ibara (maria-gp)]

the following error is still present in simplex and duplex

OS="centos"
SW_VERSION="19.01"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20190521T132734Z"

JOB="STX_build_master_master"
<email address hidden>"
BUILD_NUMBER="111"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-05-21 13:27:34 +0000"

"Resize Instances :: Resize Cirros instances. | FAIL |
No valid host was found. No valid host found for resize (HTTP 400) (Request-ID: req-37f63382-1647-44da-9b2b-caac34273cd2) "

[zhipeng liu (zhipengs)]

Hi Maria,

Have you enabled this flag before test as Copec mentioned. If yes, please help provide logs, thanks!

Zhipeng

[Cristopher Lemus (cjlemusc)]

We tried the mentioned workaround and AIO was able to resize without any issues.

What is going to happen? Is resize going to be enabled by default? Or is going to remain disabled?

If it's going to be disabled, we can remove this test from the daily sanity test.

[zhipeng liu (zhipengs)]

I propose to remove it from AIO test case and just test it in multinode setup since
Nova project keep this disabled by default.
When we have to test it in AIO, we can enable it manually.

I sent an email about this proposal in mailing list as well.

Zhipeng

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/661875
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=4f74ea1d391696341c50c97eb3baf90755582098
Submitter: Zuul
Branch: master

commit 4f74ea1d391696341c50c97eb3baf90755582098
Author: zhipengl <email address hidden>
Date: Wed May 29 18:42:40 2019 +0800

    Add "allow_resize_to_same_host=true" to nova chart

    Openstack-helm upstream removed this flag by default. Since we
    have had requests from user, we need enable it again.

    Closes-Bug: #1824412

    Change-Id: Ib8a07d41405f838fd79a3b16f3dac9923514a06d
    Signed-off-by: zhipengl <email address hidden>

[Peng Peng (ppeng)]

Issue not reproduced on
Lab: SM_2
Load: 20190604T144018Z