Brief Description
-----------------
Our VM resize/migrate tests are failing
Severity
--------
<Major: System/Feature is usable but degraded>. Running a test suite this might affect the tests executed after.
Steps to Reproduce
------------------
Try to resize VM
Expected Behavior
------------------
It should work fine
Actual Behavior
----------------
Timeout during VM resize
Reproducibility
---------------
This happens pretty often, I guess it might be 100%
System Configuration
--------------------
One node system, Two node system, Multi-node system, Dedicated storage
Branch/Pull Time/Commit
-----------------------
master
Last Pass
---------
-
Timestamp/Logs
--------------
The collected logs will be attached.
[2022-01-01 08:59:51,126] 54 DEBUG MainThread conftest.update_results:: ***Failure at test call: /localdisk/starlingx/s1_duplex/test/automated-pytest-suite/keywords/vm_helper.py:1994: uti
***Details: launch_instances = 'b07d5807-5d77-4461-a400-96c517a7dc24'
create_flavors_and_images = {'flavor1': '182c2525-7a39-4882-bdc1-6a0e577c5dc4', 'flavor2': 'a9f9be0f-0862-491f-aa40-496666297aa1', 'image': '769818a4-4812-4ceb-bef0-79ecb0580b73'}
@mark.robotsanity
def test_resize_instances(launch_instances, create_flavors_and_images):
> vm_helper.resize_vm(vm_id=launch_instances, flavor_id=create_flavors_and_images["flavor2"])
testcases/sanity/sanity_openstack/test_instance_from_image.py:155:.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _.
keywords/vm_helper.py:1829: in resize_vm
timeout=300, con_ssh=con_ssh)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _.
vm_id = 'b07d5807-5d77-4461-a400-96c517a7dc24'
status = ['VERIFY_RESIZE', 'ERROR'], timeout = 300, check_interval = 3
fail_ok = False, con_ssh = None
auth_info = {'domain': 'Default', 'password': 'St4rlingX*', 'tenant': 'admin', 'user': 'admin'}
def wait_for_vm_status(vm_id, status=VMStatus.ACTIVE,
timeout=VMTimeout.STATUS_CHANGE, check_interval=3,
fail_ok=False,
con_ssh=None, auth_info=Tenant.get('admin')):
"""
....
Args:
vm_id:
status (list|str):
timeout:
check_interval:
fail_ok (bool):
con_ssh:
auth_info:
....
Returns: The Status of the vm_id depend on what Status it is looking for
....
"""
end_time = time.time() + timeout
if isinstance(status, str):
status = [status]
....
current_status = get_vm_values(vm_id, 'status', strict=True,
con_ssh=con_ssh, auth_info=auth_info)[0]
while time.time() < end_time:
for expected_status in status:
if current_status == expected_status:
LOG.info("VM status has reached {}".format(expected_status))
return expected_status
....
time.sleep(check_interval)
current_status = get_vm_values(vm_id, 'status', strict=True,
con_ssh=con_ssh, auth_info=auth_info)[0]
....
err_msg = "Timed out waiting for vm status: {}. Actual vm status: " \
"{}".format(status, current_status)
if fail_ok:
LOG.warning(err_msg)
return None
else:
> raise exceptions.VMTimeout(err_msg)
E utils.exceptions.VMTimeout: VM operation timed out.
E Details: Timed out waiting for vm status: ['VERIFY_RESIZE', 'ERROR']. Actual vm status: ACTIVE
keywords/vm_helper.py:1994: VMTimeout
Test Activity
-------------
Sanity
Workaround
----------
-
I observed this behavior too, yesterday. My comments below:
The cold migrate that is failing logs errors like these on nova-compute:
https:/
And because it fails, it returns the instance to ACTIVE state, so that it never gets on VERIFY_RESIZE and causes the tests to fail.
Resize (not as part of cold migrate) seems to fail intermittently.