cold migrate not reject for server group set VM

[Peng Peng]

Brief Description
-----------------
Create a server group with policy set to affinity, Boot VMs with flavor in the server group. Tried to cold_migrate VMs, cold migrate was not rejected. VMs status stays in "ACTIVE", not in "'VERIFY_RESIZE'or 'ERROR' status as expected.

Severity
--------
Major

Steps to Reproduce
------------------
Create a server group with policy set to affinity,
Boot VMs with flavor in the server group.
cold_migrate VMs
Check whether cold migrate was rejected
and check whether VMs status is in "'VERIFY_RESIZE'or 'ERROR'

TC-name: nova/test_server_group.py::test_server_group_boot_vms[affinity-2]

Expected Behavior
------------------
cold migrate was rejected
or VMs status is in "'VERIFY_RESIZE'or 'ERROR'

Actual Behavior
----------------
cold_migrate was not rejected

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Two node system

Lab-name: WCP_76-77

Branch/Pull Time/Commit
-----------------------
20200127T000002Z

Last Pass
---------
unknown

Timestamp/Logs
--------------
[2020-02-05 14:11:59,220] 314 DEBUG MainThread ssh.send :: Send 'nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne boot --boot-volume=9b8b6ae6-63e6-4f62-9ecd-4196d1ce16cf --key-name=keypair-tenant1 --poll --flavor=d92b00d2-0076-4a54-8fb4-d4918857b99a --nic net-id=4468f8ac-afc0-40a2-8b5e-bc04a9d61bdc --nic net-id=21948d96-cbd9-4357-a23c-fc00af7fe207 --hint group=d76975f0-c49e-42e5-997e-67c9f2bec491 tenant1-srv_grp-10'
[2020-02-05 14:13:02,271] 436 DEBUG MainThread ssh.expect :: Output:
+--------------------------------------+-------------------------------------------------+
| Property | Value |
+--------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | - |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| adminPass | b5eNmaKnuAvA |
| config_drive | |
| created | 2020-02-05T14:12:08Z |
| description | - |
| flavor:disk | 2 |
| flavor:ephemeral | 0 |
| flavor:extra_specs | {"hw:mem_page_size": "large"} |
| flavor:original_name | srv_grp |
| flavor:ram | 1024 |
| flavor:swap | 0 |
| flavor:vcpus | 1 |
| hostId | |
| id | 0f33d4ce-285a-4da3-a53a-8d69235b9042 |
| image | Attempt to boot from volume - no image supplied |
| key_name | keypair-tenant1 |
| locked | False |
| metadata | {} |
| name | tenant1-srv_grp-10 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| server_groups | ["d76975f0-c49e-42e5-997e-67c9f2bec491"] |
| status | BUILD |
| tags | [] |
| tenant_id | 0b140400d5774738840206592b5af3d0 |
| trusted_image_certificates | - |
| updated | 2020-02-05T14:12:08Z |
| user_id | dc37f416558341799c3805ed4d29c7f7 |
+--------------------------------------+-------------------------------------------------+

[2020-02-05 14:13:17,645] 479 DEBUG MainThread ssh.exec_cmd:: Executing command...
[2020-02-05 14:13:17,645] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show 0f33d4ce-285a-4da3-a53a-8d69235b9042'
[2020-02-05 14:13:22,819] 436 DEBUG MainThread ssh.expect :: Output:
+-------------------------------------+------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | controller-0 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-0 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000003a |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2020-02-05T14:12:56.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | tenant1-mgmt-net=192.168.141.51; tenant1-net1=172.16.1.234 |
| config_drive | |
| created | 2020-02-05T14:12:08Z |
| flavor | srv_grp (d92b00d2-0076-4a54-8fb4-d4918857b99a) |
| hostId | 76834791a7f376de9ff4c99bb71a81d1dfdad650d14764688a0f0058 |
| id | 0f33d4ce-285a-4da3-a53a-8d69235b9042 |
| image | |
| key_name | keypair-tenant1 |
| name | tenant1-srv_grp-10 |
| progress | 0 |
| project_id | 0b140400d5774738840206592b5af3d0 |
| properties | |
| security_groups | name='default' |
| | name='default' |
| status | ACTIVE |
| updated | 2020-02-05T14:12:56Z |
| user_id | dc37f416558341799c3805ed4d29c7f7 |
| volumes_attached | id='9b8b6ae6-63e6-4f62-9ecd-4196d1ce16cf' |
+-------------------------------------+------------------------------------------------------------+
[sysadmin@controller-1 ~(keystone_admin)]$

[2020-02-05 14:18:11,010] 314 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne migrate --poll 0f33d4ce-285a-4da3-a53a-8d69235b9042'

[2020-02-05 14:23:34,529] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show 0f33d4ce-285a-4da3-a53a-8d69235b9042'
[2020-02-05 14:23:43,582] 436 DEBUG MainThread ssh.expect :: Output:
+-------------------------------------+------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | controller-0 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-0 |
| OS-EXT-SRV-ATTR:instance_name | instance-0000003a |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2020-02-05T14:12:56.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | tenant1-mgmt-net=192.168.141.51; tenant1-net1=172.16.1.234 |
| config_drive | |
| created | 2020-02-05T14:12:08Z |
| flavor | srv_grp (d92b00d2-0076-4a54-8fb4-d4918857b99a) |
| hostId | 76834791a7f376de9ff4c99bb71a81d1dfdad650d14764688a0f0058 |
| id | 0f33d4ce-285a-4da3-a53a-8d69235b9042 |
| image | |
| key_name | keypair-tenant1 |
| name | tenant1-srv_grp-10 |
| progress | 0 |
| project_id | 0b140400d5774738840206592b5af3d0 |
| properties | |
| security_groups | name='default' |
| | name='default' |
| status | ACTIVE |
| updated | 2020-02-05T14:18:33Z |
| user_id | dc37f416558341799c3805ed4d29c7f7 |
| volumes_attached | id='9b8b6ae6-63e6-4f62-9ecd-4196d1ce16cf' |
+-------------------------------------+------------------------------------------------------------+

Test Activity
-------------
Sanity

[Peng Peng (ppeng)]

Log @
https://bugs.launchpad.net/starlingx/+bug/1862060

[Ghada Khalil (gkhalil)]

Assigning to distro.openstack PL for review/triage

[Ghada Khalil (gkhalil)]

@Yong Hu, can you please recommend a release tag and priority for this? Will it be marked for resolution in stx.4.0?

[Zhang Kunpeng (zhangkunpeng)]

@Peng Peng, could you attach these log files?

[Peng Peng (ppeng)]

The issue was reproduced on
Lab: WCP_76_77
Load: 20200311T013001Z
log attached

[Zhang Kunpeng (zhangkunpeng)]

As the openstack have upgraded to train version in stx3.0, there are some changes in cold migrate process, that the VM state will be rollback after migrate failed.

2020-05-08 04:22:44.343 755021 INFO nova.compute.manager [req-5f2e3ec2-9ced-41b2-a5be-9fd4180dcc3d e30d8a9ccb254aad88ef83b1c22aa5cf cfb538569c2346d2971f9a8b49ba405c - default default] [instance: 891a95dc-1a76-403a-a912-c360c2a85548] Setting instance back to active after: Instance rollback performed due to: Unable to migrate instance (891a95dc-1a76-403a-a912-c360c2a85548) to current host (controller-0).

So I think the active state is't a bug

[yong hu (yhu6)]

How about retesting this one with the latest "Ussuri"??

[yong hu (yhu6)]

the code migration is important, so we do need to double confirm the behavior with the latest "U" we integrated. @Chant or @Zhang Kunpeng, please retest it with the latest build from RC or master.

[yong hu (yhu6)]

I move this LP out of stx.4.0 gating, but still we need to retest it with the latest to confirm this feature is working, as @Kunpeng commented.

[YU CHENGDE (chant)]

hello
Test cold-migration successfully with ussuri version of openstack.

deployment:
  starlingx baremetal aio duplex.

images:
 http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/flock/outputs/iso/bootimage.iso
 http://mirror.starlingx.cengn.ca/mirror/starlingx/release/4.0.1/centos/containers/outputs/helm-charts/stx-openstack-1.0-49-centos-stable-latest.tgz

environment:
System Information
Manufacturer: Sugon
Product Name: I620-G10
Version: Romly-EP
Serial Number: 9800069400613122
UUID: FB3D8EEC-A4CE-11E5-B81C-3A2624729700
Wake-up Type: Power Switch
SKU Number: To be filled by O.E.M.
Family: Rack
Version: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
Core Count: 8
Core Enabled: 8
Thread Count: 16
Signature: Type 0, Family 6, Model 45, Stepping 7

Handle 0x0030, DMI type 20, 35 bytes
Memory Device Mapped Address
Starting Address: 0x00000000000
Ending Address: 0x003FFFFFFFF
Range Size: 16 GB
Physical Device Handle: 0x002F
Memory Array Mapped Address Handle: 0x002E
Partition Row Position: 1
16Gx8

NAME="Ubuntu"
VERSION="18.04.4 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.4 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

VM:
2c2g 20g centos7.4

logs:
 add attachements tomorrow(Monday)

[YU CHENGDE (chant)]

controller-0:/var/log/containers$ openstack server group list
+--------------------------------------+--------------+----------+
| ID | Name | Policies |
+--------------------------------------+--------------+----------+
| 65221f8b-4e08-4fb4-8bc5-4ea435278129 | issue1862060 | affinity |
+--------------------------------------+--------------+----------+
controller-0:/var/log/containers$ openstack server group show 65221f8b-4e08-4fb4-8bc5-4ea435278129
+----------+--------------------------------------+
| Field | Value |
+----------+--------------------------------------+
| id | 65221f8b-4e08-4fb4-8bc5-4ea435278129 |
| members | fb221bd3-749b-4267-9403-4a5e20e27b30 |
| name | issue1862060 |
| policies | affinity |
+----------+--------------------------------------+
controller-0:/var/log/containers$ openstack server list
+--------------------------------------+----------------+---------+----------------------------+-------+----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+----------------+---------+----------------------------+-------+----------+
| fb221bd3-749b-4267-9403-4a5e20e27b30 | centosaffinity | SHUTOFF | public-net0=192.168.101.80 | | m1.small |
| 1889db5d-3caf-4b76-b57d-7d9075072115 | centos1 | SHUTOFF | public-net0=192.168.101.50 | | m1.small |
+--------------------------------------+----------------+---------+----------------------------+-------+----------+

log from schedular

  2020-08-19T06:57:00.560418481Z stdout F 2020-08-19 06:57:00.559 1 INFO nova.scheduler.host_manager [req-5642252a-6f47-440b-bd56-397de61468bd 6ad74fd9452645649ca8be99c8318891 e91d4b3186b9450e94c4a4bb599fc7d7 - default default] Host filter ignoring hosts: controller-1
2020-08-19T06:57:00.561782599Z stdout F 2020-08-19 06:57:00.559 1 INFO nova.scheduler.host_manager [req-5642252a-6f47-440b-bd56-397de61468bd 6ad74fd9452645649ca8be99c8318891 e91d4b3186b9450e94c4a4bb599fc7d7 - default default] Host filter ignoring hosts: controller-1
2020-08-19T06:57:00.571170128Z stdout F 2020-08-19 06:57:00.568 1 INFO nova.filters [req-5642252a-6f47-440b-bd56-397de61468bd 6ad74fd9452645649ca8be99c8318891 e91d4b3186b9450e94c4a4bb599fc7d7 - default default] Filter ServerGroupAffinityFilter returned 0 hosts
2020-08-19T06:57:00.571197104Z stdout F 2020-08-19 06:57:00.568 1 INFO nova.filters [req-5642252a-6f47-440b-bd56-397de61468bd 6ad74fd9452645649ca8be99c8318891 e91d4b3186b9450e94c4a4bb599fc7d7 - default default] Filter ServerGroupAffinityFilter returned 0 hosts
2020-08-19T06:57:00.571207506Z stdout F 2020-08-19 06:57:00.570 1 INFO nova.filters [req-5642252a-6f47-440b-bd56-397de61468bd 6ad74fd9452645649ca8be99c8318891 e91d4b3186b9450e94c4a4bb599fc7d7 - default default] Filtering removed all hosts for the request with instance ID 'fb221bd3-749b-4267-9403-4a5e20e27b30'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'Aggregate...

[YU CHENGDE (chant)]

The VM in affinity-policy doesn't migrate.
Neither rejected hint
nor VM status in "Shutoff"

Expected Behavior
------------------
cold migrate was rejected
or VMs status is in "'VERIFY_RESIZE'or 'ERROR'

[YU CHENGDE (chant)]

My apology
 here has an action-log on VM

Displaying 18 items
Request ID Action Start Time User ID Message
req-f347c758-f768-498a-9fad-d00fc6213f61 Stop Aug. 19, 2020, 7:24 a.m. 6ad74fd9452645649ca8be99c8318891 -
req-e4691ec2-0208-4888-9ff2-7b1793b5b886 Migrate Aug. 19, 2020, 6:43 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-e95df9b7-7353-4d5d-96af-6bde3e6d825d Migrate Aug. 19, 2020, 6:38 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-dadf14f7-a2bb-4b4f-a103-066a3d0b1cbe Create Aug. 19, 2020, 6:07 a.m. 6ad74fd9452645649ca8be99c8318891 -
req-d42e66f9-3605-4b73-a426-efd0ca66efdf Migrate Aug. 19, 2020, 6:45 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-c4f72a96-4a47-4ffd-9bb7-4dd5159fd3ae Start Aug. 19, 2020, 7:22 a.m. 6ad74fd9452645649ca8be99c8318891 -
req-abdb0c21-81a5-43d3-a962-679663565c42 Migrate Aug. 19, 2020, 6:11 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-5642252a-6f47-440b-bd56-397de61468bd Migrate Aug. 19, 2020, 6:56 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-5336285f-a47c-4850-b5e4-af10ced8a5ff Migrate Aug. 19, 2020, 6:47 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-480184de-8bda-444d-a12e-b605a7f9ca85 Migrate Aug. 19, 2020, 7:15 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-9157c3fa-b36d-4ca2-b65f-9968a04f0991 Migrate Aug. 19, 2020, 6:16 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-364e0104-b4f1-469b-8eec-eab2f507f4b3 Migrate Aug. 19, 2020, 6:44 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-0353b4f1-e261-4b1a-a500-ca9461680da6 Migrate Aug. 19, 2020, 6:12 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-342ea57f-d032-4ebf-99bf-10e97313bfb5 Migrate Aug. 19, 2020, 6:56 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-91e5e4cf-83a9-46b6-bc17-fe5d0ead2545 Migrate Aug. 19, 2020, 6:18 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-8c08303a-26c1-4021-93b1-c85aaec9164a Stop Aug. 19, 2020, 6:10 a.m. 6ad74fd9452645649ca8be99c8318891 -
req-1cfd56ed-7437-4ed7-8dc5-eb068f78e02d Migrate Aug. 19, 2020, 6:46 a.m. 6ad74fd9452645649ca8be99c8318891 Error
req-00cb3227-466c-4f20-8ac1-58b7e8e64e00 Migrate Aug. 19, 2020, 7:05 a.m. 6ad74fd9452645649ca8be99c8318891 Error
Displaying 18 items

[zhipeng liu (zhipengs)]

Hi Chant,

Could you refer to comment#6 from kunpeng, to double check if it works as we expected.
Then we can close this ticket.

Thanks!
Zhipeng

[YU CHENGDE (chant)]

cold migrate was rejected.
VMs in server-group with affinity-role can't migrate

controller-0:~$ export OS_CLOUD=openstack_helm
controller-0:~$ openstack server list
+--------------------------------------+------------------+---------+--------------------------+-------+---------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------------------+---------+--------------------------+-------+---------+
| 0cc870f8-f3a1-4c28-b054-4abec896d6c8 | centosaffinity-1 | SHUTOFF | internal-net0=10.1.1.237 | | m1.tiny |
| 8c9595ea-8483-4a82-88ce-fe8faafa8a7c | centosaffinity-2 | SHUTOFF | internal-net0=10.1.1.150 | | m1.tiny |
+--------------------------------------+------------------+---------+--------------------------+-------+---------+:
controller-0:~$ openstack server group list
+--------------------------------------+--------------+----------+
| ID | Name | Policies |
+--------------------------------------+--------------+----------+
| 65221f8b-4e08-4fb4-8bc5-4ea435278129 | issue1862060 | affinity |
+--------------------------------------+--------------+----------+
controller-0:~$ openstack server group show 65221f8b-4e08-4fb4-8bc5-4ea435278129
+----------+----------------------------------------------------------------------------+
| Field | Value |
+----------+----------------------------------------------------------------------------+
| id | 65221f8b-4e08-4fb4-8bc5-4ea435278129 |
| members | 0cc870f8-f3a1-4c28-b054-4abec896d6c8, 8c9595ea-8483-4a82-88ce-fe8faafa8a7c |
| name | issue1862060 |
| policies | affinity |
+----------+----------------------------------------------------------------------------+
controller-0:~$ openstack server migrate 8c9595ea-8483-4a82-88ce-fe8faafa8a7c
controller-0:~$ date
Tue Aug 25 07:19:52 UTC 2020

from log
openstack_nova-conductor-f8dc744f4-sshtj_7e6bcf93-82c1-4ea0-a4ec-c97b58d191ab/nova-conductor/1.log

2020-08-25T07:19:54.536450219Z stdout F : nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.
2020-08-25T07:19:54.536458829Z stdout F 2020-08-25 07:19:54.528 1 WARNING nova.scheduler.utils [req-12f46034-3d56-49b5-9c0a-adce013200a3 6ad74fd9452645649ca8be99c8318891 e91d4b3186b9450e94c4a4bb599fc7d7 - default default] Failed to compute_task_migrate_server: No valid host was found. There are not enough hosts available.
2020-08-25T07:19:54.536467581Z stdout F Traceback (most recent call last):
2020-08-25T07:19:54.536474409Z stdout F
2020-08-25T07:19:54.536482017Z stdout F File "/var/lib/openstack/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 241, in inner
2020-08-25T07:19:54.536489485Z stdout F return func(*args, **kwargs)
2020-08-25T07:19:54.536496357Z stdout F
2020-08-25T07:19:54.536504181Z stdout F File "/var/lib/openstack/lib/pytho...

[YU CHENGDE (chant)]

It is normal that VMs can't migrate if they are in same affinity and in same host.
I suggest issue can be closed.

[zhipeng liu (zhipengs)]

Thanks Kunpeng and Chant!
Set it to invalid according to current analysis!