StarlingX

VM fail to live migrate after evacuation

Bug #1836402 reported by Anujeyan Manokeran on 2019-07-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	Fix Released	High	YU CHENGDE

Bug Description

Brief Description
-----------------

It was observed after VM was evacuated live migration was failing on same VM. Eg VM UUID ab3048c4-b10f-49ad-a93d-16dc24c97b93 evacuated successfully from compute-0 to compute-1 then fails Live migration any of the other computes from compute-1. Below steps are showed on cli.

ysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --
os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93

+-------------------------------------+-------------------------------------------------------------+

| Field | Value |

+-------------------------------------+-------------------------------------------------------------+

| OS-DCF:diskConfig | MANUAL |

| OS-EXT-AZ:availability_zone | nova |

| OS-EXT-SRV-ATTR:host | compute-0 |

| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0 |

| OS-EXT-SRV-ATTR:instance_name | instance-00000016 |

| OS-EXT-STS:power_state | Running |

| OS-EXT-STS:task_state | None |

| OS-EXT-STS:vm_state | active |

| OS-SRV-USG:launched_at | 2019-07-12T15:23:56.000000 |

| OS-SRV-USG:terminated_at | None |

| accessIPv4 | |

| accessIPv6 | |

| addresses | tenant1-mgmt-net=192.168.146.55; tenant1-net1=172.16.1.159 |

| config_drive | |

| created | 2019-07-12T15:23:45Z |

| flavor | flavor-default-size2 (0c7d0962-bf0c-44fd-a5b9-ff6b9d0a93b3) |

| hostId | da8f07be2656cda0f4faff2389b7ee95d00879957c5b4bbea77fe622 |

| id | ab3048c4-b10f-49ad-a93d-16dc24c97b93 |

| image | |

| key_name | keypair-tenant1 |

| name | tenant1-vm-1 |

| progress | 0 |

| project_id | 196a1151724d4d5d8fe6e24bfd4acf5f |

| properties | |

| security_groups | name='default' |

| | name='default' |

| status | ACTIVE |

| updated | 2019-07-12T15:23:56Z |

| user_id | ffd19119f10c4ab1aa99a9608a66923a |

| volumes_attached | id='cedd17da-2480-4e04-999e-2fc3de8274c3' |

+-------------------------------------+-------------------------------------------------------------+

[sysadmin@controller-1 ~(keystone_admin)]$ echo $?

compute-0:~$ sudo reboot -f
sudo reboot -f

Rebooting.

packet_write_wait: Connection to 192.168.204.138 port 22: Broken pipe

[sysadmin@controller-1 ~(keystone_admin)]$ system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne host-list
system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type int
ernalURL --os-region-name RegionOne host-list

+----+--------------+-------------+----------------+-------------+--------------+

+----+--------------+-------------+----------------+-------------+--------------+

+----+--------------+-------------+----------------+-------------+--------------+

+----+--------------+-------------+----------------+-------------+--------------+

+----+--------------+-------------+----------------+-------------+--------------+
---------------------+-------------------------------------------+

| Property | Value |

+---------------------+-------------------------------------------+

| action | none |

| administrative | unlocked |

| availability | available |

| bm_ip | 128.224.64.219 |

| bm_type | bmc |

| bm_username | root |

| boot_device | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |

| capabilities | {} |

| config_applied | 2dfb83d6-9616-4509-b8d4-e3c204d443ab |

| config_status | None |

| config_target | 2dfb83d6-9616-4509-b8d4-e3c204d443ab |

| console | ttyS0,115200 |

| created_at | 2019-07-11T19:47:49.078714+00:00 |

| hostname | compute-0 |

| id | 5 |

| install_output | text |

| install_state | completed |

| install_state_info | None |

| invprovision | provisioned |

| location | {} |

| mgmt_ip | 192.168.204.138 |

| mgmt_mac | 90:e2:ba:c6:90:60 |

| operational | enabled |

| personality | worker |

| reserved | False |

| rootfs_device | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |

| serialid | None |

| software_load | 19.01 |

| subfunctions | worker,lowlatency |

| task | |

| tboot | false |

| ttys_dcd | None |

| updated_at | 2019-07-12T15:30:39.811171+00:00 |

| uptime | 203 |

| uuid | fae1ee1f-a5ed-4ae5-b68b-ab3e9711ef24 |

| vim_progress_status | services-enabled |

+---------------------+-------------------------------------------+

---------------------+-------------------------------------------+

| Property | Value |

+---------------------+-------------------------------------------+

| action | none |

| administrative | unlocked |

| availability | available |

| bm_ip | 128.224.64.219 |

| bm_type | bmc |

| bm_username | root |

| boot_device | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |

| capabilities | {} |

| config_applied | 2dfb83d6-9616-4509-b8d4-e3c204d443ab |

| config_status | None |

| config_target | 2dfb83d6-9616-4509-b8d4-e3c204d443ab |

| console | ttyS0,115200 |

| created_at | 2019-07-11T19:47:49.078714+00:00 |

| hostname | compute-0 |

| id | 5 |

| install_output | text |

| install_state | completed |

| install_state_info | None |

| invprovision | provisioned |

| location | {} |

| mgmt_ip | 192.168.204.138 |

| mgmt_mac | 90:e2:ba:c6:90:60 |

| operational | enabled |

| personality | worker |

| reserved | False |

| rootfs_device | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |

| serialid | None |

| software_load | 19.01 |

| subfunctions | worker,lowlatency |

| task | |

| tboot | false |

| ttys_dcd | None |

| updated_at | 2019-07-12T15:30:39.811171+00:00 |

| uptime | 203 |

| uuid | fae1ee1f-a5ed-4ae5-b68b-ab3e9711ef24 |

| vim_progress_status | services-enabled |

+---------------------+-------------------------------------------+

[sysadmin@controller-1 ~(keystone_admin)]$ nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne live-migration ab3048c4-b10f-49ad-a93d-16dc24c97b93 compute-0
nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-e
ndpoint-type internalURL --os-region-name RegionOne live-migration ab3048c4-b10f-49ad-a93d-16dc24c97b93 compute-0

sysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --
os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93

+-------------------------------------+-------------------------------------------------------------+

| Field | Value |

+-------------------------------------+-------------------------------------------------------------+

| OS-DCF:diskConfig | MANUAL |

| OS-EXT-AZ:availability_zone | nova |

| OS-EXT-SRV-ATTR:host | compute-1 |

| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1 |

| OS-EXT-SRV-ATTR:instance_name | instance-00000016 |

| OS-EXT-STS:power_state | Running |

| OS-EXT-STS:task_state | None |

| OS-EXT-STS:vm_state | active |

| OS-SRV-USG:launched_at | 2019-07-12T15:24:52.000000 |

| OS-SRV-USG:terminated_at | None |

| accessIPv4 | |

| accessIPv6 | |

| addresses | tenant1-mgmt-net=192.168.146.55; tenant1-net1=172.16.1.159 |

| config_drive | |

| created | 2019-07-12T15:23:45Z |

| flavor | flavor-default-size2 (0c7d0962-bf0c-44fd-a5b9-ff6b9d0a93b3) |

| hostId | a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c |

| id | ab3048c4-b10f-49ad-a93d-16dc24c97b93 |

| image | |

| key_name | keypair-tenant1 |

| name | tenant1-vm-1 |

| progress | 0 |

| project_id | 196a1151724d4d5d8fe6e24bfd4acf5f |

| properties | |

| security_groups | name='default' |

| | name='default' |

| status | ACTIVE |

| updated | 2019-07-12T15:32:01Z |

| user_id | ffd19119f10c4ab1aa99a9608a66923a |

| volumes_attached | id='cedd17da-2480-4e04-999e-2fc3de8274c3' |

+-------------------------------------+-------------------------------------------------------------

+--------------------------------------+--------------+--------+------------------------------------------------------------+-------+----------------------+

+--------------------------------------+--------------+--------+------------------------------------------------------------+-------+----------------------+

+--------------------------------------+--------------+--------+------------------------------------------------------------+-------+----------------------+

]$ nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne migration-list
nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-e
ndpoint-type internalURL --os-region-name RegionOne migration-list

+----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+

| 31 | 3fc2d997-3c9c-44f6-8991-39ad3c87ed12 | - | - | compute-1 | - | - | error | ab3048c4-b10f-49ad-a93d-16dc24c97b93 | 46 | 46 | 2019-07-12T15:31:59.000000 | 2019-07-12T15:32:01.000000 | live-migration |

[sysadmin@controller-1 ~(keystone_admin)]$ echo $?

Severity
--------
Major
Steps to Reproduce
------------------
1. Launch VM . VM was created in compute-0
n)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --
os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93

+-------------------------------------+-------------------------------------------------------------+

| Field | Value |

+-------------------------------------+-------------------------------------------------------------+

| OS-DCF:diskConfig | MANUAL |

| OS-EXT-AZ:availability_zone | nova |

| OS-EXT-SRV-ATTR:host | compute-0 |

| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0 |

| OS-EXT-SRV-ATTR:instance_name | instance-00000016 |

| OS-EXT-STS:power_state | Running |

| OS-EXT-STS:task_state | None |

| OS-EXT-STS:vm_state | active |

| OS-SRV-USG:launched_at | 2019-07-12T15:23:56.000000 |

| OS-SRV-USG:terminated_at | None |

| accessIPv4 | |

| accessIPv6 | |

| addresses | tenant1-mgmt-net=192.168.146.55; tenant1-net1=172.16.1.159 |

| config_drive | |

| created | 2019-07-12T15:23:45Z |

| flavor | flavor-default-size2 (0c7d0962-bf0c-44fd-a5b9-ff6b9d0a93b3) |

| hostId | da8f07be2656cda0f4faff2389b7ee95d00879957c5b4bbea77fe622 |

| id | ab3048c4-b10f-49ad-a93d-16dc24c97b93 |

| image | |

| key_name | keypair-tenant1 |

| name | tenant1-vm-1 |

| progress | 0 |

| project_id | 196a1151724d4d5d8fe6e24bfd4acf5f |

| properties | |

| security_groups | name='default' |

| | name='default' |

| status | ACTIVE |

| updated | 2019-07-12T15:23:56Z |

| user_id | ffd19119f10c4ab1aa99a9608a66923a |

| volumes_attached | id='cedd17da-2480-4e04-999e-2fc3de8274c3' |

+-------------------------------------+-------------------------------------------------------------+
2. Reboot compute-0 sudo reboot –f
3. wait for compute to come up.
+----+--------------+-------------+----------------+-------------+--------------+

+----+--------------+-------------+----------------+-------------+--------------+

+----+--------------+-------------+----------------+-------------+--------------+

+----+--------------+-------------+----------------+-------------+--------------+

+----+--------------+-------------+----------------+-------------+--------------+

4.Verify VM evacuation VM running on compute-1

5. Livemigrate VM after evacute

[sysadmin@controller-1 ~(keystone_admin)]$ echo $?
echo $?

[sysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --
os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --
os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93

+-------------------------------------+-------------------------------------------------------------+

| Field | Value |

+-------------------------------------+-------------------------------------------------------------+

| OS-DCF:diskConfig | MANUAL |

| OS-EXT-AZ:availability_zone | nova |

| OS-EXT-SRV-ATTR:host | compute-1 |

| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1 |

| OS-EXT-SRV-ATTR:instance_name | instance-00000016 |

| OS-EXT-STS:power_state | Running |

| OS-EXT-STS:task_state | migrating |

| OS-EXT-STS:vm_state | active |

| OS-SRV-USG:launched_at | 2019-07-12T15:24:52.000000 |

| OS-SRV-USG:terminated_at | None |

| accessIPv4 | |

| accessIPv6 | |

| addresses | tenant1-mgmt-net=192.168.146.55; tenant1-net1=172.16.1.159 |

| config_drive | |

| created | 2019-07-12T15:23:45Z |

| flavor | flavor-default-size2 (0c7d0962-bf0c-44fd-a5b9-ff6b9d0a93b3) |

| hostId | a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c |

| id | ab3048c4-b10f-49ad-a93d-16dc24c97b93 |

| image | |

| key_name | keypair-tenant1 |

| name | tenant1-vm-1 |

| progress | 0 |

| project_id | 196a1151724d4d5d8fe6e24bfd4acf5f |

| properties | |

| security_groups | name='default' |

| | name='default' |

| status | MIGRATING |

| updated | 2019-07-12T15:31:59Z |

| user_id | ffd19119f10c4ab1aa99a9608a66923a |

| volumes_attached | id='cedd17da-2480-4e04-999e-2fc3de8274c3' |

+-------------------------------------+-------------------------------------------------------------+

| Field | Value |

+-------------------------------------+-------------------------------------------------------------+

| OS-DCF:diskConfig | MANUAL |

| OS-EXT-AZ:availability_zone | nova |

| OS-EXT-SRV-ATTR:host | compute-1 |

| OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1 |

| OS-EXT-SRV-ATTR:instance_name | instance-00000016 |

| OS-EXT-STS:power_state | Running |

| OS-EXT-STS:task_state | None |

| OS-EXT-STS:vm_state | active |

| OS-SRV-USG:launched_at | 2019-07-12T15:24:52.000000 |

| OS-SRV-USG:terminated_at | None |

| accessIPv4 | |

| accessIPv6 | |

| addresses | tenant1-mgmt-net=192.168.146.55; tenant1-net1=172.16.1.159 |

| config_drive | |

| created | 2019-07-12T15:23:45Z |

| flavor | flavor-default-size2 (0c7d0962-bf0c-44fd-a5b9-ff6b9d0a93b3) |

| hostId | a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c |

| id | ab3048c4-b10f-49ad-a93d-16dc24c97b93 |

| image | |

| key_name | keypair-tenant1 |

| name | tenant1-vm-1 |

| progress | 0 |

| project_id | 196a1151724d4d5d8fe6e24bfd4acf5f |

| properties | |

| security_groups | name='default' |

| | name='default' |

| status | ACTIVE |

| updated | 2019-07-12T15:32:01Z |

| user_id | ffd19119f10c4ab1aa99a9608a66923a |

| volumes_attached | id='cedd17da-2480-4e04-999e-2fc3de8274c3' |

+-------------------------------------+-------------------------------------------------------------

6.Migration list shows error to migrate from compute-1

nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne migration-list
nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-e
ndpoint-type internalURL --os-region-name RegionOne migration-list

System Configuration
--------------------
Regular system

Expected Behavior
------------------
Able to migrate after evacuate.
Actual Behavior
----------------
Failure on migration after evacuate.
Reproducibility
---------------
100% reproducible
System Configuration
--------------------
Regular system
Load
----
2019-07-11 01:30:00 +0000
Last Pass
---------
Not sure in this lab.

Timestamp/Logs
--------------
2019-07-12T15:31:59.000000

Test Activity
-------------
Regression test

Tags:

Revision history for this message

Anujeyan Manokeran (anujeyan) wrote on 2019-07-12:

collect logs Edit (24.5 MiB, application/x-tar)

tags:	added: stx.2.0 stx.nfv stx.regression stx.retestneded
tags:	removed: stx.nfv

Revision history for this message

Ghada Khalil (gkhalil) wrote on 2019-07-12:

Assigning to Yong Hu as this appears to be a nova issue. Given that the issue is 100% reproducible, my recommendation is that this should be gating stx.2.0

tags:	added: stx.retestneeded removed: stx.2.0 stx.retestneded
Changed in starlingx:
assignee:	nobody → yong hu (yhu6)
tags:	added: stx.distro.openstack

Numan Waheed (nwaheed) on 2019-07-15

summary:

- After VM was evacuated failing to livemigate
+ VM fail to live migrate after evacuation

Ghada Khalil (gkhalil) on 2019-07-15

tags:	added: stx.2.0
Changed in starlingx:
importance:	Undecided → High
status:	New → Triaged

Revision history for this message

yong hu (yhu6) wrote on 2019-07-15:

@zhipeng, please analyze this LP to check if it is a STX issue or upstream Nova issue.

Changed in starlingx:
assignee:	yong hu (yhu6) → nobody
assignee:	nobody → zhipeng liu (zhipengs)

Revision history for this message

zhipeng liu (zhipengs) wrote on 2019-07-16:

Hi Manokeran,

Could you help share TIS_AUTOMATION.log
How about the flavor info of the VM. You used dedicated cpu mode?
Could you also check if it is OK only do live migration from compute-0 to compute-1, then back to compute-0
It may not related to first step of evacuation.

From log what I can see is
Failed to compute_task_migrate_server: No valid host was found. There are not enough hosts available.
Not found specific cause so far. Need more information.

Thanks!
Zhipeng

yong hu (yhu6) on 2019-07-19

Changed in starlingx:
assignee:	zhipeng liu (zhipengs) → Shuquan Huang (shuquan)

hutianhao27 (hutianhao) on 2019-07-22

Changed in starlingx:
assignee:	Shuquan Huang (shuquan) → hutianhao27 (hutianhao)

YU CHENGDE (chant) on 2019-07-30

Changed in starlingx:
assignee:	hutianhao27 (hutianhao) → YU CHENGDE (chant)

Revision history for this message

YU CHENGDE (chant) wrote on 2019-07-30:

Hi,
  By testing in our site, VM works normally while instance live-migrates after evacuates.
  The detail information listed below.
    Thanks.
    Chengde

  Environment
   2 control nodes
   2 compute nodes
   with Ceph, shared storage
https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard

Deployment
Get 0717 mirror from CENGN
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/20190717T013000Z/outputs/iso/bootimage.iso

Revision history for this message

yong hu (yhu6) wrote on 2019-07-30:

@chant, please make futher analysis based on the discussion in the meeting:

1. CPU spec, RAM, any flavor extra information might be relevant, please provide this kind of info.
2. looked into the original log and had an analysis why in the first place.
- logs in Nova-* containers (pods or services) and
- libvirtd related logs

Revision history for this message

YU CHENGDE (chant) wrote on 2019-07-31: Re: [Bug 1836402] Re: VM fail to live migrate after evacuation

Download full text (40.6 KiB)

Hi,
   Copy that. Thanks for your meeting-minutes
   Thanks
                ChengDe.

> yong hu <email address hidden> 於 2019年7月30日下午9:58 寫道：
>
> @chant, please make futher analysis based on the discussion in the
> meeting:
>
> 1. CPU spec, RAM, any flavor extra information might be relevant, please provide this kind of info.
> 2. looked into the original log and had an analysis why in the first place.
> - logs in Nova-* containers (pods or services) and
> - libvirtd related logs
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1836402
>
> Title:
> VM fail to live migrate after evacuation
>
> Status in StarlingX:
> Triaged
>
> Bug description:
> Brief Description
> -----------------
>
> It was observed after VM was evacuated live
> migration was failing on same VM. Eg VM UUID ab3048c4-b10f-49ad-
> a93d-16dc24c97b93 evacuated successfully from compute-0 to compute-1
> then fails Live migration any of the other computes from compute-1.
> Below steps are showed on cli.
>
> ysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --
> os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>
> +-------------------------------------+-------------------------------------------------------------+
>
> | Field | Value
> |
>
> +-------------------------------------+-------------------------------------------------------------+
>
> | OS-DCF:diskConfig | MANUAL
> |
>
> | OS-EXT-AZ:availability_zone | nova
> |
>
> | OS-EXT-SRV-ATTR:host | compute-0
> |
>
> | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0
> |
>
> | OS-EXT-SRV-ATTR:instance_name | instance-00000016
> |
>
> | OS-EXT-STS:power_state | Running
> |
>
> | OS-EXT-STS:task_state | None
> |
>
> | OS-EXT-STS:vm_state | active
> |
>
> | OS-SRV-USG:launched_at | 2019-07-12T15:23:56.000000
> |
>
> | OS-SRV-USG:terminated_at | None
> |
>
> | accessIPv4 |
> |
>
> | accessIPv6 |
> |
>
> | addresses | tenant1-mgmt-
> net=192.168.146.55; tenant1-net1=172.16.1.159 |
>
> | config_drive |
> |
>
> | created | 2019-07-12T15:23:45Z
> |
>
> | flavor | flavor-default-size2 (0c7d0962
> -bf0c-44fd-a5b9-ff6b9d0a93b3) |
>
> | hostId |
>...

Hi, 
   Copy that. Thanks for your meeting-minutes
   Thanks 
                ChengDe.

> yong hu <1836402@bugs.launchpad.net> 於 2019年7月30日 下午9:58 寫道：
> 
> @chant, please make futher analysis based on the discussion in the
> meeting:
> 
> 1. CPU spec, RAM, any flavor extra information might be relevant, please provide this kind of info.
> 2. looked into the original log and had an analysis why in the first place.
>    - logs in Nova-* containers (pods or services) and 
>    - libvirtd related logs
> 
> -- 
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1836402
> 
> Title:
>  VM fail to live migrate after evacuation
> 
> Status in StarlingX:
>  Triaged
> 
> Bug description:
>  Brief Description
>  -----------------
> 
>                    It was observed after VM was evacuated live
>  migration was failing on same VM. Eg   VM UUID ab3048c4-b10f-49ad-
>  a93d-16dc24c97b93 evacuated successfully from compute-0 to compute-1
>  then fails Live migration any of the other computes from compute-1.
>  Below steps are showed on cli.
> 
>  ysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | Field                               | Value
>  |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | OS-DCF:diskConfig                   | MANUAL
>  |
> 
>  | OS-EXT-AZ:availability_zone         | nova
>  |
> 
>  | OS-EXT-SRV-ATTR:host                | compute-0
>  |
> 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0
>  |
> 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016
>  |
> 
>  | OS-EXT-STS:power_state              | Running
>  |
> 
>  | OS-EXT-STS:task_state               | None
>  |
> 
>  | OS-EXT-STS:vm_state                 | active
>  |
> 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:23:56.000000
>  |
> 
>  | OS-SRV-USG:terminated_at            | None
>  |
> 
>  | accessIPv4                          |
>  |
> 
>  | accessIPv6                          |
>  |
> 
>  | addresses                           | tenant1-mgmt-
>  net=192.168.146.55; tenant1-net1=172.16.1.159  |
> 
>  | config_drive                        |
>  |
> 
>  | created                             | 2019-07-12T15:23:45Z
>  |
> 
>  | flavor                              | flavor-default-size2 (0c7d0962
>  -bf0c-44fd-a5b9-ff6b9d0a93b3) |
> 
>  | hostId                              |
>  da8f07be2656cda0f4faff2389b7ee95d00879957c5b4bbea77fe622    |
> 
>  | id                                  | ab3048c4-b10f-49ad-a93d-
>  16dc24c97b93                        |
> 
>  | image                               |
>  |
> 
>  | key_name                            | keypair-tenant1
>  |
> 
>  | name                                | tenant1-vm-1
>  |
> 
>  | progress                            | 0
>  |
> 
>  | project_id                          |
>  196a1151724d4d5d8fe6e24bfd4acf5f                            |
> 
>  | properties                          |
>  |
> 
>  | security_groups                     | name='default'
>  |
> 
>  |                                     | name='default'
>  |
> 
>  | status                              | ACTIVE
>  |
> 
>  | updated                             | 2019-07-12T15:23:56Z
>  |
> 
>  | user_id                             |
>  ffd19119f10c4ab1aa99a9608a66923a                            |
> 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-
>  2fc3de8274c3'                   |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ echo $?
> 
>  compute-0:~$ sudo reboot -f
>  sudo reboot -f
> 
>  Rebooting.
> 
>  packet_write_wait: Connection to 192.168.204.138 port 22: Broken pipe
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne host-list
>  system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type int 
>  ernalURL --os-region-name RegionOne host-list
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | id | hostname     | personality | administrative | operational |
>  availability |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | 1  | controller-0 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 2  | controller-1 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 3  | compute-1    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 4  | compute-2    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 5  | compute-0    | worker      | unlocked       | disabled    |
>  offline      |
> 
>  | 6  | storage-0    | storage     | unlocked       | enabled     |
>  available    |
> 
>  | 7  | storage-1    | storage     | unlocked       | enabled     |
>  available    |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | id | hostname     | personality | administrative | operational |
>  availability |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | 1  | controller-0 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 2  | controller-1 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 3  | compute-1    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 4  | compute-2    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 5  | compute-0    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 6  | storage-0    | storage     | unlocked       | enabled     |
>  available    |
> 
>  | 7  | storage-1    | storage     | unlocked       | enabled     |
>  available    |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
>  ---------------------+-------------------------------------------+
> 
>  | Property            | Value                                     |
> 
>  +---------------------+-------------------------------------------+
> 
>  | action              | none                                      |
> 
>  | administrative      | unlocked                                  |
> 
>  | availability        | available                                 |
> 
>  | bm_ip               | 128.224.64.219                            |
> 
>  | bm_type             | bmc                                       |
> 
>  | bm_username         | root                                      |
> 
>  | boot_device         | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |
> 
>  | capabilities        | {}                                        |
> 
>  | config_applied      | 2dfb83d6-9616-4509-b8d4-e3c204d443ab      |
> 
>  | config_status       | None                                      |
> 
>  | config_target       | 2dfb83d6-9616-4509-b8d4-e3c204d443ab      |
> 
>  | console             | ttyS0,115200                              |
> 
>  | created_at          | 2019-07-11T19:47:49.078714+00:00          |
> 
>  | hostname            | compute-0                                 |
> 
>  | id                  | 5                                         |
> 
>  | install_output      | text                                      |
> 
>  | install_state       | completed                                 |
> 
>  | install_state_info  | None                                      |
> 
>  | invprovision        | provisioned                               |
> 
>  | location            | {}                                        |
> 
>  | mgmt_ip             | 192.168.204.138                           |
> 
>  | mgmt_mac            | 90:e2:ba:c6:90:60                         |
> 
>  | operational         | enabled                                   |
> 
>  | personality         | worker                                    |
> 
>  | reserved            | False                                     |
> 
>  | rootfs_device       | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |
> 
>  | serialid            | None                                      |
> 
>  | software_load       | 19.01                                     |
> 
>  | subfunctions        | worker,lowlatency                         |
> 
>  | task                |                                           |
> 
>  | tboot               | false                                     |
> 
>  | ttys_dcd            | None                                      |
> 
>  | updated_at          | 2019-07-12T15:30:39.811171+00:00          |
> 
>  | uptime              | 203                                       |
> 
>  | uuid                | fae1ee1f-a5ed-4ae5-b68b-ab3e9711ef24      |
> 
>  | vim_progress_status | services-enabled                          |
> 
>  +---------------------+-------------------------------------------+
> 
> 
>  ---------------------+-------------------------------------------+
> 
>  | Property            | Value                                     |
> 
>  +---------------------+-------------------------------------------+
> 
>  | action              | none                                      |
> 
>  | administrative      | unlocked                                  |
> 
>  | availability        | available                                 |
> 
>  | bm_ip               | 128.224.64.219                            |
> 
>  | bm_type             | bmc                                       |
> 
>  | bm_username         | root                                      |
> 
>  | boot_device         | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |
> 
>  | capabilities        | {}                                        |
> 
>  | config_applied      | 2dfb83d6-9616-4509-b8d4-e3c204d443ab      |
> 
>  | config_status       | None                                      |
> 
>  | config_target       | 2dfb83d6-9616-4509-b8d4-e3c204d443ab      |
> 
>  | console             | ttyS0,115200                              |
> 
>  | created_at          | 2019-07-11T19:47:49.078714+00:00          |
> 
>  | hostname            | compute-0                                 |
> 
>  | id                  | 5                                         |
> 
>  | install_output      | text                                      |
> 
>  | install_state       | completed                                 |
> 
>  | install_state_info  | None                                      |
> 
>  | invprovision        | provisioned                               |
> 
>  | location            | {}                                        |
> 
>  | mgmt_ip             | 192.168.204.138                           |
> 
>  | mgmt_mac            | 90:e2:ba:c6:90:60                         |
> 
>  | operational         | enabled                                   |
> 
>  | personality         | worker                                    |
> 
>  | reserved            | False                                     |
> 
>  | rootfs_device       | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |
> 
>  | serialid            | None                                      |
> 
>  | software_load       | 19.01                                     |
> 
>  | subfunctions        | worker,lowlatency                         |
> 
>  | task                |                                           |
> 
>  | tboot               | false                                     |
> 
>  | ttys_dcd            | None                                      |
> 
>  | updated_at          | 2019-07-12T15:30:39.811171+00:00          |
> 
>  | uptime              | 203                                       |
> 
>  | uuid                | fae1ee1f-a5ed-4ae5-b68b-ab3e9711ef24      |
> 
>  | vim_progress_status | services-enabled                          |
> 
>  +---------------------+-------------------------------------------+
> 
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne live-migration ab3048c4-b10f-49ad-a93d-16dc24c97b93 compute-0
>  nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-e 
>  ndpoint-type internalURL --os-region-name RegionOne live-migration ab3048c4-b10f-49ad-a93d-16dc24c97b93 compute-0
> 
>  sysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | Field                               | Value
>  |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | OS-DCF:diskConfig                   | MANUAL
>  |
> 
>  | OS-EXT-AZ:availability_zone         | nova
>  |
> 
>  | OS-EXT-SRV-ATTR:host                | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016
>  |
> 
>  | OS-EXT-STS:power_state              | Running
>  |
> 
>  | OS-EXT-STS:task_state               | None
>  |
> 
>  | OS-EXT-STS:vm_state                 | active
>  |
> 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:24:52.000000
>  |
> 
>  | OS-SRV-USG:terminated_at            | None
>  |
> 
>  | accessIPv4                          |
>  |
> 
>  | accessIPv6                          |
>  |
> 
>  | addresses                           | tenant1-mgmt-
>  net=192.168.146.55; tenant1-net1=172.16.1.159  |
> 
>  | config_drive                        |
>  |
> 
>  | created                             | 2019-07-12T15:23:45Z
>  |
> 
>  | flavor                              | flavor-default-size2 (0c7d0962
>  -bf0c-44fd-a5b9-ff6b9d0a93b3) |
> 
>  | hostId                              |
>  a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c    |
> 
>  | id                                  | ab3048c4-b10f-49ad-a93d-
>  16dc24c97b93                        |
> 
>  | image                               |
>  |
> 
>  | key_name                            | keypair-tenant1
>  |
> 
>  | name                                | tenant1-vm-1
>  |
> 
>  | progress                            | 0
>  |
> 
>  | project_id                          |
>  196a1151724d4d5d8fe6e24bfd4acf5f                            |
> 
>  | properties                          |
>  |
> 
>  | security_groups                     | name='default'
>  |
> 
>  |                                     | name='default'
>  |
> 
>  | status                              | ACTIVE
>  |
> 
>  | updated                             | 2019-07-12T15:32:01Z
>  |
> 
>  | user_id                             |
>  ffd19119f10c4ab1aa99a9608a66923a                            |
> 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-
>  2fc3de8274c3'                   |
> 
>  +-------------------------------------+-------------------------------------------------------------
> 
>   [sysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a
> 
>  +--------------------------------------+--------------+--------+------------------------------------------------------------+-------+----------------------+
> 
>  | ID                                   | Name         | Status |
>  Networks                                                   | Image |
>  Flavor               |
> 
>  +--------------------------------------+--------------+--------+------------------------------------------------------------+-------+----------------------+
> 
>  | ab3048c4-b10f-49ad-a93d-16dc24c97b93 | tenant1-vm-1 | ACTIVE |
>  tenant1-mgmt-net=192.168.146.55; tenant1-net1=172.16.1.159 |       |
>  flavor-default-size2 |
> 
>  +--------------------------------------+--------------+--------+------------------------------------------------------------+-------+----------------------+
> 
> 
> 
>  ]$ nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne migration-list
>  nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-e 
>  ndpoint-type internalURL --os-region-name RegionOne migration-list
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  | Id | UUID                                 | Source Node | Dest Node
>  | Source Compute | Dest Compute | Dest Host       | Status    |
>  Instance UUID                        | Old Flavor | New Flavor |
>  Created At                 | Updated At                 | Type
>  |
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  | 31 | 3fc2d997-3c9c-44f6-8991-39ad3c87ed12 | -           | -
>  | compute-1      | -            | -               | error     |
>  ab3048c4-b10f-49ad-a93d-16dc24c97b93 | 46         | 46         |
>  2019-07-12T15:31:59.000000 | 2019-07-12T15:32:01.000000 | live-
>  migration |
> 
>  | 28 | 5e82192b-f11f-4829-8a61-6ee196ecbb32 | compute-0   | compute-1
>  | compute-0      | compute-1    | 192.168.205.236 | completed |
>  ab3048c4-b10f-49ad-a93d-16dc24c97b93 | None       | None       |
>  2019-07-12T15:24:34.000000 | 2019-07-12T15:30:34.000000 | evacuation
>  |
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ echo $?
> 
>  Severity
>  --------
>  Major
>  Steps to Reproduce
>  ------------------
>  1. Launch VM . VM was created in compute-0
>  n)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | Field                               | Value
>  |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | OS-DCF:diskConfig                   | MANUAL
>  |
> 
>  | OS-EXT-AZ:availability_zone         | nova
>  |
> 
>  | OS-EXT-SRV-ATTR:host                | compute-0
>  |
> 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0
>  |
> 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016
>  |
> 
>  | OS-EXT-STS:power_state              | Running
>  |
> 
>  | OS-EXT-STS:task_state               | None
>  |
> 
>  | OS-EXT-STS:vm_state                 | active
>  |
> 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:23:56.000000
>  |
> 
>  | OS-SRV-USG:terminated_at            | None
>  |
> 
>  | accessIPv4                          |
>  |
> 
>  | accessIPv6                          |
>  |
> 
>  | addresses                           | tenant1-mgmt-
>  net=192.168.146.55; tenant1-net1=172.16.1.159  |
> 
>  | config_drive                        |
>  |
> 
>  | created                             | 2019-07-12T15:23:45Z
>  |
> 
>  | flavor                              | flavor-default-size2 (0c7d0962
>  -bf0c-44fd-a5b9-ff6b9d0a93b3) |
> 
>  | hostId                              |
>  da8f07be2656cda0f4faff2389b7ee95d00879957c5b4bbea77fe622    |
> 
>  | id                                  | ab3048c4-b10f-49ad-a93d-
>  16dc24c97b93                        |
> 
>  | image                               |
>  |
> 
>  | key_name                            | keypair-tenant1
>  |
> 
>  | name                                | tenant1-vm-1
>  |
> 
>  | progress                            | 0
>  |
> 
>  | project_id                          |
>  196a1151724d4d5d8fe6e24bfd4acf5f                            |
> 
>  | properties                          |
>  |
> 
>  | security_groups                     | name='default'
>  |
> 
>  |                                     | name='default'
>  |
> 
>  | status                              | ACTIVE
>  |
> 
>  | updated                             | 2019-07-12T15:23:56Z
>  |
> 
>  | user_id                             |
>  ffd19119f10c4ab1aa99a9608a66923a                            |
> 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-
>  2fc3de8274c3'                   |
> 
>  +-------------------------------------+-------------------------------------------------------------+
>  2. Reboot compute-0 sudo reboot –f
>  3. wait for compute  to come up.
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | id | hostname     | personality | administrative | operational |
>  availability |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | 1  | controller-0 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 2  | controller-1 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 3  | compute-1    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 4  | compute-2    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 5  | compute-0    | worker      | unlocked       | disabled    |
>  offline      |
> 
>  | 6  | storage-0    | storage     | unlocked       | enabled     |
>  available    |
> 
>  | 7  | storage-1    | storage     | unlocked       | enabled     |
>  available    |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | id | hostname     | personality | administrative | operational |
>  availability |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | 1  | controller-0 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 2  | controller-1 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 3  | compute-1    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 4  | compute-2    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 5  | compute-0    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 6  | storage-0    | storage     | unlocked       | enabled     |
>  available    |
> 
>  | 7  | storage-1    | storage     | unlocked       | enabled     |
>  available    |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  4.Verify VM evacuation VM running on compute-1
> 
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93 
>  +-------------------------------------+-------------------------------------------------------------+ 
>  | Field                               | Value                                                       | 
>  +-------------------------------------+-------------------------------------------------------------+ 
>  | OS-DCF:diskConfig                   | MANUAL                                                      | 
>  | OS-EXT-AZ:availability_zone         | nova                                                        | 
>  | OS-EXT-SRV-ATTR:host                | compute-1                                                   | 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1                                                   | 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016                                           | 
>  | OS-EXT-STS:power_state              | Running                                                     | 
>  | OS-EXT-STS:task_state               | None                                                        | 
>  | OS-EXT-STS:vm_state                 | active                                                      | 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:24:52.000000                                  | 
>  | OS-SRV-USG:terminated_at            | None                                                        | 
>  | accessIPv4                          |                                                             | 
>  | accessIPv6                          |                                                             | 
>  | addresses                           | tenant1-mgmt-net=192.168.146.55; tenant1-net1=172.16.1.159  | 
>  | config_drive                        |                                                             | 
>  | created                             | 2019-07-12T15:23:45Z                                        | 
>  | flavor                              | flavor-default-size2 (0c7d0962-bf0c-44fd-a5b9-ff6b9d0a93b3) | 
>  | hostId                              | a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c    | 
>  | id                                  | ab3048c4-b10f-49ad-a93d-16dc24c97b93                        | 
>  | image                               |                                                             | 
>  | key_name                            | keypair-tenant1                                             | 
>  | name                                | tenant1-vm-1                                                | 
>  | progress                            | 0                                                           | 
>  | project_id                          | 196a1151724d4d5d8fe6e24bfd4acf5f                            | 
>  | properties                          |                                                             | 
>  | security_groups                     | name='default'                                              | 
>  |                                     | name='default'                                              | 
>  | status                              | ACTIVE                                                      | 
>  | updated                             | 2019-07-12T15:30:33Z                                        | 
>  | user_id                             | ffd19119f10c4ab1aa99a9608a66923a                            | 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-2fc3de8274c3'                   | 
>  +-------------------------------------+-------------------------------------------------------------+ 
>  [sysadmin@controller-1 ~(keystone_admin)]$ echo $?
> 
> 
> 
>  5. Livemigrate VM after evacute
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne live-migration ab3048c4-b10f-49ad-a93d-16dc24c97b93 compute-0
>  nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-e 
>  ndpoint-type internalURL --os-region-name RegionOne live-migration ab3048c4-b10f-49ad-a93d-16dc24c97b93 compute-0
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ echo $?
>  echo $?
> 
>  0
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | Field                               | Value
>  |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | OS-DCF:diskConfig                   | MANUAL
>  |
> 
>  | OS-EXT-AZ:availability_zone         | nova
>  |
> 
>  | OS-EXT-SRV-ATTR:host                | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016
>  |
> 
>  | OS-EXT-STS:power_state              | Running
>  |
> 
>  | OS-EXT-STS:task_state               | migrating
>  |
> 
>  | OS-EXT-STS:vm_state                 | active
>  |
> 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:24:52.000000
>  |
> 
>  | OS-SRV-USG:terminated_at            | None
>  |
> 
>  | accessIPv4                          |
>  |
> 
>  | accessIPv6                          |
>  |
> 
>  | addresses                           | tenant1-mgmt-
>  net=192.168.146.55; tenant1-net1=172.16.1.159  |
> 
>  | config_drive                        |
>  |
> 
>  | created                             | 2019-07-12T15:23:45Z
>  |
> 
>  | flavor                              | flavor-default-size2 (0c7d0962
>  -bf0c-44fd-a5b9-ff6b9d0a93b3) |
> 
>  | hostId                              |
>  a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c    |
> 
>  | id                                  | ab3048c4-b10f-49ad-a93d-
>  16dc24c97b93                        |
> 
>  | image                               |
>  |
> 
>  | key_name                            | keypair-tenant1
>  |
> 
>  | name                                | tenant1-vm-1
>  |
> 
>  | progress                            | 0
>  |
> 
>  | project_id                          |
>  196a1151724d4d5d8fe6e24bfd4acf5f                            |
> 
>  | properties                          |
>  |
> 
>  | security_groups                     | name='default'
>  |
> 
>  |                                     | name='default'
>  |
> 
>  | status                              | MIGRATING
>  |
> 
>  | updated                             | 2019-07-12T15:31:59Z
>  |
> 
>  | user_id                             |
>  ffd19119f10c4ab1aa99a9608a66923a                            |
> 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-
>  2fc3de8274c3'                   |
> 
>  	
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | Field                               | Value
>  |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | OS-DCF:diskConfig                   | MANUAL
>  |
> 
>  | OS-EXT-AZ:availability_zone         | nova
>  |
> 
>  | OS-EXT-SRV-ATTR:host                | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016
>  |
> 
>  | OS-EXT-STS:power_state              | Running
>  |
> 
>  | OS-EXT-STS:task_state               | None
>  |
> 
>  | OS-EXT-STS:vm_state                 | active
>  |
> 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:24:52.000000
>  |
> 
>  | OS-SRV-USG:terminated_at            | None
>  |
> 
>  | accessIPv4                          |
>  |
> 
>  | accessIPv6                          |
>  |
> 
>  | addresses                           | tenant1-mgmt-
>  net=192.168.146.55; tenant1-net1=172.16.1.159  |
> 
>  | config_drive                        |
>  |
> 
>  | created                             | 2019-07-12T15:23:45Z
>  |
> 
>  | flavor                              | flavor-default-size2 (0c7d0962
>  -bf0c-44fd-a5b9-ff6b9d0a93b3) |
> 
>  | hostId                              |
>  a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c    |
> 
>  | id                                  | ab3048c4-b10f-49ad-a93d-
>  16dc24c97b93                        |
> 
>  | image                               |
>  |
> 
>  | key_name                            | keypair-tenant1
>  |
> 
>  | name                                | tenant1-vm-1
>  |
> 
>  | progress                            | 0
>  |
> 
>  | project_id                          |
>  196a1151724d4d5d8fe6e24bfd4acf5f                            |
> 
>  | properties                          |
>  |
> 
>  | security_groups                     | name='default'
>  |
> 
>  |                                     | name='default'
>  |
> 
>  | status                              | ACTIVE
>  |
> 
>  | updated                             | 2019-07-12T15:32:01Z
>  |
> 
>  | user_id                             |
>  ffd19119f10c4ab1aa99a9608a66923a                            |
> 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-
>  2fc3de8274c3'                   |
> 
>  +-------------------------------------+-------------------------------------------------------------
> 
>  6.Migration list shows error to migrate from  compute-1
> 
> 
>  nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne migration-list
>  nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-e 
>  ndpoint-type internalURL --os-region-name RegionOne migration-list
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  | Id | UUID                                 | Source Node | Dest Node
>  | Source Compute | Dest Compute | Dest Host       | Status    |
>  Instance UUID                        | Old Flavor | New Flavor |
>  Created At                 | Updated At                 | Type
>  |
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  | 31 | 3fc2d997-3c9c-44f6-8991-39ad3c87ed12 | -           | -
>  | compute-1      | -            | -               | error     |
>  ab3048c4-b10f-49ad-a93d-16dc24c97b93 | 46         | 46         |
>  2019-07-12T15:31:59.000000 | 2019-07-12T15:32:01.000000 | live-
>  migration |
> 
>  | 28 | 5e82192b-f11f-4829-8a61-6ee196ecbb32 | compute-0   | compute-1
>  | compute-0      | compute-1    | 192.168.205.236 | completed |
>  ab3048c4-b10f-49ad-a93d-16dc24c97b93 | None       | None       |
>  2019-07-12T15:24:34.000000 | 2019-07-12T15:30:34.000000 | evacuation
>  |
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  System Configuration
>  --------------------
>  Regular system
> 
> 
>  Expected Behavior
>  ------------------
>  Able to migrate after evacuate. 
>  Actual Behavior
>  ----------------
>  Failure on migration after evacuate.
>  Reproducibility
>  ---------------
>  100% reproducible
>  System Configuration
>  --------------------
>  Regular system
>  Load 
>  ----
>  2019-07-11 01:30:00 +0000
>  Last Pass
>  ---------
>  Not sure in this lab.
> 
>  Timestamp/Logs
>  --------------
>  2019-07-12T15:31:59.000000
> 
>  Test Activity
>  -------------
>  Regression test
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/starlingx/+bug/1836402/+subscriptions

Revision history for this message

ya.wang (ya.wang) wrote on 2019-07-31:

Download full text (9.5 KiB)

@yhu6, we have retest these bug, the steps are:

1. Create an instance on compute-0(with any flavor extra specs and image properties)
2. reboot compute-0
3. wait the evacuate success
4. wait the compute-0's services up
5. live migrate instance to compute-0

At the beginning our test show that live migrate works well. Then we found that our test step in the fourth part is different from the reporter's step. We use `openstack compute service list` to ensure that compute-0's service is available.

controller-0:~$ openstack compute service list
+----+------------------+-----------------------------------+----------+---------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+----+------------------+-----------------------------------+----------+---------+-------+----------------------------+
| 29 | nova-compute | compute-1 | nova | enabled | up | 2019-07-31T13:55:37.000000 |
| 32 | nova-compute | compute-0 | nova | enabled | up | 2019-07-31T13:55:37.000000 |
| 50 | nova-consoleauth | nova-consoleauth-748bffc767-vslb5 | internal | enabled | up | 2019-07-31T13:55:38.000000 |
| 52 | nova-conductor | nova-conductor-5977dbb7c5-jjfwm | internal | enabled | up | 2019-07-31T13:55:35.000000 |
| 54 | nova-scheduler | nova-scheduler-6f78459858-gbtnb | internal | enabled | up | 2019-07-31T13:55:36.000000 |
| 61 | nova-consoleauth | nova-consoleauth-748bffc767-s5lsp | internal | enabled | up | 2019-07-31T13:55:32.000000 |
| 62 | nova-scheduler | nova-scheduler-6f78459858-46shx | internal | enabled | up | 2019-07-31T13:55:39.000000 |
| 63 | nova-conductor | nova-conductor-5977dbb7c5-l5f6x | internal | enabled | up | 2019-07-31T13:55:39.000000 |
+----+------------------+-----------------------------------+----------+---------+-------+----------------------------+

We also check the k8s pod's status.

@yhu6, we have retest these bug, the steps are:

controller-0:~$ openstack compute service list
+----+------------------+-----------------------------------+----------+---------+-------+----------------------------+
| ID | Binary           | Host                              | Zone     | Status  | State | Updated At                 |
+----+------------------+-----------------------------------+----------+---------+-------+----------------------------+
| 29 | nova-compute     | compute-1                         | nova     | enabled | up    | 2019-07-31T13:55:37.000000 |
| 32 | nova-compute     | compute-0                         | nova     | enabled | up    | 2019-07-31T13:55:37.000000 |
| 50 | nova-consoleauth | nova-consoleauth-748bffc767-vslb5 | internal | enabled | up    | 2019-07-31T13:55:38.000000 |
| 52 | nova-conductor   | nova-conductor-5977dbb7c5-jjfwm   | internal | enabled | up    | 2019-07-31T13:55:35.000000 |
| 54 | nova-scheduler   | nova-scheduler-6f78459858-gbtnb   | internal | enabled | up    | 2019-07-31T13:55:36.000000 |
| 61 | nova-consoleauth | nova-consoleauth-748bffc767-s5lsp | internal | enabled | up    | 2019-07-31T13:55:32.000000 |
| 62 | nova-scheduler   | nova-scheduler-6f78459858-46shx   | internal | enabled | up    | 2019-07-31T13:55:39.000000 |
| 63 | nova-conductor   | nova-conductor-5977dbb7c5-l5f6x   | internal | enabled | up    | 2019-07-31T13:55:39.000000 |
+----+------------------+-----------------------------------+----------+---------+-------+----------------------------+

We also check the k8s pod's status.

controller-0:/home/sysadmin# kubectl get pod -n openstack
NAME                                                 READY   STATUS      RESTARTS   AGE
cinder-api-64955798b7-9c9mh                          1/1     Running     0          4h47m
cinder-api-64955798b7-gbjlx                          1/1     Running     1          10h
cinder-backup-77fb945bb-6qpz5                        1/1     Running     0          10h
cinder-backup-77fb945bb-mxg78                        1/1     Running     0          4h47m
cinder-scheduler-974d7bc9b-nbkxl                     1/1     Running     0          10h
cinder-scheduler-974d7bc9b-w259r                     1/1     Running     0          4h47m
cinder-volume-56fd7994cc-vsb2h                       1/1     Running     0          10h
cinder-volume-56fd7994cc-xwr9w                       1/1     Running     0          4h47m
cinder-volume-usage-audit-1564580700-qfqj7           0/1     Completed   0          12m
cinder-volume-usage-audit-1564581000-lp9vj           0/1     Completed   0          7m33s
cinder-volume-usage-audit-1564581300-lmmk4           0/1     Completed   0          2m30s
glance-api-7574db7ff9-pfg9s                          1/1     Running     1          10h
glance-api-7574db7ff9-q8z7m                          1/1     Running     1          4h47m
heat-api-7c5c769c99-9sgzw                            1/1     Running     0          4h47m
heat-api-7c5c769c99-lhrv7                            1/1     Running     1          10h
heat-cfn-7f76797c9f-8nptm                            1/1     Running     1          10h
heat-cfn-7f76797c9f-v8xqn                            1/1     Running     0          4h47m
heat-engine-6b94b76595-7j28d                         1/1     Running     0          10h
heat-engine-6b94b76595-dz8vx                         1/1     Running     0          4h47m
heat-engine-cleaner-1564580700-4d8wm                 0/1     Completed   0          12m
heat-engine-cleaner-1564581000-br2zd                 0/1     Completed   0          7m33s
heat-engine-cleaner-1564581300-tzmmj                 0/1     Completed   0          2m30s
horizon-6fcbcdcbfb-8cpw8                             1/1     Running     0          4h41m
ingress-bdfbc4ccc-2wlnk                              1/1     Running     0          4h47m
ingress-bdfbc4ccc-w9kxv                              1/1     Running     0          10h
ingress-error-pages-7b789b5df8-hrtr9                 1/1     Running     0          4h47m
ingress-error-pages-7b789b5df8-qbtfb                 1/1     Running     0          10h
keystone-api-6fffbb6f7c-hgmlk                        1/1     Running     5          10h
keystone-api-6fffbb6f7c-qcd62                        1/1     Running     0          4h47m
keystone-fernet-rotate-1564574400-6xfpv              0/1     Completed   0          117m
libvirt-libvirt-default-k9qg2                        1/1     Running     2          6d7h
libvirt-libvirt-default-v5pgf                        1/1     Running     2          6d7h
mariadb-ingress-6ff964556d-5btgl                     1/1     Running     0          10h
mariadb-ingress-6ff964556d-bjmwh                     1/1     Running     0          4h47m
mariadb-ingress-error-pages-764cfd869b-bdv94         1/1     Running     0          8h
mariadb-server-0                                     1/1     Running     0          4h40m
mariadb-server-1                                     1/1     Running     0          10h
neutron-dhcp-agent-compute-0-75ea0372-vv4xq          1/1     Running     2          6d7h
neutron-dhcp-agent-compute-1-271fedba-74znj          1/1     Running     2          6d7h
neutron-l3-agent-compute-0-75ea0372-5hlh6            1/1     Running     2          6d7h
neutron-l3-agent-compute-1-eae26dba-h8v4q            1/1     Running     2          6d7h
neutron-metadata-agent-compute-0-75ea0372-dlwq2      1/1     Running     2          6d7h
neutron-metadata-agent-compute-1-eae26dba-2wfg2      1/1     Running     2          6d7h
neutron-ovs-agent-compute-0-75ea0372-b9kqh           1/1     Running     2          6d7h
neutron-ovs-agent-compute-1-eae26dba-p58rm           1/1     Running     24         6d7h
neutron-server-df4d6757b-wnq2l                       1/1     Running     1          10h
neutron-server-df4d6757b-xjlrt                       1/1     Running     0          4h47m
neutron-sriov-agent-compute-0-75ea0372-rwcp7         1/1     Running     2          6d7h
neutron-sriov-agent-compute-1-eae26dba-f76mk         1/1     Running     2          6d7h
nova-api-metadata-7c64c8458f-467r6                   1/1     Running     1          10h
nova-api-metadata-7c64c8458f-5q645                   1/1     Running     0          4h47m
nova-api-osapi-56798b4b68-62bf9                      1/1     Running     0          10h
nova-api-osapi-56798b4b68-8ljzh                      1/1     Running     1          4h47m
nova-api-proxy-598684846d-7jtsv                      1/1     Running     0          8h
nova-compute-compute-0-75ea0372-kpzgj                2/2     Running     4          6d7h
nova-compute-compute-1-eae26dba-w4f9m                2/2     Running     26         6d7h
nova-conductor-5977dbb7c5-jjfwm                      1/1     Running     0          10h
nova-conductor-5977dbb7c5-l5f6x                      1/1     Running     0          4h47m
nova-consoleauth-748bffc767-s5lsp                    1/1     Running     0          4h47m
nova-consoleauth-748bffc767-vslb5                    1/1     Running     0          10h
nova-novncproxy-ff6456d77-jksz7                      1/1     Running     0          4h47m
nova-novncproxy-ff6456d77-qthmd                      1/1     Running     1          10h
nova-scheduler-6f78459858-46shx                      1/1     Running     0          4h47m
nova-scheduler-6f78459858-gbtnb                      1/1     Running     0          10h
openvswitch-db-6nctl                                 1/1     Running     2          6d7h
openvswitch-db-jghgk                                 1/1     Running     2          6d7h
openvswitch-vswitchd-4vfp2                           1/1     Running     2          6d7h
openvswitch-vswitchd-qv9bz                           1/1     Running     2          6d7h
osh-openstack-garbd-garbd-8d64b6886-498qt            1/1     Running     0          3h54m
osh-openstack-memcached-memcached-6c94979765-rtsmc   1/1     Running     0          8h
osh-openstack-rabbitmq-rabbitmq-0                    1/1     Running     0          4h41m
osh-openstack-rabbitmq-rabbitmq-1                    1/1     Running     1          10h
placement-api-5798c855bc-8hq8b                       1/1     Running     0          10h
placement-api-5798c855bc-l62f6                       1/1     Running     0          4h47m

When the both show that compute-0 is ready, we start live-migrate and it works well.

@anujeyan is using starlingx's command to ensure that compute-0's service is available. Therefore, we try to use this method to take a test. Use `watch -n5 "system host-list"` to monitor compute-0's status.
When start reboot compute-0, it shows compute-0 is offline, after waiting for a while, it shows compute-0 is available. At this moment, we use `openstack compute service list` to check compute-0's status, the output show that compute-0's status is 'enabled' and state is still 'down'. We also use `kubectl get pod -n namespace`, and the result show that compute-0's compute service is in init.
Try to live-migrate instance to compute-0, it'll be failed and the log show scheduler failed with "NoValidHost: No valid host was found", which is same as @anujeyan uploaded.

@yong, can you re-test it and ensure nova compute service is available before live-migrate?
If it still not work, please paste nova-scheduler debug log here.

Revision history for this message

yong hu (yhu6) wrote on 2019-08-01:

@wang Ya, pls confirm my understanding.
I think your conclusion was that you have used the different methods of checking compute services readiness from what the reporter used.
If following your methods, and services on compute node were actually running and ready, this live-migration did work normally.
On the other hand, as reported did, if only checking host availability (by system host-list) was not enough to confirm nova services ready to work.

If this is confirmed, we can say this issue is invalid.

Revision history for this message

YU CHENGDE (chant) wrote on 2019-08-01:

#10

Download full text (40.7 KiB)

Hi,
    Correct.
    Our re-testing method is same as your understanding.
    Thanks

> yong hu <email address hidden> 於 2019年8月1日下午12:21 寫道：
>
> @wang Ya, pls confirm my understanding.
> I think your conclusion was that you have used the different methods of checking compute services readiness from what the reporter used.
> If following your methods, and services on compute node were actually running and ready, this live-migration did work normally.
> On the other hand, as reported did, if only checking host availability (by system host-list) was not enough to confirm nova services ready to work.
>
> If this is confirmed, we can say this issue is invalid.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1836402
>
> Title:
> VM fail to live migrate after evacuation
>
> Status in StarlingX:
> Triaged
>
> Bug description:
> Brief Description
> -----------------
>
> It was observed after VM was evacuated live
> migration was failing on same VM. Eg VM UUID ab3048c4-b10f-49ad-
> a93d-16dc24c97b93 evacuated successfully from compute-0 to compute-1
> then fails Live migration any of the other computes from compute-1.
> Below steps are showed on cli.
>
> ysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --
> os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>
> +-------------------------------------+-------------------------------------------------------------+
>
> | Field | Value
> |
>
> +-------------------------------------+-------------------------------------------------------------+
>
> | OS-DCF:diskConfig | MANUAL
> |
>
> | OS-EXT-AZ:availability_zone | nova
> |
>
> | OS-EXT-SRV-ATTR:host | compute-0
> |
>
> | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0
> |
>
> | OS-EXT-SRV-ATTR:instance_name | instance-00000016
> |
>
> | OS-EXT-STS:power_state | Running
> |
>
> | OS-EXT-STS:task_state | None
> |
>
> | OS-EXT-STS:vm_state | active
> |
>
> | OS-SRV-USG:launched_at | 2019-07-12T15:23:56.000000
> |
>
> | OS-SRV-USG:terminated_at | None
> |
>
> | accessIPv4 |
> |
>
> | accessIPv6 |
> |
>
> | addresses | tenant1-mgmt-
> net=192.168.146.55; tenant1-net1=172.16.1.159 |
>
> | config_drive |
> |
>
> | created | 2019-07-12T...

Hi, 
    Correct. 
    Our re-testing method is same as your understanding. 
    Thanks

> yong hu <1836402@bugs.launchpad.net> 於 2019年8月1日 下午12:21 寫道：
> 
> @wang Ya, pls confirm my understanding. 
> I think your conclusion was that you have used the different methods of checking compute services readiness from what the reporter used.
> If following your methods, and services on compute node were actually running and ready, this live-migration did work normally. 
> On the other hand, as reported did, if only checking host availability (by system host-list) was not enough to confirm nova services ready to work.
> 
> If this is confirmed, we can say this issue is invalid.
> 
> -- 
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1836402
> 
> Title:
>  VM fail to live migrate after evacuation
> 
> Status in StarlingX:
>  Triaged
> 
> Bug description:
>  Brief Description
>  -----------------
> 
>                    It was observed after VM was evacuated live
>  migration was failing on same VM. Eg   VM UUID ab3048c4-b10f-49ad-
>  a93d-16dc24c97b93 evacuated successfully from compute-0 to compute-1
>  then fails Live migration any of the other computes from compute-1.
>  Below steps are showed on cli.
> 
>  ysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | Field                               | Value
>  |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | OS-DCF:diskConfig                   | MANUAL
>  |
> 
>  | OS-EXT-AZ:availability_zone         | nova
>  |
> 
>  | OS-EXT-SRV-ATTR:host                | compute-0
>  |
> 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0
>  |
> 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016
>  |
> 
>  | OS-EXT-STS:power_state              | Running
>  |
> 
>  | OS-EXT-STS:task_state               | None
>  |
> 
>  | OS-EXT-STS:vm_state                 | active
>  |
> 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:23:56.000000
>  |
> 
>  | OS-SRV-USG:terminated_at            | None
>  |
> 
>  | accessIPv4                          |
>  |
> 
>  | accessIPv6                          |
>  |
> 
>  | addresses                           | tenant1-mgmt-
>  net=192.168.146.55; tenant1-net1=172.16.1.159  |
> 
>  | config_drive                        |
>  |
> 
>  | created                             | 2019-07-12T15:23:45Z
>  |
> 
>  | flavor                              | flavor-default-size2 (0c7d0962
>  -bf0c-44fd-a5b9-ff6b9d0a93b3) |
> 
>  | hostId                              |
>  da8f07be2656cda0f4faff2389b7ee95d00879957c5b4bbea77fe622    |
> 
>  | id                                  | ab3048c4-b10f-49ad-a93d-
>  16dc24c97b93                        |
> 
>  | image                               |
>  |
> 
>  | key_name                            | keypair-tenant1
>  |
> 
>  | name                                | tenant1-vm-1
>  |
> 
>  | progress                            | 0
>  |
> 
>  | project_id                          |
>  196a1151724d4d5d8fe6e24bfd4acf5f                            |
> 
>  | properties                          |
>  |
> 
>  | security_groups                     | name='default'
>  |
> 
>  |                                     | name='default'
>  |
> 
>  | status                              | ACTIVE
>  |
> 
>  | updated                             | 2019-07-12T15:23:56Z
>  |
> 
>  | user_id                             |
>  ffd19119f10c4ab1aa99a9608a66923a                            |
> 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-
>  2fc3de8274c3'                   |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ echo $?
> 
>  compute-0:~$ sudo reboot -f
>  sudo reboot -f
> 
>  Rebooting.
> 
>  packet_write_wait: Connection to 192.168.204.138 port 22: Broken pipe
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne host-list
>  system --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://192.168.204.2:5000/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type int 
>  ernalURL --os-region-name RegionOne host-list
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | id | hostname     | personality | administrative | operational |
>  availability |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | 1  | controller-0 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 2  | controller-1 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 3  | compute-1    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 4  | compute-2    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 5  | compute-0    | worker      | unlocked       | disabled    |
>  offline      |
> 
>  | 6  | storage-0    | storage     | unlocked       | enabled     |
>  available    |
> 
>  | 7  | storage-1    | storage     | unlocked       | enabled     |
>  available    |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | id | hostname     | personality | administrative | operational |
>  availability |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | 1  | controller-0 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 2  | controller-1 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 3  | compute-1    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 4  | compute-2    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 5  | compute-0    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 6  | storage-0    | storage     | unlocked       | enabled     |
>  available    |
> 
>  | 7  | storage-1    | storage     | unlocked       | enabled     |
>  available    |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
>  ---------------------+-------------------------------------------+
> 
>  | Property            | Value                                     |
> 
>  +---------------------+-------------------------------------------+
> 
>  | action              | none                                      |
> 
>  | administrative      | unlocked                                  |
> 
>  | availability        | available                                 |
> 
>  | bm_ip               | 128.224.64.219                            |
> 
>  | bm_type             | bmc                                       |
> 
>  | bm_username         | root                                      |
> 
>  | boot_device         | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |
> 
>  | capabilities        | {}                                        |
> 
>  | config_applied      | 2dfb83d6-9616-4509-b8d4-e3c204d443ab      |
> 
>  | config_status       | None                                      |
> 
>  | config_target       | 2dfb83d6-9616-4509-b8d4-e3c204d443ab      |
> 
>  | console             | ttyS0,115200                              |
> 
>  | created_at          | 2019-07-11T19:47:49.078714+00:00          |
> 
>  | hostname            | compute-0                                 |
> 
>  | id                  | 5                                         |
> 
>  | install_output      | text                                      |
> 
>  | install_state       | completed                                 |
> 
>  | install_state_info  | None                                      |
> 
>  | invprovision        | provisioned                               |
> 
>  | location            | {}                                        |
> 
>  | mgmt_ip             | 192.168.204.138                           |
> 
>  | mgmt_mac            | 90:e2:ba:c6:90:60                         |
> 
>  | operational         | enabled                                   |
> 
>  | personality         | worker                                    |
> 
>  | reserved            | False                                     |
> 
>  | rootfs_device       | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |
> 
>  | serialid            | None                                      |
> 
>  | software_load       | 19.01                                     |
> 
>  | subfunctions        | worker,lowlatency                         |
> 
>  | task                |                                           |
> 
>  | tboot               | false                                     |
> 
>  | ttys_dcd            | None                                      |
> 
>  | updated_at          | 2019-07-12T15:30:39.811171+00:00          |
> 
>  | uptime              | 203                                       |
> 
>  | uuid                | fae1ee1f-a5ed-4ae5-b68b-ab3e9711ef24      |
> 
>  | vim_progress_status | services-enabled                          |
> 
>  +---------------------+-------------------------------------------+
> 
> 
>  ---------------------+-------------------------------------------+
> 
>  | Property            | Value                                     |
> 
>  +---------------------+-------------------------------------------+
> 
>  | action              | none                                      |
> 
>  | administrative      | unlocked                                  |
> 
>  | availability        | available                                 |
> 
>  | bm_ip               | 128.224.64.219                            |
> 
>  | bm_type             | bmc                                       |
> 
>  | bm_username         | root                                      |
> 
>  | boot_device         | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |
> 
>  | capabilities        | {}                                        |
> 
>  | config_applied      | 2dfb83d6-9616-4509-b8d4-e3c204d443ab      |
> 
>  | config_status       | None                                      |
> 
>  | config_target       | 2dfb83d6-9616-4509-b8d4-e3c204d443ab      |
> 
>  | console             | ttyS0,115200                              |
> 
>  | created_at          | 2019-07-11T19:47:49.078714+00:00          |
> 
>  | hostname            | compute-0                                 |
> 
>  | id                  | 5                                         |
> 
>  | install_output      | text                                      |
> 
>  | install_state       | completed                                 |
> 
>  | install_state_info  | None                                      |
> 
>  | invprovision        | provisioned                               |
> 
>  | location            | {}                                        |
> 
>  | mgmt_ip             | 192.168.204.138                           |
> 
>  | mgmt_mac            | 90:e2:ba:c6:90:60                         |
> 
>  | operational         | enabled                                   |
> 
>  | personality         | worker                                    |
> 
>  | reserved            | False                                     |
> 
>  | rootfs_device       | /dev/disk/by-path/pci-0000:85:00.0-nvme-1 |
> 
>  | serialid            | None                                      |
> 
>  | software_load       | 19.01                                     |
> 
>  | subfunctions        | worker,lowlatency                         |
> 
>  | task                |                                           |
> 
>  | tboot               | false                                     |
> 
>  | ttys_dcd            | None                                      |
> 
>  | updated_at          | 2019-07-12T15:30:39.811171+00:00          |
> 
>  | uptime              | 203                                       |
> 
>  | uuid                | fae1ee1f-a5ed-4ae5-b68b-ab3e9711ef24      |
> 
>  | vim_progress_status | services-enabled                          |
> 
>  +---------------------+-------------------------------------------+
> 
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne live-migration ab3048c4-b10f-49ad-a93d-16dc24c97b93 compute-0
>  nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-e 
>  ndpoint-type internalURL --os-region-name RegionOne live-migration ab3048c4-b10f-49ad-a93d-16dc24c97b93 compute-0
> 
>  sysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | Field                               | Value
>  |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | OS-DCF:diskConfig                   | MANUAL
>  |
> 
>  | OS-EXT-AZ:availability_zone         | nova
>  |
> 
>  | OS-EXT-SRV-ATTR:host                | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016
>  |
> 
>  | OS-EXT-STS:power_state              | Running
>  |
> 
>  | OS-EXT-STS:task_state               | None
>  |
> 
>  | OS-EXT-STS:vm_state                 | active
>  |
> 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:24:52.000000
>  |
> 
>  | OS-SRV-USG:terminated_at            | None
>  |
> 
>  | accessIPv4                          |
>  |
> 
>  | accessIPv6                          |
>  |
> 
>  | addresses                           | tenant1-mgmt-
>  net=192.168.146.55; tenant1-net1=172.16.1.159  |
> 
>  | config_drive                        |
>  |
> 
>  | created                             | 2019-07-12T15:23:45Z
>  |
> 
>  | flavor                              | flavor-default-size2 (0c7d0962
>  -bf0c-44fd-a5b9-ff6b9d0a93b3) |
> 
>  | hostId                              |
>  a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c    |
> 
>  | id                                  | ab3048c4-b10f-49ad-a93d-
>  16dc24c97b93                        |
> 
>  | image                               |
>  |
> 
>  | key_name                            | keypair-tenant1
>  |
> 
>  | name                                | tenant1-vm-1
>  |
> 
>  | progress                            | 0
>  |
> 
>  | project_id                          |
>  196a1151724d4d5d8fe6e24bfd4acf5f                            |
> 
>  | properties                          |
>  |
> 
>  | security_groups                     | name='default'
>  |
> 
>  |                                     | name='default'
>  |
> 
>  | status                              | ACTIVE
>  |
> 
>  | updated                             | 2019-07-12T15:32:01Z
>  |
> 
>  | user_id                             |
>  ffd19119f10c4ab1aa99a9608a66923a                            |
> 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-
>  2fc3de8274c3'                   |
> 
>  +-------------------------------------+-------------------------------------------------------------
> 
>   [sysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server list --a
> 
>  +--------------------------------------+--------------+--------+------------------------------------------------------------+-------+----------------------+
> 
>  | ID                                   | Name         | Status |
>  Networks                                                   | Image |
>  Flavor               |
> 
>  +--------------------------------------+--------------+--------+------------------------------------------------------------+-------+----------------------+
> 
>  | ab3048c4-b10f-49ad-a93d-16dc24c97b93 | tenant1-vm-1 | ACTIVE |
>  tenant1-mgmt-net=192.168.146.55; tenant1-net1=172.16.1.159 |       |
>  flavor-default-size2 |
> 
>  +--------------------------------------+--------------+--------+------------------------------------------------------------+-------+----------------------+
> 
> 
> 
>  ]$ nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne migration-list
>  nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-e 
>  ndpoint-type internalURL --os-region-name RegionOne migration-list
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  | Id | UUID                                 | Source Node | Dest Node
>  | Source Compute | Dest Compute | Dest Host       | Status    |
>  Instance UUID                        | Old Flavor | New Flavor |
>  Created At                 | Updated At                 | Type
>  |
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  | 31 | 3fc2d997-3c9c-44f6-8991-39ad3c87ed12 | -           | -
>  | compute-1      | -            | -               | error     |
>  ab3048c4-b10f-49ad-a93d-16dc24c97b93 | 46         | 46         |
>  2019-07-12T15:31:59.000000 | 2019-07-12T15:32:01.000000 | live-
>  migration |
> 
>  | 28 | 5e82192b-f11f-4829-8a61-6ee196ecbb32 | compute-0   | compute-1
>  | compute-0      | compute-1    | 192.168.205.236 | completed |
>  ab3048c4-b10f-49ad-a93d-16dc24c97b93 | None       | None       |
>  2019-07-12T15:24:34.000000 | 2019-07-12T15:30:34.000000 | evacuation
>  |
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ echo $?
> 
>  Severity
>  --------
>  Major
>  Steps to Reproduce
>  ------------------
>  1. Launch VM . VM was created in compute-0
>  n)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | Field                               | Value
>  |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | OS-DCF:diskConfig                   | MANUAL
>  |
> 
>  | OS-EXT-AZ:availability_zone         | nova
>  |
> 
>  | OS-EXT-SRV-ATTR:host                | compute-0
>  |
> 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-0
>  |
> 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016
>  |
> 
>  | OS-EXT-STS:power_state              | Running
>  |
> 
>  | OS-EXT-STS:task_state               | None
>  |
> 
>  | OS-EXT-STS:vm_state                 | active
>  |
> 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:23:56.000000
>  |
> 
>  | OS-SRV-USG:terminated_at            | None
>  |
> 
>  | accessIPv4                          |
>  |
> 
>  | accessIPv6                          |
>  |
> 
>  | addresses                           | tenant1-mgmt-
>  net=192.168.146.55; tenant1-net1=172.16.1.159  |
> 
>  | config_drive                        |
>  |
> 
>  | created                             | 2019-07-12T15:23:45Z
>  |
> 
>  | flavor                              | flavor-default-size2 (0c7d0962
>  -bf0c-44fd-a5b9-ff6b9d0a93b3) |
> 
>  | hostId                              |
>  da8f07be2656cda0f4faff2389b7ee95d00879957c5b4bbea77fe622    |
> 
>  | id                                  | ab3048c4-b10f-49ad-a93d-
>  16dc24c97b93                        |
> 
>  | image                               |
>  |
> 
>  | key_name                            | keypair-tenant1
>  |
> 
>  | name                                | tenant1-vm-1
>  |
> 
>  | progress                            | 0
>  |
> 
>  | project_id                          |
>  196a1151724d4d5d8fe6e24bfd4acf5f                            |
> 
>  | properties                          |
>  |
> 
>  | security_groups                     | name='default'
>  |
> 
>  |                                     | name='default'
>  |
> 
>  | status                              | ACTIVE
>  |
> 
>  | updated                             | 2019-07-12T15:23:56Z
>  |
> 
>  | user_id                             |
>  ffd19119f10c4ab1aa99a9608a66923a                            |
> 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-
>  2fc3de8274c3'                   |
> 
>  +-------------------------------------+-------------------------------------------------------------+
>  2. Reboot compute-0 sudo reboot –f
>  3. wait for compute  to come up.
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | id | hostname     | personality | administrative | operational |
>  availability |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | 1  | controller-0 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 2  | controller-1 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 3  | compute-1    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 4  | compute-2    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 5  | compute-0    | worker      | unlocked       | disabled    |
>  offline      |
> 
>  | 6  | storage-0    | storage     | unlocked       | enabled     |
>  available    |
> 
>  | 7  | storage-1    | storage     | unlocked       | enabled     |
>  available    |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | id | hostname     | personality | administrative | operational |
>  availability |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  | 1  | controller-0 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 2  | controller-1 | controller  | unlocked       | enabled     |
>  available    |
> 
>  | 3  | compute-1    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 4  | compute-2    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 5  | compute-0    | worker      | unlocked       | enabled     |
>  available    |
> 
>  | 6  | storage-0    | storage     | unlocked       | enabled     |
>  available    |
> 
>  | 7  | storage-1    | storage     | unlocked       | enabled     |
>  available    |
> 
>  +----+--------------+-------------+----------------+-------------+--------------+
> 
>  4.Verify VM evacuation VM running on compute-1
> 
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93 
>  +-------------------------------------+-------------------------------------------------------------+ 
>  | Field                               | Value                                                       | 
>  +-------------------------------------+-------------------------------------------------------------+ 
>  | OS-DCF:diskConfig                   | MANUAL                                                      | 
>  | OS-EXT-AZ:availability_zone         | nova                                                        | 
>  | OS-EXT-SRV-ATTR:host                | compute-1                                                   | 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1                                                   | 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016                                           | 
>  | OS-EXT-STS:power_state              | Running                                                     | 
>  | OS-EXT-STS:task_state               | None                                                        | 
>  | OS-EXT-STS:vm_state                 | active                                                      | 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:24:52.000000                                  | 
>  | OS-SRV-USG:terminated_at            | None                                                        | 
>  | accessIPv4                          |                                                             | 
>  | accessIPv6                          |                                                             | 
>  | addresses                           | tenant1-mgmt-net=192.168.146.55; tenant1-net1=172.16.1.159  | 
>  | config_drive                        |                                                             | 
>  | created                             | 2019-07-12T15:23:45Z                                        | 
>  | flavor                              | flavor-default-size2 (0c7d0962-bf0c-44fd-a5b9-ff6b9d0a93b3) | 
>  | hostId                              | a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c    | 
>  | id                                  | ab3048c4-b10f-49ad-a93d-16dc24c97b93                        | 
>  | image                               |                                                             | 
>  | key_name                            | keypair-tenant1                                             | 
>  | name                                | tenant1-vm-1                                                | 
>  | progress                            | 0                                                           | 
>  | project_id                          | 196a1151724d4d5d8fe6e24bfd4acf5f                            | 
>  | properties                          |                                                             | 
>  | security_groups                     | name='default'                                              | 
>  |                                     | name='default'                                              | 
>  | status                              | ACTIVE                                                      | 
>  | updated                             | 2019-07-12T15:30:33Z                                        | 
>  | user_id                             | ffd19119f10c4ab1aa99a9608a66923a                            | 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-2fc3de8274c3'                   | 
>  +-------------------------------------+-------------------------------------------------------------+ 
>  [sysadmin@controller-1 ~(keystone_admin)]$ echo $?
> 
> 
> 
>  5. Livemigrate VM after evacute
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne live-migration ab3048c4-b10f-49ad-a93d-16dc24c97b93 compute-0
>  nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-e 
>  ndpoint-type internalURL --os-region-name RegionOne live-migration ab3048c4-b10f-49ad-a93d-16dc24c97b93 compute-0
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ echo $?
>  echo $?
> 
>  0
> 
>  [sysadmin@controller-1 ~(keystone_admin)]$ openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | Field                               | Value
>  |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | OS-DCF:diskConfig                   | MANUAL
>  |
> 
>  | OS-EXT-AZ:availability_zone         | nova
>  |
> 
>  | OS-EXT-SRV-ATTR:host                | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016
>  |
> 
>  | OS-EXT-STS:power_state              | Running
>  |
> 
>  | OS-EXT-STS:task_state               | migrating
>  |
> 
>  | OS-EXT-STS:vm_state                 | active
>  |
> 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:24:52.000000
>  |
> 
>  | OS-SRV-USG:terminated_at            | None
>  |
> 
>  | accessIPv4                          |
>  |
> 
>  | accessIPv6                          |
>  |
> 
>  | addresses                           | tenant1-mgmt-
>  net=192.168.146.55; tenant1-net1=172.16.1.159  |
> 
>  | config_drive                        |
>  |
> 
>  | created                             | 2019-07-12T15:23:45Z
>  |
> 
>  | flavor                              | flavor-default-size2 (0c7d0962
>  -bf0c-44fd-a5b9-ff6b9d0a93b3) |
> 
>  | hostId                              |
>  a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c    |
> 
>  | id                                  | ab3048c4-b10f-49ad-a93d-
>  16dc24c97b93                        |
> 
>  | image                               |
>  |
> 
>  | key_name                            | keypair-tenant1
>  |
> 
>  | name                                | tenant1-vm-1
>  |
> 
>  | progress                            | 0
>  |
> 
>  | project_id                          |
>  196a1151724d4d5d8fe6e24bfd4acf5f                            |
> 
>  | properties                          |
>  |
> 
>  | security_groups                     | name='default'
>  |
> 
>  |                                     | name='default'
>  |
> 
>  | status                              | MIGRATING
>  |
> 
>  | updated                             | 2019-07-12T15:31:59Z
>  |
> 
>  | user_id                             |
>  ffd19119f10c4ab1aa99a9608a66923a                            |
> 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-
>  2fc3de8274c3'                   |
> 
>  	
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
>  openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default -- 
>  os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show ab3048c4-b10f-49ad-a93d-16dc24c97b93
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | Field                               | Value
>  |
> 
>  +-------------------------------------+-------------------------------------------------------------+
> 
>  | OS-DCF:diskConfig                   | MANUAL
>  |
> 
>  | OS-EXT-AZ:availability_zone         | nova
>  |
> 
>  | OS-EXT-SRV-ATTR:host                | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:hypervisor_hostname | compute-1
>  |
> 
>  | OS-EXT-SRV-ATTR:instance_name       | instance-00000016
>  |
> 
>  | OS-EXT-STS:power_state              | Running
>  |
> 
>  | OS-EXT-STS:task_state               | None
>  |
> 
>  | OS-EXT-STS:vm_state                 | active
>  |
> 
>  | OS-SRV-USG:launched_at              | 2019-07-12T15:24:52.000000
>  |
> 
>  | OS-SRV-USG:terminated_at            | None
>  |
> 
>  | accessIPv4                          |
>  |
> 
>  | accessIPv6                          |
>  |
> 
>  | addresses                           | tenant1-mgmt-
>  net=192.168.146.55; tenant1-net1=172.16.1.159  |
> 
>  | config_drive                        |
>  |
> 
>  | created                             | 2019-07-12T15:23:45Z
>  |
> 
>  | flavor                              | flavor-default-size2 (0c7d0962
>  -bf0c-44fd-a5b9-ff6b9d0a93b3) |
> 
>  | hostId                              |
>  a223412c30650d0bbf5df2bd441a5a4828cf49b1020d63efb12cf90c    |
> 
>  | id                                  | ab3048c4-b10f-49ad-a93d-
>  16dc24c97b93                        |
> 
>  | image                               |
>  |
> 
>  | key_name                            | keypair-tenant1
>  |
> 
>  | name                                | tenant1-vm-1
>  |
> 
>  | progress                            | 0
>  |
> 
>  | project_id                          |
>  196a1151724d4d5d8fe6e24bfd4acf5f                            |
> 
>  | properties                          |
>  |
> 
>  | security_groups                     | name='default'
>  |
> 
>  |                                     | name='default'
>  |
> 
>  | status                              | ACTIVE
>  |
> 
>  | updated                             | 2019-07-12T15:32:01Z
>  |
> 
>  | user_id                             |
>  ffd19119f10c4ab1aa99a9608a66923a                            |
> 
>  | volumes_attached                    | id='cedd17da-2480-4e04-999e-
>  2fc3de8274c3'                   |
> 
>  +-------------------------------------+-------------------------------------------------------------
> 
>  6.Migration list shows error to migrate from  compute-1
> 
> 
>  nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-endpoint-type internalURL --os-region-name RegionOne migration-list
>  nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default  --os-e 
>  ndpoint-type internalURL --os-region-name RegionOne migration-list
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  | Id | UUID                                 | Source Node | Dest Node
>  | Source Compute | Dest Compute | Dest Host       | Status    |
>  Instance UUID                        | Old Flavor | New Flavor |
>  Created At                 | Updated At                 | Type
>  |
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  | 31 | 3fc2d997-3c9c-44f6-8991-39ad3c87ed12 | -           | -
>  | compute-1      | -            | -               | error     |
>  ab3048c4-b10f-49ad-a93d-16dc24c97b93 | 46         | 46         |
>  2019-07-12T15:31:59.000000 | 2019-07-12T15:32:01.000000 | live-
>  migration |
> 
>  | 28 | 5e82192b-f11f-4829-8a61-6ee196ecbb32 | compute-0   | compute-1
>  | compute-0      | compute-1    | 192.168.205.236 | completed |
>  ab3048c4-b10f-49ad-a93d-16dc24c97b93 | None       | None       |
>  2019-07-12T15:24:34.000000 | 2019-07-12T15:30:34.000000 | evacuation
>  |
> 
>  +----+--------------------------------------+-------------+-----------+----------------+--------------+-----------------+-----------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
> 
>  System Configuration
>  --------------------
>  Regular system
> 
> 
>  Expected Behavior
>  ------------------
>  Able to migrate after evacuate. 
>  Actual Behavior
>  ----------------
>  Failure on migration after evacuate.
>  Reproducibility
>  ---------------
>  100% reproducible
>  System Configuration
>  --------------------
>  Regular system
>  Load 
>  ----
>  2019-07-11 01:30:00 +0000
>  Last Pass
>  ---------
>  Not sure in this lab.
> 
>  Timestamp/Logs
>  --------------
>  2019-07-12T15:31:59.000000
> 
>  Test Activity
>  -------------
>  Regression test
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/starlingx/+bug/1836402/+subscriptions

Revision history for this message

ya.wang (ya.wang) wrote on 2019-08-01:

#11

@yong, can you retest it and ensure the service is available before live-migrate?
If your test pass, it may means that `system host-list` may have some issues in get node's service status.

Revision history for this message

yong hu (yhu6) wrote on 2019-08-05:

#12

@Ya, "system host-list" is ONLY supposed to check the host availability, not really look into the containerized openstack services/pods. Before using OpenStack client to do OpenStack operations, the user need use "kubectl" to have a check.

Revision history for this message

ya.wang (ya.wang) wrote on 2019-08-06:

#13

@yong, it would better to use "openstack compute service list" to confirm at the same time.

Revision history for this message

yong hu (yhu6) wrote on 2019-08-06:

#14

the live-migration works well after evacuation on recent 0802 cengn build
see the attachment.

Revision history for this message

yong hu (yhu6) wrote on 2019-08-06:

#15

vm-migrate-live-okay-after-evacuation.png Edit (220.2 KiB, image/png)

one more thing: in order to do live migration, the VM has to be created with a bootable volume.

for example:

openstack server create --flavor myflavor --nic net-id=my_net --volume=38799564-5c06-4969-9dc4-a612024c5130 vm0

here 38799564-5c06-4969-9dc4-a612024c5130 is a bootable volume.

Changed in starlingx:
status:	Triaged → Fix Released

Revision history for this message

Anujeyan Manokeran (anujeyan) wrote on 2019-08-15:

#16

Verified in load 2019-08-14 05:30:00

tags:

removed: stx.retestneeded

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.