VM live migration in error status

Bug #1858659 reported by Peng Peng on 2020-01-07
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
High
zhipeng liu

Bug Description

Brief Description
-----------------
Creating a volume from tis-centos-guest image, Boot up VM, then live migrate the VM. The VM did not change host and the status is "error"

Severity
--------
Major

Steps to Reproduce
------------------
as description

TC-name: nova/test_migrate_vms.py::test_migrate_vm[tis-centos-guest-live-None]

Expected Behavior
------------------

Actual Behavior
----------------

Reproducibility
---------------
Unknown - first time this is seen in sanity, will monitor

System Configuration
--------------------
Two node system

Lab-name: WCP_76-77

Branch/Pull Time/Commit
-----------------------
20200107T000000Z

Last Pass
---------
 20191231T000000Z

Timestamp/Logs
--------------
[2020-01-07 10:57:23,180] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show cdf5b01e-230b-4323-bc1a-7dcf97a127e6'
[2020-01-07 10:57:27,995] 433 DEBUG MainThread ssh.expect :: Output:
+-------------------------------------+------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | controller-0 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-0 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000014 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2020-01-07T10:56:45.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | tenant1-mgmt-net=192.168.141.57; tenant1-net0=172.16.0.246 |
| config_drive | |
| created | 2020-01-07T10:55:53Z |
| flavor | live-mig (5336e187-cda2-4e10-a515-11a1e73b46af) |
| hostId | 7e0075cb24eb8851edcbd575571433c8a51ddc81236bc2c0f6ba8193 |
| id | cdf5b01e-230b-4323-bc1a-7dcf97a127e6 |
| image | |
| key_name | keypair-tenant1 |
| name | tenant1-tis-centos-guest-8 |
| progress | 0 |
| project_id | 3d08e85b9a3d409e91a6e831c17aaeaa |
| properties | |
| security_groups | name='default' |
| | name='default' |
| status | ACTIVE |
| updated | 2020-01-07T10:56:46Z |
| user_id | 63f7915fc2fc40a4ad2e10c7f6a466f6 |
| volumes_attached | id='3faff924-cfb0-434d-919e-88e69d2285cd' |
+-------------------------------------+------------------------------------------------------------+
controller-1:~$

[2020-01-07 10:57:28,100] 311 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne live-migration cdf5b01e-230b-4323-bc1a-7dcf97a127e6'

[2020-01-07 10:58:35,229] 311 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-identity-api-version 3 --os-interface internal --os-region-name RegionOne server show cdf5b01e-230b-4323-bc1a-7dcf97a127e6'
[2020-01-07 10:58:39,717] 433 DEBUG MainThread ssh.expect :: Output:
+-------------------------------------+------------------------------------------------------------+
| Field | Value |
+-------------------------------------+------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | controller-0 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | controller-0 |
| OS-EXT-SRV-ATTR:instance_name | instance-00000014 |
| OS-EXT-STS:power_state | Running |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2020-01-07T10:56:45.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | tenant1-mgmt-net=192.168.141.57; tenant1-net0=172.16.0.246 |
| config_drive | |
| created | 2020-01-07T10:55:53Z |
| flavor | live-mig (5336e187-cda2-4e10-a515-11a1e73b46af) |
| hostId | 7e0075cb24eb8851edcbd575571433c8a51ddc81236bc2c0f6ba8193 |
| id | cdf5b01e-230b-4323-bc1a-7dcf97a127e6 |
| image | |
| key_name | keypair-tenant1 |
| name | tenant1-tis-centos-guest-8 |
| progress | 0 |
| project_id | 3d08e85b9a3d409e91a6e831c17aaeaa |
| properties | |
| security_groups | name='default' |
| | name='default' |
| status | ACTIVE |
| updated | 2020-01-07T10:57:48Z |
| user_id | 63f7915fc2fc40a4ad2e10c7f6a466f6 |
| volumes_attached | id='3faff924-cfb0-434d-919e-88e69d2285cd' |
+-------------------------------------+------------------------------------------------------------+

[2020-01-07 10:58:48,790] 311 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne migration-list'
[2020-01-07 10:58:53,410] 433 DEBUG MainThread ssh.expect :: Output:
+----+--------------------------------------+-------------+-----------+----------------+--------------+-----------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
| Id | UUID | Source Node | Dest Node | Source Compute | Dest Compute | Dest Host | Status | Instance UUID | Old Flavor | New Flavor | Created At | Updated At | Type |
+----+--------------------------------------+-------------+-----------+----------------+--------------+-----------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
| 6 | 323cca70-81ef-48d2-af04-cb42cc484333 | - | - | controller-0 | - | - | error | cdf5b01e-230b-4323-bc1a-7dcf97a127e6 | 30 | 30 | 2020-01-07T10:57:36.000000 | 2020-01-07T10:57:49.000000 | live-migration |
+----+--------------------------------------+-------------+-----------+----------------+--------------+-----------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
controller-1:~$

Test Activity
-------------
Sanity

Ghada Khalil (gkhalil) wrote :

Assigning to the distro.openstack PL for review/release recommendation

tags: added: stx.distro.openstack
Changed in starlingx:
assignee: nobody → yong hu (yhu6)
zhipeng liu (zhipengs) on 2020-01-09
Changed in starlingx:
assignee: yong hu (yhu6) → zhipeng liu (zhipengs)
zhipeng liu (zhipengs) wrote :

Hi peng,

I could not find openstack related log in your attached log.
ALL_NODES_20200107.145641 in folder /var/log/containers
Could you double check and make sure nova log uploaded, thanks!

Zhipeng

Changed in starlingx:
status: New → Incomplete
Peng Peng (ppeng) wrote :

Issue was reproduced on
Lab: WCP_76_77
Load: 20200111T023000Z

coolect log @ ALL_NODES_20200114.154021.tar
https://files.starlingx.kube.cengn.ca/launchpad/1858659

[2020-01-14 10:50:42,892] 314 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne live-migration 2070e355-a67b-4b77-83d5-1ff3c4812d7e'

[2020-01-14 10:52:04,335] 314 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://keystone.openstack.svc.cluster.local/v3 --os-user-domain-name Default --os-project-domain-name Default --os-endpoint-type internalURL --os-region-name RegionOne migration-list'
[2020-01-14 10:52:07,758] 436 DEBUG MainThread ssh.expect :: Output:
+----+--------------------------------------+-------------+-----------+----------------+--------------+-----------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
| Id | UUID | Source Node | Dest Node | Source Compute | Dest Compute | Dest Host | Status | Instance UUID | Old Flavor | New Flavor | Created At | Updated At | Type |
+----+--------------------------------------+-------------+-----------+----------------+--------------+-----------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
| 6 | 37599a29-5cc3-450b-8128-c5a01769f27c | - | - | controller-0 | - | - | error | 2070e355-a67b-4b77-83d5-1ff3c4812d7e | 30 | 30 | 2020-01-14T10:50:48.000000 | 2020-01-14T10:51:00.000000 | live-migration |
+----+--------------------------------------+-------------+-----------+----------------+--------------+-----------+--------+--------------------------------------+------------+------------+----------------------------+----------------------------+----------------+
controller-1:~$

Changed in starlingx:
status: Incomplete → Confirmed
Yang Liu (yliu12) on 2020-01-16
tags: added: stx.retestneeded
Ghada Khalil (gkhalil) wrote :

Marking as stx.4.0 for now given the issue was seen more than once. PL can update the release recommendation if there is new info.

tags: added: stx.4.0
Changed in starlingx:
importance: Undecided → High
Peng Peng (ppeng) wrote :

Issue was reproduced on
Lab: WCP_76_77
Load: 20200127T000002Z

Log @
https://files.starlingx.kube.cengn.ca/launchpad/1858659

zhipeng liu (zhipengs) wrote :

Hi pengpeng,

From second log, no clue found, and also the log time of nova module is not covering the issue time.

I could not download your latest log recently.
I tried many times and it always network error during downloading.
Could you help check it?

Thanks!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers