Live migration fails intermittently for boot-from-volume VM with volume stuck at attaching state even after vm deletion
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
Medium
|
Elena Taivan |
Bug Description
Brief Description
-----------------
Several live-migration tests failed silently (vm did not move to other host), these vms are booted from cinder volume, and volumes were observed to stuck at attaching status even after vm deletion.
Severity
--------
Major
Steps to Reproduce
------------------
Create a dedicated vm with 2vcpu, 2G disk, 1G ram, and boot from cinder volume using tis-centos-guest
Live migrate the vm
Expected Behavior
------------------
Live migration passes
Actual Behavior
----------------
Live migration fails intermittently with boot-from-volume vm
Reproducibility
---------------
Intermittent - there are at least 15 other live-migration with cinder volume were successful
System Configuration
-------
Duplex
Last Pass
---------
Same lab/load
Timestamp/Logs
--------------
[2020-06-13 07:31:41,425] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2020-06-13 07:31:43,547] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2020-06-13 07:34:51,875] 314 DEBUG MainThread ssh.send :: Send 'nova --os-username 'tenant1' --os-password 'Li69nux*' --os-project-name tenant1 --os-auth-url http://
[2020-06-13 07:39:00,575] 314 DEBUG MainThread ssh.send :: Send 'nova --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2020-06-13 07:41:17,587] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[sysadmin@
[2020-06-13 07:41:19,725] 314 DEBUG MainThread ssh.send :: Send 'openstack --os-username 'admin' --os-password 'Li69nux*' --os-project-name admin --os-auth-url http://
[2020-06-13 07:41:21,650] 436 DEBUG MainThread ssh.expect :: Output:
-------
ID Name Status Size Type Bootable Attached to Properties
-------
132f2598-
-------
[sysadmin@
Test Activity
-------------
Regression Testing
Changed in starlingx: | |
assignee: | nobody → Elena Taivan (etaivan) |
tags: | added: stx.5.0 stx.config stx.distro.openstack |
Changed in starlingx: | |
importance: | Undecided → Medium |
Reviewed: https:/ /review. opendev. org/763044 /opendev. org/starlingx/ config/ commit/ a37d5c2da6acd9c ee1bf499eface0f f891eea41f
Committed: https:/
Submitter: Zuul
Branch: master
commit a37d5c2da6acd9c ee1bf499eface0f f891eea41f
Author: Elena Taivan <email address hidden>
Date: Tue Nov 17 16:53:56 2020 +0000
Fix live-migration for boot-from-volume VM
The reason why live-migration is failing on AIO-DX
is because cinder-volume is hanging while running
'ceph mon dump' command.
'ceph mon dump' command reads ceph.conf file to get the
addresses of all the monitors. It leads to delays
if those addresses don't point to valid monitors.
On AIO-DX setup, ceph.conf had 3 monitors and this
caused delays of up to 1 minute when 'ceph mon dump'
was executed inside cinder pod.
To get the correct response time for 'ceph mon dump'
in the cinder pod, a single monitor pointing to the floating
IP should be used for AIO-DX.
Closes-bug: 1904594 58f7566061b65bf 11917fe904a
Change-Id: I70ecd5c4683833
Signed-off-by: Elena Taivan <email address hidden>