ISST-LTE:Ubuntu16.04.2KVM: Migration of guest fails between Ubuntu16.04.2 KVM hosts with error: load of migration failed: Invalid argument

Bug #1654150 reported by bugproxy
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Fix Released
Medium
Unassigned

Bug Description

== Comment: #0 - Manjunatha H R <email address hidden> - 2016-12-26 05:45:38 ==
Migrating Ubuntu16.04.2 guest between two Ubuntu16.04.2 KVM hosts fails :

Migration error:
--------------------
root@lotkvm:~# virsh migrate --live --domain lotg1 qemu+ssh://10.33.32.222/system --verbose --undefinesource --persistent --timeout 30 --copy-storage-all --unsafe 2>&1
error: monitor socket did not show up: No such file or directory

OR sometimes it fails with following error:
---------------------------
Linux lotkvm 4.8.0-32-generic #34~16.04.1-Ubuntu SMP Tue Dec 13 17:01:57 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux
root@lotkvm:~# virsh migrate --live --domain lotg1 qemu+ssh://10.33.32.222/system --verbose --undefinesource --persistent --copy-storage-all --timeout 60 --unsafe 2>&1

Migration: [100 %]error: internal error: early end of file from monitor, possible problem: 2016-12-26T11:09:42.030260Z qemu-system-ppc64: Unknown savevm section or instance 'pci@800000020000000:06.0/ohci' 0
2016-12-26T11:09:42.031574Z qemu-system-ppc64: load of migration failed: Invalid argument

Qemu logs show following traces:
---------------------
2016-12-26T09:35:35.709182Z qemu-system-ppc64: Unknown savevm section or instance 'pci@800000020000000:06.0/ohci' 0
2016-12-26T09:35:35.710089Z qemu-system-ppc64: load of migration failed: Invalid argument

Host build:
--------------
uname -a
Linux lotkvm 4.8.0-32-generic #34~16.04.1-Ubuntu SMP Tue Dec 13 17:01:57 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Guest build:
----------------
# uname -a
Linux lotg1 4.8.0-32-generic #34~16.04.1-Ubuntu SMP Tue Dec 13 17:01:57 UTC 2016 ppc64le ppc64le ppc64le GNU/Linux

Issue is seen in shared storage and non-shared storages scenaris (i.e with or wothout --copy-storage-all option).

== Comment: #8 - Shivaprasad G. Bhat <email address hidden> - 2016-12-27 03:28:16 ==
This is a known issue and the patch proposed to fix the issue was rejected upstream and later decided to live with a workaround as discussed in https://www.redhat.com/archives/libvir-list/2016-July/msg01055.html and https://bugzilla.redhat.com/show_bug.cgi?id=1357468

Let me know if the workarounds suggested works or not, if not may be we need to pull some patches.

== Comment: #9 - Manjunatha H R <email address hidden> - 2016-12-27 05:52:01 ==
(In reply to comment #8)
> This is a known issue and the patch proposed to fix the issue was rejected
> upstream and later decided to live with a workaround as discussed in
> https://www.redhat.com/archives/libvir-list/2016-July/msg01055.html and
> https://bugzilla.redhat.com/show_bug.cgi?id=1357468
>
> Let me know if the workarounds suggested works or not, if not may be we need
> to pull some patches.

Thanks Shiva !!

With around of adding : model='pci-ohci' to usb controller helps to resolve migration error.

Before applying workaround:
----------------
root@lotkvm:~# virsh dumpxml lotg9|grep usb
    <controller type='usb' index='0'>
      <alias name='usb'/>
root@lotkvm:~#

root@lotkvm:~# virsh migrate --live --domain lotg9 qemu+ssh://10.33.32.222/system --verbose --undefinesource --persistent --timeout 60 --unsafe 2>&1
Migration: [100 %]error: operation failed: job: unexpectedly failed

After applying work around:
-------------------------
root@lotkvm:~# virsh dumpxml lotg9|grep usb
    <controller type='usb' index='0' model='pci-ohci'>
      <alias name='usb'/>
root@lotkvm:~#

root@lotkvm:~# virsh migrate --live --domain lotg9 qemu+ssh://10.33.32.222/system --verbose --undefinesource --persistent --timeout 60 --unsafe 2>&1
Migration: [100 %]
root@lotkvm:~#

Does this need to be documented in Ubuntu release notes ?

Thanks,
Manju

== Comment: #11 - Shivaprasad G. Bhat <email address hidden> - 2017-01-02 02:31:00 ==
Hi Manju, Chandan,

Yes, I think this should be documented as its likely to happen and because this is seen only on PPC without a documentation it is hard to figure out why its happening.

Thanks,
Shiva

Revision history for this message
bugproxy (bugproxy) wrote : Guest XML

Default Comment by Bridge

tags: added: architecture-ppc64le bugnameltc-150116 severity-critical targetmilestone-inin---
Revision history for this message
bugproxy (bugproxy) wrote : Teget host qemu log for guest

Default Comment by Bridge

Revision history for this message
bugproxy (bugproxy) wrote : Source and target hosts /var/log/syslog

Default Comment by Bridge

Revision history for this message
bugproxy (bugproxy) wrote : guest sosreport

Default Comment by Bridge

Revision history for this message
bugproxy (bugproxy) wrote : Source host sosreport

Default Comment by Bridge

Changed in ubuntu:
assignee: nobody → Taco Screen team (taco-screen-team)
affects: ubuntu → libvirt (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote :

Default Comment by Bridge

Revision history for this message
bugproxy (bugproxy) wrote : Target host sosreport

Default Comment by Bridge

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Out of the rather long mirror:
- upstream rejected patch
- workaround recommended and working
- seen only on PPC
- request updated release notes as documentation

Could you confirm that this is a valid TL;DR of the case?

Changed in libvirt (Ubuntu):
status: New → Incomplete
importance: Undecided → Medium
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2017-01-13 12:16 EDT-------
(In reply to comment #16)
> Out of the rather long mirror:
> - upstream rejected patch
> - workaround recommended and working
> - seen only on PPC
> - request updated release notes as documentation
>
> Could you confirm that this is a valid TL;DR of the case?

That sounds like an accurate summary to me.

Joshua Powers (powersj)
Changed in libvirt (Ubuntu):
status: Incomplete → Triaged
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-02-01 13:39 EDT-------
Hello.
For now I added a recommendation to our wiki page:
https://wiki.ubuntu.com/ppc64el/Recommendations#Migration_between_kvm_hosts

Please take a look and see if it looks accurate.

Regards.

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-02-02 03:29 EDT-------
(In reply to comment #18)
> Hello.
> For now I added a recommendation to our wiki page:
> https://wiki.ubuntu.com/ppc64el/Recommendations#Migration_between_kvm_hosts
>
> Please take a look and see if it looks accurate.
>
> Regards.

Documentation looks accurate. Thank you !!

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Great ,
do we need any other Doc/Note to carry this or can/should we consider this fixed by adding it to the Wiki you linked?

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2017-02-06 07:15 EDT-------
(In reply to comment #21)
> Great ,
> do we need any other Doc/Note to carry this or can/should we consider this
> fixed by adding it to the Wiki you linked?

We can consider this one fixed for now.

Revision history for this message
Manoj Iyer (manjo) wrote :

Based on comment #13, marking this as fix released.

Changed in libvirt (Ubuntu):
assignee: Taco Screen team (taco-screen-team) → nobody
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.