migration from qemu 2.5 to qemu 2.11 fails for pc-i440fx-wily machines
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| qemu (Ubuntu) |
Medium
|
Unassigned | ||
| Bionic |
Undecided
|
Unassigned | ||
| Cosmic |
Undecided
|
Unassigned | ||
| Disco |
Undecided
|
Unassigned | ||
| Eoan |
Medium
|
Unassigned |
Bug Description
[Impact]
* the machine type for wily (which we keep for migration from an early
xenials qemu) is broken >=qemu 2.11
* fix by correcting the definition of that type
[Test Case]
- Xenial / Bionic system
$ lxc launch ubuntu-daily:x x-wily --profile default --profile kvm
$ lxc launch ubuntu-daily:b b-wily --profile default --profile kvm
- set hostid to be different (as we have containers)
$ vim /etc/libvirt/
$ systemctl restart libvirtd
- exchange ssh keys
$ ssh-keygen
$ cat ~/.ssh/id_rsa.pub > ~/.ssh/
(host)
$ lxc file pull --recursive x-wily/root/.ssh .
$ lxc file push --recursive .ssh b-wily/root/
$ lxc exec b-wily -- chown -R root:root /root/.ssh
- use uvtool to create the same guest on both systems (FS layout of images)
$ time uvt-simplestrea
arch=amd64 label=daily release=eoan
$ uvt-kvm create --password ubuntu wilymigrate arch=amd64 release=eoan label=daily
- edit to set wily machine type (on source = Xenial)
$ virsh edit wilymigrate
- on target remove the former definition
$ virsh undefine wilymigrate
- Migrate
$ virsh migrate --unsafe --live wilymigrate qemu+ssh:
error: internal error: qemu unexpectedly closed the monitor: 2019-05-
system-x86_64: warning: Unknown firmware file in legacy mode: etc/msr_
2019-
2019-
With the fix on the target the issue is gone.
[Regression Potential]
The purpose to keep these old types is only to allow people to "migrate off" the older releases as long as they are still supported. Never the less sooner or later people are strongly encouraged [1]. Given your great analysis you know that already, but others might come by this bug.
It is important to be considered when deciding on if/what to change.
No one should ever start a "new" guest of a wily type on Bionic or later.
So I'm not too concerned about the delta we introduce to people having done that. The time this took to be found confirms that even migrations from Xenial with a Wily type are rare. We can not really differ between wily types coming from e.g. Xenial (or Trusty-Mitaka) hosts and wily type migrations coming from other Bionic systems (only a problem after a fix to this bug here and if they are on a different patch level).
So as mentioned the one real case for the wily type to still exist is to migrate off of older systems, and that use-case is broken. So I'm considering the thoughts above as a known, but less important than "the main use case" regression.
[Other Info]
* n/a
---
In qemu 2.11 pc-i440fx-wily machine type is defined the following way by the ubuntu patch:
101 +static void pc_wily_
102 +{
103 + pc_i440fx_
104 + pc_i440fx_
105 + m->desc = "Ubuntu 15.04 PC (i440FX + PIIX, 1996)",
106 + m->default_display = "std";
107 +}
108 +
109 +DEFINE_
110 + pc_wily_
In qemu 2.5, pc_compat_2_3 contained the following skip flags: https:/
in qemu 2.11 those skips moved to pc_i440fx_
https:/
https:/
https:/
https:/
It happened in commits:
https:/
https:/
https:/
but pc_wily_
LC_ALL=C PATH=/usr/
2019-05-14 11:53:57.583+0000: Domain id=22 is tainted: shell-scripts
2019-05-
2019-05-
2019-05-
2019-05-14 11:53:57.998+0000: shutting down, reason=failed
description: | updated |
description: | updated |
Christian Ehrhardt (paelzer) wrote : | #2 |
Thanks Vladyslav for the bug report and the great analysis done already.
/self rant break
Oh how I hate machine types, it is sad that we still need to add them at all for some special cases and encapsulating changes. But among all of them being sort of "ok" the wily type clearly is the top on my personal hate list.
/end rant
---
The purpose to keep these old types is only to allow people to "migrate off" the older releases as long as they are still supported. Never the less sooner or later people are strongly encouraged [1]. Given your great analysis you know that already, but others might come by this bug.
It is important to be considered when deciding on if/what to change.
No one should ever start a "new" guest of a wily type on Bionic or later.
So I'm not too concerned about the delta we introduce to people having done that. The time this took to be found confirms that even migrations from Xenial with a Wily type are rare. We can not really differ between wily types coming from e.g. Xenial (or Trusty-Mitaka) hosts and wily type migrations coming from other Bionic systems (only a problem after a fix to this bug here and if they are on a different patch level).
So as mentioned the one real case for the wily type to still exist is to migrate off of older systems, and that use-case is broken. So I'm considering the thoughts above as a known, but less important than "the main use case" regression.
---
I'd want to avoid touching pc_compat_2_3 itself (would affect other types), but just switching from wily from pc_i440fx_
Maybe we should define a HW_COMPAT_WILY being the fuse of HW_COMPAT_2_4 / HW_COMPAT_2_3 that actually matches the past.
Let me prep a change with that and we can give it a try
Changed in qemu (Ubuntu): | |
status: | Confirmed → Triaged |
importance: | Undecided → Medium |
Christian Ehrhardt (paelzer) wrote : | #3 |
Ok, the first build (yet untested) is complete in PPA
https:/
Let me setup a testbed to check on that ...
Christian Ehrhardt (paelzer) wrote : | #4 |
TODO add one wily migration to the regression test suite (could work via cross + setmt?)
Collecting steps for the later following SRU teamplate.
- Xenial / Bionic system
$ lxc launch ubuntu-daily:x x-wily --profile default --profile kvm
$ lxc launch ubuntu-daily:b b-wily --profile default --profile kvm
- set hostid to be different (as we have containers)
$ vim /etc/libvirt/
$ systemctl restart libvirtd
- exchange ssh keys
$ ssh-keygen
$ cat ~/.ssh/id_rsa.pub > ~/.ssh/
(host)
$ lxc file pull --recursive x-wily/root/.ssh .
$ lxc file push --recursive .ssh b-wily/root/
$ lxc exec b-wily -- chown -R root:root /root/.ssh
- use uvtool to create the same guest on both systems (FS layout of images)
$ time uvt-simplestrea
arch=amd64 label=daily release=eoan
$ uvt-kvm create --password ubuntu wilymigrate arch=amd64 release=eoan label=daily
- edit to set wily machine type (on source = Xenial)
$ virsh edit wilymigrate
- on target remove the former definition
$ virsh undefine wilymigrate
- Migrate
$ virsh migrate --unsafe --live wilymigrate qemu+ssh:
error: internal error: qemu unexpectedly closed the monitor: 2019-05-
system-x86_64: warning: Unknown firmware file in legacy mode: etc/msr_
2019-
2019-
(Useful to do extra checks e.g. if the guest is alive)
Ok, the above reproduced the bug with qemu 1:2.11+
Trying 1:2.11+
- Migrate
root@x-wily:~# virsh migrate --unsafe --live wilymigrate qemu+ssh:
(worked)
And back:
root@b-wily:~# virsh migrate --unsafe --live wilymigrate qemu+ssh:
(worked)
Check if it is alive and still up (trivial)
root@x-wily:~# uvt-kvm ssh --insecure wilymigrate "uptime"
13:07:44 up 9 min, 0 users, load average: 0.00, 0.03, 0.04
Ok, the PPA would work in the direct test case for this reported issue.
@Vladyslav - would you mind testing the PPA for you as well?
Once you report it good as well I'd shove it into some bigger regression tests that I have.
TODO: add an explicit wily migration to latest LTS and to latest -dev release to the tests.
Once all of these seem good we can fix it in Eoan (current development release) and then open up SRUs for Bionic/Cosmic/Disco
Changed in qemu (Ubuntu): | |
status: | Triaged → Incomplete |
Christian Ehrhardt (paelzer) wrote : | #5 |
Incomplete waiting for user confirmation (of the PPA)
Vladyslav Drok (vdrok) wrote : | #6 |
Thanks you for the fix Christian. We have a version of ubuntu types definition patch different from the one in ppa, I'll try to sync it first and check.
Christian Ehrhardt (paelzer) wrote : | #7 |
Ok, let me know what you find in your case
If you don't like dissecting debdiffs from the PPA, here the links to git:
https:/
https:/
Vladyslav Drok (vdrok) wrote : | #8 |
The changes you propose have worked for me, but I also requested customer to validate this, just in case.
Vladyslav Drok (vdrok) wrote : | #9 |
Customer also confirmed the fix worked for them, thank you!
Changed in qemu (Ubuntu): | |
status: | Incomplete → Triaged |
Changed in qemu (Ubuntu Disco): | |
status: | New → Triaged |
Changed in qemu (Ubuntu Cosmic): | |
status: | New → Triaged |
Changed in qemu (Ubuntu Bionic): | |
status: | New → Triaged |
Christian Ehrhardt (paelzer) wrote : | #10 |
Tested migrations between X <-> B/C/D/E all working now.
Tested migration of a wily type guest between B<->C/D as an extra check.
With the fix all is good now.
Paths like X->C-D->B->X worked.
The only degradation is between updated and unupdated systems, but that was expected and acceptable in comparison to the fix.
Christian Ehrhardt (paelzer) wrote : | #11 |
Uploaded to Eoan and (as it will need there a while) also already to B/C/D -unapproved.
Launchpad Janitor (janitor) wrote : | #12 |
This bug was fixed in the package qemu - 1:3.1+dfsg-2ubuntu5
---------------
qemu (1:3.1+
* d/p/ubuntu/
broken since 2.11 due to 2.3/2.4 version mismatch in its definition to
fix migrations from old machines (LP: #1829868).
* d/p/ubuntu/
toleration for future machines (LP: #1830704
-- Christian Ehrhardt <email address hidden> Tue, 28 May 2019 11:30:42 +0200
Changed in qemu (Ubuntu Eoan): | |
status: | Triaged → Fix Released |
Łukasz Zemczak (sil2100) wrote : | #13 |
Can we get the SRU information on this bug? Would be great to get a test case and regression potential analysis here.
Christian Ehrhardt (paelzer) wrote : | #14 |
Added the SRU template, I had all in the comments and forgot I haven't copied it to the description yet - sorry.
Thanks for the ping sil2100
description: | updated |
Hello Vladyslav, or anyone else affected,
Accepted qemu into disco-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
Changed in qemu (Ubuntu Disco): | |
status: | Triaged → Fix Committed |
tags: | added: verification-needed verification-needed-disco |
Changed in qemu (Ubuntu Cosmic): | |
status: | Triaged → Fix Committed |
tags: | added: verification-needed-cosmic |
Timo Aaltonen (tjaalton) wrote : | #16 |
Hello Vladyslav, or anyone else affected,
Accepted qemu into cosmic-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
Changed in qemu (Ubuntu Bionic): | |
status: | Triaged → Fix Committed |
tags: | added: verification-needed-bionic |
Timo Aaltonen (tjaalton) wrote : | #17 |
Hello Vladyslav, or anyone else affected,
Accepted qemu into bionic-proposed. The package will build now and be available at https:/
Please help us by testing this new package. See https:/
If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-
Further information regarding the verification process can be found at https:/
N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.
Christian Ehrhardt (paelzer) wrote : | #18 |
Trying Xenial -> test-subject and back.
root@x-wily:~# virsh dumpxml wilymigrate | grep '<type'
<type arch='x86_64' machine=
Cosmic:
root@x-wily:~# virsh migrate --unsafe --live wilymigrate qemu+ssh:
root@c-wily:~# virsh migrate --unsafe --live wilymigrate qemu+ssh:
(worked without issues, checking the guest on the target via login)
Bionic:
root@x-wily:~# virsh migrate --unsafe --live wilymigrate qemu+ssh:
root@b-wily:~# virsh migrate --unsafe --live wilymigrate qemu+ssh:
(worked without issues, checking the guest on the target via login)
Disco:
root@x-wily:~# virsh migrate --unsafe --live wilymigrate qemu+ssh:
root@d-wily:~# virsh migrate --unsafe --live wilymigrate qemu+ssh:
(worked without issues, checking the guest on the target via login)
Poor little guest was punted around all the time, go to sleep ...
$ virsh shutdown wilymigrate
Setting verified.
tags: |
added: verification-done verification-done-bionic verification-done-cosmic verification-done-disco removed: verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco |
Launchpad Janitor (janitor) wrote : | #19 |
This bug was fixed in the package qemu - 1:2.11+
---------------
qemu (1:2.11+
* d/p/ubuntu/
broken since 2.11 due to 2.3/2.4 version mismatch in its definition to
fix migrations from old machines (LP: #1829868).
* d/p/ubuntu/
toleration for future machines (LP: #1830704
-- Christian Ehrhardt <email address hidden> Wed, 22 May 2019 13:14:15 +0200
Changed in qemu (Ubuntu Bionic): | |
status: | Fix Committed → Fix Released |
The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.
Launchpad Janitor (janitor) wrote : | #21 |
This bug was fixed in the package qemu - 1:2.12+
---------------
qemu (1:2.12+
* d/p/ubuntu/
broken since 2.11 due to 2.3/2.4 version mismatch in its definition to
fix migrations from old machines (LP: #1829868).
* d/p/ubuntu/
toleration for future machines (LP: #1830704
-- Christian Ehrhardt <email address hidden> Tue, 28 May 2019 10:49:09 +0200
Changed in qemu (Ubuntu Cosmic): | |
status: | Fix Committed → Fix Released |
Launchpad Janitor (janitor) wrote : | #22 |
This bug was fixed in the package qemu - 1:3.1+dfsg-
---------------
qemu (1:3.1+
* d/p/ubuntu/
broken since 2.11 due to 2.3/2.4 version mismatch in its definition to
fix migrations from old machines (LP: #1829868).
* d/p/ubuntu/
toleration for future machines (LP: #1830704
* d/control-in, d/control: add versioned dependencies to libseccomp 2.4 as
any rebuild against 2.4 as it is in proposed right now will otherwise
crash (LP: #1830859).
-- Christian Ehrhardt <email address hidden> Tue, 28 May 2019 10:52:47 +0200
Changed in qemu (Ubuntu Disco): | |
status: | Fix Committed → Fix Released |
Status changed to 'Confirmed' because the bug affects multiple users.