error migrating blocked on virtio-net-pci.rom

[Christian Ehrhardt ]

From Zesty to Artful

# virsh migrate --live kvmguest-trusty-normal qemu+ssh://10.22.69.90/system
error: internal error: qemu unexpectedly closed the monitor: 2017-08-28T13:03:01.352604Z qemu-system-x86_64: -chardev pty,id=charserial0: char device redirected to /dev/pts/0 (label charserial0)
2017-08-28T13:03:01.390972Z qemu-system-x86_64: warning: Unknown firmware file in legacy mode: etc/msr_feature_control
2017-08-28T13:03:01.435414Z qemu-system-x86_64: Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x40000 in != 0x80000: Invalid argument
2017-08-28T13:03:01.435429Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'
2017-08-28T13:03:01.435551Z qemu-system-x86_64: load of migration failed: Invalid argument

Worked before merging a newer ipxe, so the current assumption is that the rom changed and we need some quirk to handle the forward migration through the releases.

Solution might be in qemu/ipxe/libvirt therefore three bug tasks.

[Christian Ehrhardt  (paelzer)]

qemu-ipxe:
Z: 1.0.0+git-20150424.a25a16d-1ubuntu2
A: 1.0.0+git-20161027.b991c67-1ubuntu1

Check diff in detail:
- List of files did not change
- content and md5 obviously changed due to the update

=> Downgrading the ipxe package on artful to the level it had before resolves the issue.

We can not "never" pick up ipxe fixes, but if the consequence that we can not migrate forward anymore we need a better quirk (not being able to migrate back is ok, but forward should work).

[Christian Ehrhardt  (paelzer)]

The dev is as expected the virtio network card
    <interface type='network'>
      <mac address='52:54:00:cc:a3:b7'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

There is no explicit rom specified on the guest commandlines, so they pick their defaults.
Trying to list what they load:
 $ strace -e open /usr/bin/qemu-system-x86_64 -M pc-i440fx-trusty -device virtio-net -S -nographic -nodefaults 2>&1 | grep virtio
Shows /usr/lib/ipxe/qemu/efi-virtio.rom in both cases.

And this is actually matching the error sizes 0x4000 = 256k vs 0x8000 = 512k (next best power of 2).

I remember vague to have seen a similar issue long ago, trusty maybe even precise.
After a while I found it - a revival of one of my most hated issues bug 1291321.

Almost tempted to downgrade ipxe in artful to do this properly and with some time into 18.04 - but for now take some time to debug further.

[Christian Ehrhardt  (paelzer)]

It is "just" the size that breaks it, as soon as I append a bunch of zero's to the source of the migration it will allocate 0x8000 as well and work to migrate then.
 $ truncate --size=272384 /usr/lib/ipxe/qemu/efi-virtio.rom

Unfortunately this isn't much of a "solution" as:
- it needs to be (re-)started on the source to pick up the new size
- it would need to be back SRU'ed to about everywhere

I wonder if there is a better way.
Know about the change and adapt the Ubuntu machine types so that it increases the size on incoming migration maybe?

This was tried long ago in other bugs but failed, yet maybe today the SW stack is ready to do so.
I think I have to reach out to qmeu-devel to clarify if there is a way these days.

References to same/similar issues:
- https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1536331
- https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg01881.html
- https://bugzilla.redhat.com/show_bug.cgi?id=1293566
- https://bugzilla.redhat.com/show_bug.cgi?id=1090093
- https://forge.univention.org/bugzilla/show_bug.cgi?id=38877

[Christian Ehrhardt  (paelzer)]

As a follow on for ipxe we certainly want to implement a size check post build to catch changes out of a power-of-two bucket in advance next time.
Might even be worth to discuss on ipxe upstream.

Furthermore I checked that Pike (for Xenial as in sudo add-apt-repository cloud-archive:pike-proposed) does not include the ipxe from artful.
It could but it currently does not. Depending on the way to solve this we might only upgrade src:ipxe cross such rom changes in LTS->LTS.

I reached out to qemu-devel [1] for the best mechanism to catch this these days (in case there is a better one now).

[1]: http://lists.nongnu.org/archive/html/qemu-devel/2017-08/msg05365.html

[Christian Ehrhardt  (paelzer)]

I'm currently working on various options from turning back the time via an AA or some magic (unlikely :-) ), or patching back to the old version (plus a few fixes that are needed for FTBFS) but also trying to shrink size on the new versions if that is reasonable.

It is already built with -Os so not much the compiler can help automatically, but OTOH at least reverting gets us inside the old range - so it is not that toolchain changes were the cause for the size increase.

   Yakkety Artful Artful-revert
    246784 272896 245760 ./usr/lib/ipxe/qemu/efi-e1000.rom
    245760 270848 244224 ./usr/lib/ipxe/qemu/efi-eepro100.rom
    243712 268800 242688 ./usr/lib/ipxe/qemu/efi-ne2k_pci.rom
    244224 269312 243200 ./usr/lib/ipxe/qemu/efi-pcnet.rom
    247808 272896 246272 ./usr/lib/ipxe/qemu/efi-rtl8139.rom
    242176 272384 241152 ./usr/lib/ipxe/qemu/efi-virtio.rom
     81920 86528 81408 ./usr/lib/ipxe/qemu/pxe-e1000.rom
     81920 86528 81408 ./usr/lib/ipxe/qemu/pxe-eepro100.rom
     81408 86016 80896 ./usr/lib/ipxe/qemu/pxe-ne2k_isa.rom
     81920 86016 81408 ./usr/lib/ipxe/qemu/pxe-ne2k_pci.rom
     81920 86016 81408 ./usr/lib/ipxe/qemu/pxe-pcnet.rom
     82944 87040 82432 ./usr/lib/ipxe/qemu/pxe-rtl8139.rom
     81408 87040 80896 ./usr/lib/ipxe/qemu/pxe-virtio.rom

All efi roms grew over 256k.

[Christian Ehrhardt  (paelzer)]

All base roms grow ~4-5k, the EFI addon grew 20-24k.
I discussed the "might need to go backward plan" (at least for now), and there are examples of this like:
https://lists.ubuntu.com/archives/artful-changes/2017-August/007901.html
https://lists.ubuntu.com/archives/artful-changes/2017-July/005052.html
https://lists.ubuntu.com/archives/artful-changes/2017-May/001471.html
https://lists.ubuntu.com/archives/artful-changes/2017-May/001728.html

[Christian Ehrhardt  (paelzer)]

To quote something comparable, best old example to match the current case seems to be the following:
https://lists.ubuntu.com/archives/xenial-changes/2015-November/000988.html

[Christian Ehrhardt  (paelzer)]

The test builds in [1] soon abandoned) are good now.

Moving it into the same ppa that has the qemu-rc4 tests [2] to do a full regression test over night before uploading.

Along that preparing the changes for an extra review to get extra confidence.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2929
[2]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2918

[Christian Ehrhardt  (paelzer)]

No changes to Qemu planned to auto-fixup the migration, that is too complex to try so late in the cycle. Also due to Ubuntu Cloud Archive(s) not having ipxe in it we would have to add there as well or otherwise qemu 2.10 wouldn't be 2.10.
On LTS releases it might be better to push ipxe changes that break the migratability as all migrations have to go "though" LTS (like updates), so we can fix up there if we get that in qemu.

Spinning off a task for that in 18.04 in https://trello.com/c/ngcGOuAH

[Christian Ehrhardt  (paelzer)]

Related MP up in https://code.launchpad.net/~paelzer/ubuntu/+source/ipxe/+git/ipxe/+merge/329820

[Launchpad Janitor (janitor)]

This bug was fixed in the package ipxe - 1.0.0+git-20161027.b991c67+really20150424.a25a16d-1ubuntu1

---------------
ipxe (1.0.0+git-20161027.b991c67+really20150424.a25a16d-1ubuntu1) artful; urgency=medium

  * Revert to the former git snapshot 20150424.a25a16d to fix changed
    rom sizes that break cross release migrations (LP: #1713490).
    This makes it effectively identical to 1.0.0+git-20150424.a25a16d-1ubuntu2
    in regard to the upstream source, but keeps the changes to debian/*.
    On next merge we need to either ensure that rom sizes don't change, or
    that we can encapsulate that in qemu so that on forward migration it is
    taken care off.
    - This brings back debian/patches/0002-Don-t-use-libiberty.patch as needed
      on the older source.
    - Adapt d/p/0001-rom-change-banner-timeout.diff.patch to former state to
      match old source.
    - drop d/p/util-elf2efi-GNU_SOURCE.patch as it was not needed on old
      source
  * Fix FTBFS with newer perl versions (were dropped due to the revert above
    but is needed to build in artful)
    - d/p/0006-build-Fix-.ids.o-creation-for-drivers-not-in-the-all.patch
    - d/p/0007-build-Remove-nested-my-declaration.patch
  * d/util/check-rom-sizes, d/rules: check sizes of generated roms to avoid
    accidentially breaking KVM live migration on updates/fixes.

 -- Christian Ehrhardt <email address hidden> Tue, 29 Aug 2017 10:37:57 +0200

[Christian Ehrhardt  (paelzer)]

Reported it to Debian to agree on a solution which they can use in >=Buster and we in >=Bionic.
=> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=881263

[Christian Ehrhardt  (paelzer)]

change will need a new src package for the compat roms, so ipxe is not affected (but the compat is).
Qemu will carry the machine type dependent link changes matching the compat package.

[Christian Ehrhardt  (paelzer)]

actually ipxe will get the new sizes and a bump, so yes there is also a fix in src:ipxe so setting back to triaged on that package bug task.

[Christian Ehrhardt  (paelzer)]

Bonus: since the ipxe roms are reused arm64 and ppc64le are also affected and need the compat code in qemu to map to the compat rom files.

[Christian Ehrhardt  (paelzer)]

I can't file a MIR for the new src:ipxe-qemu-256k-compat we need for this bug 1713490.
Only after new queue I can open up requests.
We discussed that via mail and the TL;DR is: src:ipxe-qemu-256k-compat is the same as src:ipxe (xenial) and has the same timeline of support and therefore should fall under the same ack IMHO.

[Christian Ehrhardt  (paelzer)]

qemu 2.11 is in proposed

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu1

---------------
qemu (1:2.11+dfsg-1ubuntu1) bionic; urgency=medium

  * Merge with Debian testing, among other fixes this includes
    - fix fatal error on negative maxcpus (LP: #1722495)
    - fix segfault on dump-guest-memory on guests without memory (LP: #1723381)
    - linux user threading issues (LP: #1350435)
    - TOD-Clock Epoch Extension Support on s390x (LP: #1732691)
    Remaining changes:
    - qemu-kvm to systemd unit
      - d/qemu-kvm-init: script for QEMU KVM preparation modules, ksm,
        hugepages and architecture specifics
      - d/qemu-kvm.service: systemd unit to call qemu-kvm-init
      - d/qemu-system-common.install: install systemd unit and helper script
      - d/qemu-system-common.maintscript: clean old sysv and upstart scripts
      - d/qemu-system-common.qemu-kvm.default: defaults for
        /etc/default/qemu-kvm
      - d/rules: install /etc/default/qemu-kvm
    - Enable nesting by default
      - set nested=1 module option on intel. (is default on amd)
      - re-load kvm_intel.ko if it was loaded without nested=1
      - d/p/ubuntu/expose-vmx_qemu64cpu.patch: expose nested kvm by default
        in qemu64 cpu type.
      - d/p/ubuntu/enable-svm-by-default.patch: Enable nested svm by default
        in qemu64 on amd
    - libvirt/qemu user/group support
      - qemu-system-common.postinst: remove acl placed by udev, and add udevadm
        trigger.
      - qemu-system-common.preinst: add kvm group if needed
    - Distribution specific machine type
      - d/p/ubuntu/define-ubuntu-machine-types.patch: define distro machine
        types to ease future live vm migration.
      - d/qemu-system-x86.NEWS Info on fixed machine type defintions
    - improved dependencies
      - Make qemu-system-common depend on qemu-block-extra
      - Make qemu-utils depend on qemu-block-extra
      - let qemu-utils recommend sharutils
    - s390x support
      - Create qemu-system-s390x package
      - Include s390-ccw.img firmware
      - Enable numa support for s390x
    - ppc64[le] support
      - d/qemu-system-ppc.links provide usr/bin/qemu-system-ppc64le symlink
    - arch aware kvm wrappers
  * Added Changes
    - update VCS-git to match the bionic branch
    - sdl2 is yet too unstable for the LTS Ubuntu release given the reports
      we still see upstream and in Debian - furthermore sdl2 isn't in main yet,
      so we revert related changes to stick with the proven for now:
      - 0fd25810 - do not build-depend on libx11-dev (libsdl2-dev already
                   depends on it)
      - 9594f820 - switch from sdl1.2 to sdl2 (#870025)
    - d/qemu-system-x86.README.Debian: document intention of nested being
      default is comfort, not full support
    - update Ubuntu machine types for qemu 2.11
    - qemu-guest-agent: freeze-hook fixes (LP: #1484990)
      - d/p/guest-agent-freeze-hook-skip-dpkg-artifacts.patch
      - d/qemu-guest-agent.install: provide /etc/qemu/fsfreeze-hook
      - d/qemu-guest-agent.dirs: provide /etc/qemu/fsfreeze-hook.d
    - Create and install pxe netboot images for KVM s390x (LP: #1732094)
      - d/rules enable install s390x-netbo...

[Launchpad Janitor (janitor)]

This bug was fixed in the package ipxe - 1.0.0+git-20180124.fbe8c52d-0ubuntu2

---------------
ipxe (1.0.0+git-20180124.fbe8c52d-0ubuntu2) bionic; urgency=medium

  * debian/copyright: update copyright information to satisfy lintian
    dep5 checks (LP: #1747071)
  * d/p/enable-https.patch: adding proper dep3 header

 -- Christian Ehrhardt <email address hidden> Mon, 05 Feb 2018 15:09:28 +0100

[Christian Zunker (christian-zunker)]

We just upgraded an OpenStack compute from 16.04.4 to 16.04.5 and live migration is broken for VMs running on a 16.04.4 host to a 16.04.5 host.

We see these error messages in nova-compute.log on a 16.04.4 host:
2019-01-08 07:43:56.117 100671 ERROR nova.virt.libvirt.driver [-] [instance: 3ac1a38d-b3fa-4d02-a94e-b71136bcf07f] Migration operation has aborted
2019-01-08 07:43:56.230 100671 INFO nova.compute.manager [-] [instance: 3ac1a38d-b3fa-4d02-a94e-b71136bcf07f] Swapping old allocation on 61c651ad-62e7-4137-ab35-924cd204ebb9 held by migration
 1368009a-67aa-4574-9ed5-4fb7a513d06e for instance
2019-01-08 07:43:56.311 100671 ERROR nova.virt.libvirt.driver [-] [instance: 3ac1a38d-b3fa-4d02-a94e-b71136bcf07f] Live Migration failure: internal error: qemu unexpectedly closed the monitor
: 2019-01-08T07:43:55.658574Z qemu-system-x86_64: -drive file=rbd:volumes_hdd/volume-08bf868c-23a9-43c5-b00f-91e3dc7e4220:id=cinder:auth_supported=cephx\;none:mon_host=172.16.0.144\:6789\;172
.16.0.145\:6789\;172.16.0.149\:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,serial=08bf868c-23a9-43c5-b00f-91e3dc7e4220,cache=writeback,discard=unma
p: 'serial' is deprecated, please use the corresponding option of '-device' instead
2019-01-08T07:43:55.951328Z qemu-system-x86_64: Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x40000 in != 0x80000: Invalid argument
2019-01-08T07:43:55.951350Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'
2019-01-08T07:43:55.951499Z qemu-system-x86_64: load of migration failed: Invalid argument: libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-01-08T07:43:55.658574Z qem
u-system-x86_64: -drive file=rbd:volumes_hdd/volume-08bf868c-23a9-43c5-b00f-91e3dc7e4220:id=cinder:auth_supported=cephx\;none:mon_host=172.16.0.144\:6789\;172.16.0.145\:6789\;172.16.0.149\:67
89,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,serial=08bf868c-23a9-43c5-b00f-91e3dc7e4220,cache=writeback,discard=unmap: 'serial' is deprecated, please
 use the corresponding option of '-device' instead

During the update the package ipxe-qemu was installed in version 1.0.0+git-20180124.fbe8c52d-0ubuntu2.1~cloud0.

As suggested in comment #1 downgrading to 1.0.0+git-20150424.a25a16d-1ubuntu1.2 fixed the problem for us.

[Christian Ehrhardt  (paelzer)]

I subscribed James and Corey as this might be special to Ubuntu Cloud Archive.
Maybe an upgrade path that we have missed?

@James/Corey - do you see that as well?

@Christian Z. - what version of qemu was installed when failing?

Overall this looks like the ipxe from Bionic in the cloud archive for Xenial.
@James - does the UCA for B-to-X hold also a source of ipxe-qemu-256k-compat-efi-roms?
@James - does the qemu in UCA for B-to-X has [1] and [2]?

https://salsa.debian.org/qemu-team/qemu/commit/7b917f26ac6a224195f68088dcd8e8f99d932675
https://salsa.debian.org/qemu-team/qemu/commit/6409970b59979c6b0a80222d168956fc9867cac4

Overall what should happen is that
- if you had only the older qemu installed then guests used the old ipxe roms and got defined as pre 2.10 machine
- on a migration to a newer system (and Xenial + UCA copied from Bionic counts as new) it would detect that it is an old type and use the compat roms (matching old size for allocation)
- migration should work
- but the report says that is not true, hmm - needs more details

@Christian Z - could you share a "virsh dumpxml" of one of the guests (on the source) that fails to migrate?

[Christian Zunker (christian-zunker)]

@Christian E.

qemu version stayed the same:
ii qemu 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 fast processor emulator
ii qemu-block-extra:amd64 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU Full virtualization on x86 hardware
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU full system emulation binaries
ii qemu-system-arm 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU full system emulation binaries (arm)
ii qemu-system-common 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-mips 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU full system emulation binaries (mips)
ii qemu-system-misc 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU full system emulation binaries (miscellaneous)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU full system emulation binaries (ppc)
ii qemu-system-s390x 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU full system emulation binaries (s390x)
ii qemu-system-sparc 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU full system emulation binaries (sparc)
ii qemu-system-x86 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU full system emulation binaries (x86)
ii qemu-user 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU user mode emulation binaries
ii qemu-user-binfmt 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU user mode binfmt registration for qemu-user
ii qemu-utils 1:2.11+dfsg-1ubuntu7.5~cloud0 amd64 QEMU utilities

[Christian Zunker (christian-zunker)]

And this is the virsh dump: http://paste.openstack.org/show/740502/

[Christian Ehrhardt  (paelzer)]

Your guest already is pc-i440fx-bionic (because that is what the cloud archive is based on).
That means that qemu expects it to already run with the newer ipxe.

The only case I can think of is that this guest was started with
- qemu 1:2.11+dfsg-1ubuntu... installed
- ipxe not yet at 1.0.0+git-20180124.fbe8c52d-0ubuntu2

That never happens for Bionic as-is as people only get there with dist-upgrades (or new installs).
So they always have the new ipxe.
OTOH there isn't a struct dependency on >= 1.0.0+git-20180124.fbe8c52d-0ubuntu2

Therefore IMHO the problem is special to UCA, or almost broken (half) upgrades to Bionic

@James - I don't think we can safe guests started with Bionic level qemu but not Bionic level ipxe. Those just have to be restarted to get out of this (or temporarily downgrade on the target as Christian Z. did)
But we can think about adding a versioned dependency to avoid more of these conflicts to come up.
What is your opinion:
1. would you like the versioned dependency solution?
2. would you want that in UCA only or would you prefer to do that in Bionic as well?

P.S. if we go on we need a new bug for that.

[James Page (james-page)]

@paelzer

"@James - does the UCA for B-to-X hold also a source of ipxe-qemu-256k-compat-efi-roms?"

Yes

"@James - does the qemu in UCA for B-to-X has [1] and [2]?

https://salsa.debian.org/qemu-team/qemu/commit/7b917f26ac6a224195f68088dcd8e8f99d932675
https://salsa.debian.org/qemu-team/qemu/commit/6409970b59979c6b0a80222d168956fc9867cac4"

As long as those are in Bionic, they should be in the UCA for Queens (as it backports from bionic).

My preference would be to enforce use of the correct ipxe version if such a dependency exists.

[Christian Ehrhardt  (paelzer)]

As discussed on IRC this would most likely be a Cloud Archive only patch on top.
Therefore we don't even need a new qemu bug for it, we can add a task here.

[Christian Ehrhardt  (paelzer)]

For the actual change, you'd want to do this:
in qemu's debian/contol.in there are a bunch of Recommends and Depends like this:
  ipxe-qemu (>= 1.0.0+git-20131111.c3d1e78-1~)
change all those in UCA to
  ipxe-qemu (>= 1.0.0+git-20180124.fbe8c52d-0ubuntu2.1~)

That should avoid further issues with partial upgrades like this.
I never uploaded to UCA, will you do that James?

[norman shen (jshen28)]

Hello, thank you very much for working on this and reverting to an older ipxe package does solve the issue. but I am curious why qemu needs to compare it with local romfile? why cannot rom be synced from source vm?

[Christian Ehrhardt  (paelzer)]

@Norman - Not a bad idea. The content is synced from the source already, just the size allocation is what can mismatch - it happens so early that it is local. It was suggested like 3 times (I'm sure you could find them on the ML-archive somewhere) or so to transfer the source-size as part of the initial migration meta-data (that would then need a new arg to qemu to specify the size, so it would be qemu arg + libvirt migration data + libvirt rendering that arg accordingly + capability checks for the same). But so far no one ever (neither upstream nor any distro) found the time to implement this feature.