s390x: migration to 4.2 makes it unable to migrate back to 4.0

[Christian Ehrhardt ]

Further testing has found more s390x migration issues.
Migration from a former release to the new stack (works) and back (fails).
Lets pick the closest release Eoan as an example.

A guest started on Eoan (libvirt 5.4 qemu 4.0) migrates to Focal (libvirt 6.0 / qemu 4.2) fine.
But on the way back it breaks with:
  error: internal error: qemu unexpectedly closed the monitor: 2020-01-29T10:41:04.977115Z qemu-system-s390x: can't apply global z13-base-s390x-cpu.apqi=on: Property '.apqi' not found

As you know this is a typical scenario on upgrades in case you want to e.g. roll back this issue would break your case.

apqi was added in 4.1 - so that all fits together.
But my expectation from recent years on x86 would have been that on the forward migration the attribute isn't added and then due to that it could properly migrate backward again.

All this seems to be part of the cpu modelling.
- guest is started on source without <cpu ... model>
- migration to the new stack adds <cpu ... model>
- reverse migration is impossible

If I start with host-model on the source then the model stays unmodified and reverse migration works as well.

I know you need proper CPU modelling to make any guarantees, but if the source has none I'd expect the migration not to add one making it impossible to move back.

Please let me know if for you (=for IBM) this is a "oh crap that needs to be fixed" or a "well that is what it is" severity bug?

---

please ignore comment #2 and comment #3 - that is a good case by using host-model on the source.
The data for the bad case is in comment #6

[Christian Ehrhardt  (paelzer)]

Host-model as in base template being something like:
<domain type='kvm'>
  <os>
    <type>hvm</type>
    <boot dev='hd'/>
  </os>
  <cpu mode='host-model'/> <-- here
  <devices>
    <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
    </interface>
    <console type='pty' tty='/dev/pts/3'>
      <source path='/dev/pts/3'/>
      <target type='sclp' port='0'/>
      <alias name='console0'/>
    </console>
  </devices>
</domain>

[Christian Ehrhardt  (paelzer)]

XMLs and commandlines

1. start on Eoan

root@testkvm-eoan-from:~# uvt-kvm create --template /root/guesttemplate-hm.xml --password=ubuntu --ssh-public-key-file /home/ubuntu/.ssh/authorized_keys test1 release=eoan arch=s390x label=daily
Warning: using --password from the command line is not secure and should be used for debugging only.
root@testkvm-eoan-from:~# cat /var/log/libvirt/qemu/test1.log
2020-01-29 15:09:01.850+0000: starting up libvirt version: 5.4.0, package: 0ubuntu5 (Matthias Klose <email address hidden> Thu, 05 Sep 2019 11:00:53 +0000), qemu version: 4.0.0Debian 1:4.0+dfsg-0ubuntu9.2, kernel: 5.4.0-9-generic, hostname: testkvm-eoan-from
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-1-test1 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-test1/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-test1/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-test1/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-s390x \
-name guest=test1,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-test1/master-key.aes \
-machine s390-ccw-virtio-eoan,accel=kvm,usb=off,dump-guest-core=off \
-cpu z13-base,aen=on,aefsi=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,edat2=on,vx=on,ipter=on,cei=on,ap=on,gpereh=on,esop=on,ib=on,siif=on,ibs=on,apft=on,sief2=on,apqci=on,cte=on,bpb=on,64bscao=on,ppa15=on,zpci=on,sea_esop2=on,te=on,cmm=on,gsls=on \
-m 512 \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 8ac65dcc-72be-4ae6-b397-766bc7a0c3ad \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=25,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-drive file=/var/lib/uvtool/libvirt/images/test1.qcow,format=qcow2,if=none,id=drive-virtio-disk0 \
-device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
-drive file=/var/lib/uvtool/libvirt/images/test1-ds.qcow,format=qcow2,if=none,id=drive-virtio-disk1 \
-device virtio-blk-ccw,scsi=off,devno=fe.0.0001,drive=drive-virtio-disk1,id=virtio-disk1 \
-netdev tap,fd=27,id=hostnet0 \
-device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:c7:ef:c6,devno=fe.0.0002 \
-chardev pty,id=charconsole0 \
-device sclpconsole,chardev=charconsole0,id=console0 \
-device virtio-balloon-ccw,id=balloon0,devno=fe.0.0003 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charconsole0)

root@testkvm-eoan-from:~# virsh dumpxml test1
<domain type='kvm' id='1'>
  <name>test1</name>
  <uuid>8ac65dcc-72be-4ae6-b397-766bc7a0c3ad</uuid>
  <metadata>
    <uvt:ssh_known_hosts xmlns:uvt="https://launchpad.net/uvtool/libvirt/1">ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDYCkr8p9r/bQu+sam1L6kmfX+QPMrOtBMrGUWi4WeictIbcyYrRdJunmMptLlGMs6501D4+vkYE0xdMoYtpSb75AsmHNly59oBb9EnSGDNowXTyntDP/5akXuy/IqRBh+ZOlctrq5jInRn+lq+sAYW4twrVPDxCeWTjcrRQdSCayR3pu5JEk2s0sUp//pB2XIuexPQV2BuX7q0+ww45gm2MGpodP1vRh4ENNm0OQHQWSC5w36lvz2fgTwSH0rssMN1EREtR7pb3Rf7iZYmYxWZ7...

[Christian Ehrhardt  (paelzer)]

XMLs and commandlines

2. after migration to Focal on the target

root@testkvm-focal-from:~# cat /var/log/libvirt/qemu/test1.log
2020-01-29 15:10:41.317+0000: starting up libvirt version: 6.0.0, package: 0ubuntu1~ppa3 (Christian Ehrhardt <email address hidden> Mon, 13 Jan 2020 13:14:14 +0100), qemu version: 4.2.0Debian 1:4.2-1ubuntu1~ppa4, kernel: 5.4.0-9-generic, hostname: testkvm-focal-from
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-1-test1 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-test1/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-test1/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-test1/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-s390x \
-name guest=test1,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-1-test1/master-key.aes \
-machine s390-ccw-virtio-eoan,accel=kvm,usb=off,dump-guest-core=off \
-cpu z13-base,aen=on,aefsi=on,msa5=on,msa4=on,msa3=on,msa2=on,msa1=on,sthyi=on,edat=on,ri=on,edat2=on,vx=on,ipter=on,cei=on,ap=on,gpereh=on,esop=on,ib=on,siif=on,ibs=on,apft=on,sief2=on,apqci=on,cte=on,bpb=on,64bscao=on,ppa15=on,zpci=on,sea_esop2=on,te=on,cmm=on,gsls=on \
-m 512 \
-overcommit mem-lock=off \
-smp 1,sockets=1,cores=1,threads=1 \
-uuid 8ac65dcc-72be-4ae6-b397-766bc7a0c3ad \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=28,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-shutdown \
-boot strict=on \
-blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/x-uvt-b64-Y29tLnVidW50dS5jbG91ZC5kYWlseTpzZXJ2ZXI6MTkuMTA6czM5MHggMjAyMDAxMjg=","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/test1.qcow","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \
-device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=libvirt-2-format,id=virtio-disk0,bootindex=1 \
-blockdev '{"driver":"file","filename":"/var/lib/uvtool/libvirt/images/test1-ds.qcow","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device virtio-blk-ccw,scsi=off,devno=fe.0.0001,drive=libvirt-1-format,id=virtio-disk1 \
-netdev tap,fd=31,id=hostnet0 \
-device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:c7:ef:c6,devno=fe.0.0002 \
-chardev pty,id=charconsole0 \
-device sclpconsole,chardev=charconsole0,id=console0 \
-incoming defer \
-device virtio-balloon-ccw,id=balloon0,devno=fe.0.0003 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charconsole0)

root@testkvm-focal-from:~# virsh dumpxml test1
<do...

[Christian Ehrhardt  (paelzer)]

The above is an example that works.
In other cases (I need to spot the difference) the target becomes an xml that contains:

   <feature policy='\''require'\'' name='\''apqi'\''/>

And that then fails on the reverse migration.

[Christian Ehrhardt  (paelzer)]

Oh I see, it seems to be the inverse - the issue triggers when NOT using <cpu mode='host-model'/> on the initial source.

base template is like:
<domain type='kvm'>
  <os>
    <type>hvm</type>
    <boot dev='hd'/>
  </os>
  <devices>
    <interface type='network'>
      <source network='default'/>
      <model type='virtio'/>
    </interface>
    <console type='pty' tty='/dev/pts/3'>
      <source path='/dev/pts/3'/>
      <target type='sclp' port='0'/>
      <alias name='console0'/>
    </console>
  </devices>
</domain>

[Christian Ehrhardt  (paelzer)]

XMLs and commandlines

1. start on Eoan

root@testkvm-eoan-from:~# virsh dumpxml test2
<domain type='kvm' id='3'>
  <name>test2</name>
  <uuid>e85386c2-025e-4174-9838-6a8d8abfdff5</uuid>
  <metadata>
    <uvt:ssh_known_hosts xmlns:uvt="https://launchpad.net/uvtool/libvirt/1">ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDoIHwbOHYsm0Qcie2bavPSYd2qy3f60ZpDLGW/4S8Gob6CLyLrgNg9xkbQCOwHPsXncMpJ/x6S+ngtd4C4XnsUOq9S94DUFWvanuVqWRqxX1nryLbIGUnpTRDHQHmxRDPx752zqhzRG9s4ouDzWIkulnF4TRnFt86+ofhQl4QPvM19l6l+bNaekxoDwoVcHTAcnj1EVcc3ro4DzZNgsRNd7rSlOMCTPaGlPxLZo2x2RjcoLj9mQrH/8zS0Gb6YDK3vCQid6mifEnQr/BfXMdEC31LAAqIeh0big8aodVWM4QwzvaXLdNnRxDYvJreEHbRRIBc3wCVma/aWl2XqfGO8szbONQAyAR39JmBoqkR5ZLTjLvJAyOEMJ76owhHftn1o4DJtOPYKhpE7OGOxrQfNyLNru81qxElKuvZ8UVdlt/uwYU/WM27UfytiQCyrOIRC75bywh4DDnXGuzGvAiZS1tXb1fn8L3i17sktzpvtQtWFir27ch05kpvRchrs+9U= root@localhost
ssh-dss AAAAB3NzaC1kc3MAAACBAKbEdP8m+lNW1Jrr9B3FtY8KjmsRUneE0fEihfUFMMbFyl6eD+iGaIMqUF/3qR0zeWbGemaZN3rsEDAfuTnjnEOpB5ec7oYUYYO/lcUyeyoODzh2PvGcLUh5d1OKL1vThlQhahJ3uUGZZLtxAH2YpbdM7YdijI2KoT7Cdxr9M6qfAAAAFQCEItec7CJd85ay9GRnx41JKNWlJQAAAIANAnulXTlZC0nr/qtIWPsDC2qPQ/AuzESM0Hbpi5BfpJYZrlnaM9Ex2UJ99UFdfJ6UKYBskL0XxTaJQAUvIuI+bEz4JCQ+XgKQsBb7NMHKTt2B6PMNluHLoqMbfq20IL6H/sKmlrkEGRa0chzgKZiIdi39IucS5tdKYJE0tPLRSQAAAIADHednAthRmch9bhg06R7jL22WrQ656a6JvJ7dR2wpuxyPd4Vl5XDCWPiCBQG2KGYpg+DfP3a1fd103e4S83CXhSKFeiRR1FL2K17oh60ARihQ8IvtMx79vaI4HMmF3cC1IztfK6EA0mMzVVERb42m2xE6U+BQ6iF8u4xL6RvTiw== root@localhost
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOIi/7nkLnkwKK8VK+Nj0CW/nRTi15OmCOlcMAjWp4GomOXVD28yuO3xbkBk5FNuj3hxALbSBY9Ww7OlUYUZv2s= root@localhost
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDJhMoSQ0OcWldCluRPE9pI3YYrEYQLHAH8jWhPQM9zZ root@localhost
</uvt:ssh_known_hosts>
  </metadata>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='s390x' machine='s390-ccw-virtio-eoan'>hvm</type>
    <boot dev='hd'/>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-s390x</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/uvtool/libvirt/images/test2.qcow'/>
      <backingStore type='file' index='1'>
        <format type='qcow2'/>
        <source file='/var/lib/uvtool/libvirt/images/x-uvt-b64-Y29tLnVidW50dS5jbG91ZC5kYWlseTpzZXJ2ZXI6MTkuMTA6czM5MHggMjAyMDAxMjg='/>
        <backingStore/>
      </backingStore>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0000'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/uvtool/libvirt/images/test2-ds.qcow'/>
      <backingStore/>
      <target dev='vdb' bus='virtio'/>
      <alias name='virtio-disk1'/>
      <address type='ccw' cssid='0xfe' ssid='0x0' devno='0x0001'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </contr...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-02-03 09:25 EDT-------
So can you clarify: with host-model it DOES work.
It breaks if you have not specified a cpu model before?

[Christian Ehrhardt  (paelzer)]

Hi Christian,
summarizing for disambiguation:

a) If I specify CPU model on the source it carries over correctly and can migrate back
b) If I specify nothing on the source it "becomes" host-model on the target (with the features as detected there at the time) and those can't all go back

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-02-07 04:17 EDT-------
(In reply to comment #11)
> Hi Christian,
> summarizing for disambiguation:
>
> a) If I specify CPU model on the source it carries over correctly and can
> migrate back

This is good. host-model is supposed to work.
FWIW, custom is also supposed to work (e.g. -cpu z13).

> b) If I specify nothing on the source it "becomes" host-model on the target
> (with the features as detected there at the time) and those can't all go back

This is also kind of expected. Specifying nothing was "host-passthrough". This is known to be NOT migration safe. Do not use it.

Newer libvirt versions will now add "host-passthrough" in the xml when nothing is specified. As this is not migration-safe, I asked libvirt to change the default for s390 to become "host-model" instead of "host-passthrough".
You should also see it as a feature that we add a check that tells you: "look, your migration is not safe"

[Christian Ehrhardt  (paelzer)]

Ok, we fixed the other and this one can be kept as-is - closing bugs

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-02-24 08:43 EDT-------
IBM bugzilla status-> closed, Fix Released by Canonical