Hi,
i have several libvirt hosts servers with differents CPU generation, in particular :
- older generation : Intel Xeon E5-2640 v4 2.40GHz
- newer generation : Intel Xeon Gold 5215 2.50GHz
i recently re-install all this servers into ubuntu server 22.04.1
and since, when i live migrate a VM from a new generation processor to older generation processor
the migrated guest freeze without generating any error logs.
if i migrate the opposite way ( older cpu to newer cpu ) it works perfectly.
previous version of hosts ( same hardware on ubuntu 16.04 ) did not present the problem
the live migration is done with the following command ( issued from a third server playing the role of 'virtual-center' ) :
virsh -c qemu+ssh://root@new_server/system migrate --verbose --live --undefinesource --persistent --unsafe guest_name qemu+ssh://root@old_server/system
this one freeze the guest
while the opposite migration :
virsh -c qemu+ssh://root@old_server/system migrate --verbose --live --undefinesource --persistent --unsafe guest_name qemu+ssh://root@new_server/system
works without problem
migrate between 2 servers with same generation CPU also works perfectly
the cpu configuration of guest is generic :
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>qemu64</model>
<feature policy='require' name='x2apic'/>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='lahf_lm'/>
<feature policy='disable' name='svm'/>
</cpu>
i have tried several ( almost all ) other virtual cpu configuration , always with the same problem.
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 févr. 16 12:39 seq
crw-rw---- 1 root audio 116, 33 févr. 16 12:39 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2023-02-15 (0 days ago)
InstallationMedia: Ubuntu-Server 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: FUJITSU PRIMERGY RX2530 M2
Package: linux (not installed)
PciMultimedia:
ProcCmdline: BOOT_IMAGE=/boot/vmlinuz-5.15.0-60-generic root=UUID=13e84e97-ad18-49ed-8050-c8f7293e5e7d ro
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=fr_FR.UTF-8
SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-60-generic root=UUID=13e84e97-ad18-49ed-8050-c8f7293e5e7d ro
ProcVersionSignature: Ubuntu 5.15.0-60.66-generic 5.15.78
RelatedPackageVersions:
linux-restricted-modules-5.15.0-60-generic N/A
linux-backports-modules-5.15.0-60-generic N/A
linux-firmware 20220329.git681281e4-0ubuntu3.10
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy
Uname: Linux 5.15.0-60-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 09/29/2016
dmi.bios.release: 1.10
dmi.bios.vendor: FUJITSU // American Megatrends Inc.
dmi.bios.version: V5.0.0.11 R1.10.0 for D3279-B1x
dmi.board.name: D3279-B1
dmi.board.vendor: FUJITSU
dmi.board.version: S26361-D3279-B12 WGS03 GS02
dmi.chassis.asset.tag: System Asset Tag
dmi.chassis.type: 23
dmi.chassis.vendor: FUJITSU
dmi.chassis.version: RX2530M2R1
dmi.modalias: dmi:bvnFUJITSU//AmericanMegatrendsInc.:bvrV5.0.0.11R1.10.0forD3279-B1x:bd09/29/2016:br1.10:svnFUJITSU:pnPRIMERGYRX2530M2:pvrGS01:rvnFUJITSU:rnD3279-B1:rvrS26361-D3279-B12WGS03GS02:cvnFUJITSU:ct23:cvrRX2530M2R1:skuABNK1565-V101-236:
dmi.product.family: SERVER
dmi.product.name: PRIMERGY RX2530 M2
dmi.product.sku: ABN:K1565-V101-236
dmi.product.version: GS01
dmi.sys.vendor: FUJITSU
---
ProblemType: Bug
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 févr. 16 12:39 seq
crw-rw---- 1 root audio 116, 33 févr. 16 12:39 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.3
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: pass
DistroRelease: Ubuntu 22.04
InstallationDate: Installed on 2023-02-15 (0 days ago)
InstallationMedia: Ubuntu-Server 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lsusb:
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 004: ID 04b3:4010 IBM Corp. XClarity Controller
Bus 001 Device 002: ID 2a4b:0400 EMULEX Corporation Pilot4 Integrated Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t:
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/10p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/16p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/7p, 480M
|__ Port 6: Dev 4, If 0, Class=Communications, Driver=cdc_ether, 480M
|__ Port 6: Dev 4, If 1, Class=CDC Data, Driver=cdc_ether, 480M
MachineType: Lenovo ThinkSystem SR530 -[7X08CTO1WW]-
Package: linux (not installed)
PciMultimedia:
ProcCmdline: BOOT_IMAGE=/boot/vmlinuz-5.15.0-60-generic root=UUID=fc2c50fd-cc47-4243-918d-75af7d00b0e4 ro
ProcEnviron:
TERM=xterm-256color
PATH=(custom, no user)
XDG_RUNTIME_DIR=<set>
LANG=fr_FR.UTF-8
SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-60-generic root=UUID=fc2c50fd-cc47-4243-918d-75af7d00b0e4 ro
ProcVersionSignature: Ubuntu 5.15.0-60.66-generic 5.15.78
RelatedPackageVersions:
linux-restricted-modules-5.15.0-60-generic N/A
linux-backports-modules-5.15.0-60-generic N/A
linux-firmware 20220329.git681281e4-0ubuntu3.10
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy
Uname: Linux 5.15.0-60-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 10/31/2019
dmi.bios.release: 2.41
dmi.bios.vendor: Lenovo
dmi.bios.version: -[TEE148M-2.41]-
dmi.board.asset.tag: none
dmi.board.name: -[7X08CTO1WW]-
dmi.board.vendor: Lenovo
dmi.board.version: none
dmi.chassis.asset.tag: none
dmi.chassis.type: 23
dmi.chassis.vendor: Lenovo
dmi.chassis.version: none
dmi.ec.firmware.release: 3.8
dmi.modalias: dmi:bvnLenovo:bvr-[TEE148M-2.41]-:bd10/31/2019:br2.41:efr3.8:svnLenovo:pnThinkSystemSR530-[7X08CTO1WW]-:pvr08:rvnLenovo:rn-[7X08CTO1WW]-:rvrnone:cvnLenovo:ct23:cvrnone:sku7X08CTO1WW:
dmi.product.family: ThinkSystem
dmi.product.name: ThinkSystem SR530 -[7X08CTO1WW]-
dmi.product.sku: 7X08CTO1WW
dmi.product.version: 08
dmi.sys.vendor: Lenovo
Wow, interesting - thanks for the report.
With this CPU definition you already have almost all advanced features disabled.
And you said you even tried others - well done.
You also said that the same HW worked (with 16.04 instead of 22.04).
That is interesting.
Sadly, until we'd have access to comparable hardware that exposes this (I've tried two, not too similar, not showing the same), the best we could do it trying to corner the change that broke it.
So I wonder if you could try a few things, how possible (or not) would it be to:
- try newer code from [1], if it works we could bisect looking for a fix
- try older code, we do not yet know if it is qemu, libvirt, kernel or anything else.
We could check them once the hard way - e.g. if you say you can do so we could
bisect between qemu 2.5 (as in xenial) and 6.2 (as in jammy). Before I go in detail how
to do so, would that even be possible? Or are those production machines not to be
messed with too much?
[1]: https:/