multi-core FreeBSD guest hangs after warm reboot

[John Nielsen]

On some Linux KVM hosts in our environment, FreeBSD guests fail to reboot properly if they have more than one CPU (socket, core, and/or thread). They will boot fine the first time, but after issuing a "reboot" command via the OS the guest starts to boot but hangs during SMP initialization. Fully shutting down and restarting the guest works in all cases.

The only meaningful difference between hosts with the problem and those without is the CPU. Hosts with Xeon E5-26xx v2 processors have the problem, including at least the "Intel(R) Xeon(R) CPU E5-2667 v2" and the "Intel(R) Xeon(R) CPU E5-2650 v2".
Hosts with any other CPU, including "Intel(R) Xeon(R) CPU E5-2650 0", "Intel(R) Xeon(R) CPU E5-2620 0", or "AMD Opteron(TM) Processor 6274" do not have the problem. Note the "v2" in the names of the problematic CPUs.

On hosts with a "v2" Xeon, I can reproduce the problem under Linux kernel 3.10 or 3.12 and Qemu 1.7.0 or 2.0.0.

The problem occurs with all currently-supported versions of FreeBSD, including 8.4, 9.2, 10.0 and 11-CURRENT.

On a Linux KVM host with a "v2" Xeon, this command line is adequate to reproduce the problem:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512 -smp 2,sockets=1,cores=1,threads=2 -drive file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2 -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none

I have tried many variations including different models of -machine and -cpu for the guest with no visible difference.

A native FreeBSD installation on a host with a "v2" Xeon does not have the problem, nor do a paravirtualized FreeBSD guests under bhyve (the BSD legacy-free hypervisor) using the same FreeBSD disk images as on the Linux hosts. So it seems unlikely the cause is on the FreeBSD side of things.

I would greatly appreciate any feedback or developer attention to this. I am happy to provide additional details, test patches, etc.

[Matt (l-matts)]

I'm having this same issue. Tried with both FreeNAS (which uses FreeBSD 9), and a minimal install of FreeBSD 10. Everything seems to work great up until you try to do a warm boot. I've compiled a custom FreeBSD kernel with everything unnecessary removed, and also changed the number of CPUs assigned to the guest. Nothing seems to work above a single CPU.

I'm on a fully patched Ubuntu 14.04 system. Here is the top of my /proc/cpuinfo from the Ubuntu hypervisor.

processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
stepping : 4
microcode : 0x416
cpu MHz : 2500.010
cache size : 25600 KB
physical id : 0
siblings : 20
core id : 0
cpu cores : 10

[Serge Hallyn (serge-hallyn)]

Hi,

could you verify that you have the same result when using
https://launchpad.net/~ubuntu-virt/+archive/virt-daily-upstream

 status: incomplete
 importance: medium

[John Nielsen (r-2ohn-d)]

Our KVM hosts run CentOS 6 plus a custom 3.12 kernel rather than Ubuntu or Debian, so I won't be able to use the PPA directly. However, I will build and test with the latest Qemu git sources.

On Jun 16, 2014, at 12:17 PM, Serge Hallyn <email address hidden> wrote:

> Hi,
>
> could you verify that you have the same result when using
> https://launchpad.net/~ubuntu-virt/+archive/virt-daily-upstream

[John Nielsen (r-2ohn-d)]

I have the same result when running Qemu built today from:

commit af44da87e926ff64260b95f4350d338c4fc113ca
Merge: f277015 9dbae97
Author: Peter Maydell <email address hidden>
Date: Mon Jun 16 18:26:21 2014 +0100

I also tried a 3.15 kernel to see if any recent KVM changes would help, but the problem remains.

[John Nielsen (r-2ohn-d)]

Problem also remains with CPU microcode revision 0x427 (2014-04-10).

[radhakrishnan (gradhakr)]

I have the same issue :
KMV veriosn: 3.10.0-123.el7.x86_64 SMP mod_unload modversions
author: Qumranet

[John Nielsen (r-2ohn-d)]

For the benefit of the last commenter and anyone else who comes across this ticket:

As determined on the mailing list in June, the bug appears to be with KVM's apicv on processors that support the feature. I haven't heard anything about a fix, but the best workaround is to disable apicv when loading the KVM kernel module, e.g.:

# modprobe kvm_intel enable_apicv=N

You can verify the parameter by checking the contents of /sys/module/kvm_intel/parameters/enable_apicv.

[radhakrishnan (gradhakr)]

Thanks John !! Let me try that

[Vasiliy Tolstov (vase)]

In my testing disable apicv does not needed, but i need latest stable seabios from seabios site.

[Paolo Bonzini (bonzini)]

Vasiliy,

if a different SeaBIOS is needed, that's another bug. You can use hint.atkbd.0.disabled="1" to work around it.

But if disable apicv is not needed, can you please include your cpuinfo here?

[John Nielsen (r-2ohn-d)]

I am no longer able to reproduce this issue on a fully-updated server. My guess is that the issue was fixed in the kernel somewhere between 3.12 and 4.0, but for all I know it could be a Qemu (or even Seabios) change. Here are details of my test that failed and the one that succeeded.

Breaks (VM hangs during boot after pressing ctrl-alt-del):
kernel 3.12.22
qemu-kvm-1.7.0-3.el6.x86_64
seabios-1.7.3.1-1.el6.noarch
Intel(R) Xeon(R) CPU E5-2667 v2 @ 3.30GHz

Works (VM reboots normally):
kernel 4.0.4
qemu-kvm-2.3.0-6.el7.centos.x86_64
seabios-bin-1.8.1-1.el7.centos.noarch
Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz

I'd still like to narrow down the change that fixed it if possible.

[Venkateswara Rao Dokku (dvrao-584)]

Can you please tell me whether the issue is fixed with the latest kernel? If so, what version has the fix?

[John Nielsen (r-2ohn-d)]

Yes it is. Not sure what version first fixed it but I know 4.1 works.

> On Aug 20, 2015, at 2:30 AM, Venkateswara Rao Dokku <email address hidden> wrote:
>
> Can you please tell me whether the issue is fixed with the latest
> kernel? If so, what version has the fix?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1329956
>
> Title:
> multi-core FreeBSD guest hangs after warm reboot
>
> Status in QEMU:
> Incomplete
>
> Bug description:
> On some Linux KVM hosts in our environment, FreeBSD guests fail to
> reboot properly if they have more than one CPU (socket, core, and/or
> thread). They will boot fine the first time, but after issuing a
> "reboot" command via the OS the guest starts to boot but hangs during
> SMP initialization. Fully shutting down and restarting the guest works
> in all cases.
>
> The only meaningful difference between hosts with the problem and those without is the CPU. Hosts with Xeon E5-26xx v2 processors have the problem, including at least the "Intel(R) Xeon(R) CPU E5-2667 v2" and the "Intel(R) Xeon(R) CPU E5-2650 v2".
> Hosts with any other CPU, including "Intel(R) Xeon(R) CPU E5-2650 0", "Intel(R) Xeon(R) CPU E5-2620 0", or "AMD Opteron(TM) Processor 6274" do not have the problem. Note the "v2" in the names of the problematic CPUs.
>
> On hosts with a "v2" Xeon, I can reproduce the problem under Linux
> kernel 3.10 or 3.12 and Qemu 1.7.0 or 2.0.0.
>
> The problem occurs with all currently-supported versions of FreeBSD,
> including 8.4, 9.2, 10.0 and 11-CURRENT.
>
> On a Linux KVM host with a "v2" Xeon, this command line is adequate to
> reproduce the problem:
>
> /usr/bin/qemu-system-x86_64 -machine accel=kvm -name bsdtest -m 512
> -smp 2,sockets=1,cores=1,threads=2 -drive
> file=./20140613_FreeBSD_9.2-RELEASE_ufs.qcow2,if=none,id=drive0,format=qcow2
> -device virtio-blk-pci,scsi=off,drive=drive0 -vnc 0.0.0.0:0 -net none
>
> I have tried many variations including different models of -machine
> and -cpu for the guest with no visible difference.
>
> A native FreeBSD installation on a host with a "v2" Xeon does not have
> the problem, nor do a paravirtualized FreeBSD guests under bhyve (the
> BSD legacy-free hypervisor) using the same FreeBSD disk images as on
> the Linux hosts. So it seems unlikely the cause is on the FreeBSD side
> of things.
>
> I would greatly appreciate any feedback or developer attention to
> this. I am happy to provide additional details, test patches, etc.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1329956/+subscriptions
>

[Thomas Huth (th-huth)]

OK, according to the last comments, the bug has been fixed somewhere with the last kernel or QEMU releases, so I'm closing this ticket now.

[Matt Keys (mk6032)]

I'm able to reproduce this issue, but using latest debian 9.

Debian 9
qemu version: 1:2.8+dfsg-6+deb9u3
kernel version: Linux vm2 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u5 (2017-09-19) x86_64 GNU/Linux

I'm attempting to cold boot, or warm reboot, pfsense 2.4.2 amd64 iso image guest. If I have > 1 in virt-manager view -> details -> cpu -> allocation and maximum allocation, then the guest will not boot. My workaround was to set those both to 1, then in configuration I needed to uncheck "Copy Host CPU Configuration" (pfsense used to need this for hardware crypto support) and set the model to "clear cpu configuration" in order for it to boot. This doesn't appear to be Intel specific. I'm running amd ..

/proc/cpuinfo :

processor : 0
vendor_id : AuthenticAMD
cpu family : 21
model : 1
model name : AMD FX(tm)-8120 Eight-Core Processor
stepping : 2
microcode : 0x6000629
cpu MHz : 1400.000
cache size : 2048 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc extd_apicid aperfmperf eagerfpu pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core perfctr_nb cpb hw_pstate vmmcall arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs : fxsave_leak sysret_ss_attrs null_seg
bogomips : 6241.40
TLB size : 1536 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb

[Matt Keys (mk6032)]

I found this bug report through https://redmine.pfsense.org/issues/7925 , btw.

[Matt Keys (mk6032)]

sorry, make that https://redmine.pfsense.org/issues/4377

[John Nielsen (r-2ohn-d)]

Matt, does disabling apicv on the hypervisor as above work around the issue for you?