KVM "enabling virtualization on CPU0 failed"

Bug #1021271 reported by Jason Ashworth
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Invalid
Medium
Unassigned
qemu-kvm (Ubuntu)
Invalid
High
Unassigned

Bug Description

Background:
Ubuntu 12.04 server, running amd64 version, AMD Athlon 64 X2 5000+ CPU. KVM worked fine on the same physical hardware when previously Debian Lenny.

$ lsb_release -rd
Description: Ubuntu 12.04 LTS
Release: 12.04
$

$ apt-cache policy kvm
kvm:
  Installed: 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu13
  Candidate: 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu13
  Version table:
 *** 1:84+dfsg-0ubuntu16+1.0+noroms+0ubuntu13 0
        500 http://gb.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status
$

Expected result:

KVM should start and work without error, using hardware acceleration of CPU.

Issue:

When starting a VM, I get the following message:

$ kvm -cdrom BT5R2-KDE-32.iso -m 256 -net nic -net user -vnc 127.0.0.1:2
failed to initialize KVM: Device or resource busy
Back to tcg accelerator.

And I them get the following line appear in the dmesg output:

kvm: enabling virtualization on CPU0 failed

The virtual host will still run, but with very poor performance, because it is not able to use hardware acceleration capabilities of host CPU.

My account is in the KVM group (but I get the same symptoms even running as root):

Code:
$ ls -l /dev/kvm
crw-rw---- 1 root kvm 10, 232 Jun 30 07:42 /dev/kvm
$ whoami
jason
$ grep kvm /etc/group
kvm:x:105:jason
$

The KVM kernel modules are loaded:

$ lsmod | grep kvm
kvm_amd 55848 0
kvm 415459 1 kvm_amd
$

Here is some background info on the host to hopefully help:

Code:
$ uname -a
Linux cube 3.2.0-23-generic #36-Ubuntu SMP Tue Apr 10 20:39:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 2
On-line CPU(s) list: 0,1
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: AuthenticAMD
CPU family: 16
Model: 4
Stepping: 2
CPU MHz: 800.000
BogoMIPS: 4431.12
Virtualisation: AMD-V
L1d cache: 64K
L1i cache: 64K
L2 cache: 512K
NUMA node0 CPU(s): 0,1
$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : AMD Athlon(tm) 5000 Dual-Core Processor
stepping : 2
microcode : 0x1000086
cpu MHz : 800.000
cache size : 512 KB
physical id : 0
siblings : 2
core id : 0
cpu cores : 2
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt npt lbrv svm_lock nrip_save
bogomips : 4431.17
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

processor : 1
vendor_id : AuthenticAMD
cpu family : 16
model : 4
model name : AMD Athlon(tm) 5000 Dual-Core Processor
stepping : 2
microcode : 0x1000086
cpu MHz : 800.000
cache size : 512 KB
physical id : 0
siblings : 2
core id : 1
cpu cores : 2
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 5
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl nonstop_tsc extd_apicid pni monitor cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt npt lbrv svm_lock nrip_save
bogomips : 4431.12
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate

$

Revision history for this message
Soren Hansen (soren) wrote :

Do you perhaps have Virtualbox and/or VMWare installed as well?

Revision history for this message
Jason Ashworth (jasonin2d) wrote :

Hi Soren. No, I do not have any other virtualisation technologies installed. This is a fresh install (installed first time with base server install only, followed by "apt-get install kvm", and then a second fresh install choosing "Virtual server host" (or whatever the exact wording is) in the server roles selection screen at the end of the install.

Both methods of installation end with the same result.

Revision history for this message
Dave Walker (davewalker) wrote :

Can you provide the output of the command "kvm-ok" please?

Thanks.

Revision history for this message
Jason Ashworth (jasonin2d) wrote :

Sure:

$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
$

Thanks!

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Hi Jason,

could you please run 'apport-collect 1021271' to have apport submit some extra debugging information?

In addition, please show the result of:

    lsmod
    grep -i kvm /var/log/boot.log
    ls -l /dev/kvm
    dpkg -l | grep kvm

Finally, the only time I've seen symptoms like this so far has been when the bios reset itself to disable virtualization. Despite what kvm-ok says, could you please boot into bios and check for the hardware virtualization options?

affects: kvm (Ubuntu) → qemu-kvm (Ubuntu)
Changed in qemu-kvm (Ubuntu):
importance: Undecided → High
status: New → Incomplete
Revision history for this message
Jason Ashworth (jasonin2d) wrote :
Download full text (5.1 KiB)

Thanks, Serge.

I'm having trouble running apport-collect, as this is a headless server install, so only browser available is lynx, which seems to fail just after allowing authorisation.

However, here are the rest of the details you've requested:

$ lsmod
Module Size Used by
snd_hda_codec_hdmi 32474 1
ip6table_filter 12815 0
ip6_tables 27864 1 ip6table_filter
ebtable_nat 12807 0
ebtables 30984 1 ebtable_nat
ipt_MASQUERADE 12759 3
iptable_nat 13229 1
nf_nat 25891 2 ipt_MASQUERADE,iptable_nat
nf_conntrack_ipv4 19716 4 iptable_nat,nf_nat
nf_defrag_ipv4 12729 1 nf_conntrack_ipv4
xt_state 12578 1
nf_conntrack 81926 5 ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4,xt_state
ipt_REJECT 12576 2
xt_CHECKSUM 12549 1
iptable_mangle 12734 1
xt_tcpudp 12603 5
iptable_filter 12810 1
ip_tables 27473 3 iptable_nat,iptable_mangle,iptable_filter
x_tables 29846 12 ip6table_filter,ip6_tables,ebtables,ipt_MASQUERADE,iptable_nat,xt_state,ipt_REJECT,xt_CHECKSUM,iptable_mangle,xt_tcpudp,iptable_filter,ip_tables
bridge 91179 0
stp 12931 1 bridge
kvm_amd 55848 0
kvm 415459 1 kvm_amd
rc_dib0700_rc5 12508 0
dvb_usb_dib0700 123263 0
dib7000p 39231 3 dvb_usb_dib0700
dib0090 37608 1 dvb_usb_dib0700
dib7000m 23501 1 dvb_usb_dib0700
dib0070 18518 3 dvb_usb_dib0700
snd_hda_codec_realtek 223867 1
ir_lirc_codec 12859 0
lirc_dev 19204 1 ir_lirc_codec
ir_mce_kbd_decoder 12777 0
radeon 804372 1
snd_hda_intel 33773 0
ttm 76949 1 radeon
drm_kms_helper 46978 1 radeon
snd_hda_codec 127706 3 snd_hda_codec_hdmi,snd_hda_codec_realtek,snd_hda_intel
ir_sony_decoder 12510 0
dvb_usb 24490 1 dvb_usb_dib0700
dib8000 43141 1 dvb_usb_dib0700
snd_hwdep 13668 1 snd_hda_codec
ir_jvc_decoder 12507 0
ir_rc6_decoder 12507 0
snd_pcm 97188 3 snd_hda_codec_hdmi,snd_hda_intel,snd_hda_codec
drm 242038 3 radeon,ttm,drm_kms_helper
dvb_core 110619 3 dib7000p,dvb_usb,dib8000
edac_core 53746 0
ir_rc5_decoder 12507 0
dib3000mc 23392 1 dvb_usb_dib0700
sp5100_tco 13791 0
ir_nec_decoder 12507 0
mac_hid 13253 0
ppdev 17113 0
psmouse 87603 0
rc_core 26412 11 rc_dib0700_rc5,dvb_usb_dib0700,ir_lirc_codec,ir_mce_kbd_decoder,ir_sony_decoder,dvb_usb,ir_jvc_decoder,ir_rc6_decoder,ir_rc5_decoder,ir_nec_decoder
i2c_algo_bit 13423 1 radeon
serio_raw 13211 0
shpchp 37277 0
pl2303 17993 0
usbserial 47077 1 pl2303
snd_timer 29990 1 snd_pcm
dibx000_common 18810 5 dvb_usb_dib0700,dib7000p,dib7000m,dib8000,dib3000mc
...

Read more...

Changed in qemu-kvm (Ubuntu):
status: Incomplete → New
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Everything looks good. I have no problems running qemu-kvm in precise on an amd laptop. However it could be a chip-specific bug. I am adding the kernel to the list of affected packages.

tags: added: kernel-kvm
Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1021271

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
tags: added: precise
Revision history for this message
Jason Ashworth (jasonin2d) wrote :

Hi Brad,

I can't run apport-collect on this system, as detailed in Comment #6.

As requested, I've now set this bug to "Confirmed".

Many thanks.

Changed in qemu-kvm (Ubuntu):
status: New → Confirmed
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Jason Ashworth (jasonin2d) wrote :

As this wasn't getting much attention, and because I needed KVM, I decided to install Debian Squeeze amd64 on this box instead. KVM now works perfectly.

Unfortunately, this now means that I cannot test any more the problem that I had with Ubuntu 12.04. So, unless anyone else is affected by this bug, I guess it will get closed.

Thanks...

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

(For searchability, this is a problem with 12.04 qemu-kvm on a AMD Athlon(tm) 5000 Dual-Core Processor )

Jason, could you tell us the kernel version you are successfully using in squeeze?

Revision history for this message
Jason Ashworth (jasonin2d) wrote :

Certainly:

$ uname -a
Linux cube 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64 GNU/Linux
$
$ cat /etc/issue.net
Debian GNU/Linux 6.0
$
$ kvm --version
QEMU PC emulator version 0.12.5 (qemu-kvm-0.12.5), Copyright (c) 2003-2008 Fabrice Bellard
$

For clarity, this is the exact same hardware, and no BIOS settings modified before the install of Debian Squeeze.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

The kvm_amd module has had some other regressions since 2.6.32, so i wouldn't be surprised if a newer (3.2 or 3.5) kernel produced the same problem for you with debian. Hopefully we'll get to the bottom of it.

I'll go ahead and mark the bug invalid - an 'advanced search' will still bring up this bug if someone runs into it.

Thanks again.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Actually I'll mark it incomplete instead.

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Changed in qemu-kvm (Ubuntu):
status: Confirmed → Incomplete
Changed in linux (Ubuntu):
status: Incomplete → Invalid
Changed in qemu-kvm (Ubuntu):
status: Incomplete → Invalid
Revision history for this message
Jason Ashworth (jasonin2d) wrote :

Ok, thanks again for your help.

Revision history for this message
Hong Yuan (hongyuan) wrote :

Any progress with the bug?

I encouter the same problem now when upgrading from 10.04 to 12.04.3LTS. I am also using amd CPU, AMD Phenom(tm) II X4 940 to be precise.

After the upgrade, no virtual machine can be started with exactly the same error. I am considering a distribution downgrade if there is no progress with this bug.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Please run
   sudo kvm-ok

and show the results here. Please reboot and enter your bios setup and make sure that 'virtualization extensions' are on.

Revision history for this message
Hong Yuan (hongyuan) wrote :

$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

BIOS is OK. I haven't change anything during the upgrade to 12.04. And now I installed a parallel installation of 10.04 on another partition. When the machine is booted into that distribution, all VM's run as usual. It's just with 12.04 that the error occur:

$ dmesg | grep kvm
[ 16.662398] kvm: Nested Virtualization enabled
[ 16.662402] kvm: Nested Paging enabled
[ 18.496106] kvm: enabling virtualization on CPU0 failed
[ 18.556637] kvm: enabling virtualization on CPU0 failed
[ 212.738592] kvm: enabling virtualization on CPU0 failed
[ 213.179775] kvm: enabling virtualization on CPU0 failed
[ 213.540391] kvm: enabling virtualization on CPU0 failed

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Which kernel are you running? (Please show "uname -a" output)

To figure out whether this is a kernel or qemu problem, it would be useful to test kvm in a saucy chroot or container. Basically, you would

sudo apt-get -y install lxc
sudo lxc-create -t ubuntu -n s1 -- -r saucy
sudo lxc-start -n s1
   # log in as user ubuntu password ubuntu, then
   sudo apt-get -y install qemu-system-x86
   sudo mknod /dev/kvm c 10 232
   kvm

Revision history for this message
Hong Yuan (hongyuan) wrote :

hong@vhost01:~$ uname -a
Linux vhost01 3.2.0-54-generic #82-Ubuntu SMP Tue Sep 10 20:08:42 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

The result with saucy is the same as with precise:

ubuntu@s1:~$ sudo kvm
W: kvm binary is deprecated, please use qemu-system-x86_64 instead
failed to initialize KVM: Device or resource busy
Back to tcg accelerator.

(qemu-system-x86_64:4607): Gtk-WARNING **: cannot open display:

ubuntu@s1:~$ dmesg | tail -n1
[43436.367067] kvm: enabling virtualization on CPU0 failed

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Could you try setting

devices/system/cpu/cpu0/cpufreq/scaling_governor = performance
in /etc/sysfs.conf?

http://etbe.coker.com.au/2010/03/17/starting-with-kvm/ suggests that kvm may not like frequency scaling on on that cpu.

Changed in linux (Ubuntu):
status: Invalid → Incomplete
Revision history for this message
Hong Yuan (hongyuan) wrote :

Doesn't seem to work:

hong@vhost01:~$ sudo sh -c "echo 'performance' > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor"
hong@vhost01:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
hong@vhost01:~$ sudo kvm
failed to initialize KVM: Device or resource busy
Back to tcg accelerator.
Could not initialize SDL(No available video device) - exiting
hong@vhost01:~$ dmesg | grep kvm | tail -n1
[ 405.167600] kvm: enabling virtualization on CPU0 failed

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks. I think the best course here is to try the HWE (hardware-enablement) kernel. I believe the best bet would be to install linux-image-generic-lts-raring (3.8), however someone on the kernel team can hopefully confirm.

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
importance: Undecided → Medium
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I'm told it should be linux-generic-lts-<series> to get the headers as well as the kernel.

Revision history for this message
Hong Yuan (hongyuan) wrote :

Thanks for the help but alas it does not work this time either.

I have installed and tried both linux-generic-lts-quantal and linux-generic-lts-raring:

$ sudo wajig install linux-generic-lts-*

Result with quantal:

hong@vhost01:~$ uname -a
Linux vhost01 3.5.0-42-generic #65~precise1-Ubuntu SMP Wed Oct 2 20:57:18 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
hong@vhost01:~$ sudo kvm
failed to initialize KVM: Device or resource busy
Back to tcg accelerator.
Could not initialize SDL(No available video device) - exiting
hong@vhost01:~$ dmesg | grep kvm | tail
[ 14.381796] kvm: enabling virtualization on CPU2 failed
[ 14.381805] kvm: enabling virtualization on CPU0 failed
[ 14.438460] kvm: enabling virtualization on CPU0 failed
[ 14.438462] kvm: enabling virtualization on CPU1 failed
[ 14.438466] kvm: enabling virtualization on CPU2 failed
[ 14.438474] kvm: enabling virtualization on CPU3 failed
[ 33.561382] kvm: enabling virtualization on CPU0 failed
[ 33.561385] kvm: enabling virtualization on CPU1 failed
[ 33.561387] kvm: enabling virtualization on CPU3 failed
[ 33.561396] kvm: enabling virtualization on CPU2 failed

Then I boot into each one. Result with raring:

hong@vhost01:~$ uname -a
Linux vhost01 3.8.0-32-generic #47~precise1-Ubuntu SMP Wed Oct 2 16:19:35 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
hong@vhost01:~$ sudo kvm
failed to initialize KVM: Device or resource busy
Back to tcg accelerator.
Could not initialize SDL(No available video device) - exiting
hong@vhost01:~$ dmesg | grep kvm | tail
[ 14.255276] kvm: enabling virtualization on CPU2 failed
[ 14.255284] kvm: enabling virtualization on CPU0 failed
[ 64.876582] kvm: enabling virtualization on CPU0 failed
[ 64.876584] kvm: enabling virtualization on CPU3 failed
[ 64.876587] kvm: enabling virtualization on CPU2 failed
[ 64.876595] kvm: enabling virtualization on CPU1 failed
[ 290.615655] kvm: enabling virtualization on CPU0 failed
[ 290.615660] kvm: enabling virtualization on CPU1 failed
[ 290.615666] kvm: enabling virtualization on CPU3 failed
[ 290.615685] kvm: enabling virtualization on CPU2 failed

While with precies the error is only seen with CPU0, with the new kernels it is repeated on all CPU's.

With further Googling I stumled upon a post:

https://bbs.archlinux.org/viewtopic.php?id=148163

which suggests a BIOS upgrade might work. But since kvm worked perfectly under 10.04LTS, it should be software related and a BIOS upgrade would be a last resort as a workaround rather than a solution.

Revision history for this message
penalvch (penalvch) wrote :

Jason Ashworth, this bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? If so, could you please test for this with the latest development release of Ubuntu server? ISO images are available from http://cdimage.ubuntu.com/ubuntu-server/daily/current/ .

If it remains an issue, could you please attach your apport-collect following https://help.ubuntu.com/community/ReportingBugs#Filing_bugs_when_off-line ?

Changed in linux (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Jason Ashworth (jasonin2d) wrote :

Hi Christopher,
Unfortunately, I went back to Debian (which didn't have the bug) on this hardware, but now I no longer have access to this machine, so I can't do any further testing.
Unless someone else watching this thread can reproduce the problem, there's no further testing/troubleshooting that can be done.
Many thanks.

Revision history for this message
penalvch (penalvch) wrote :

Jason Ashworth, this bug report is being closed due to your last comment https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1021271/comments/27 regarding you no longer have access to the machine. For future reference you can manage the status of your own bugs by clicking on the current status in the yellow line and then choosing a new status in the revealed drop down box. You can learn more about bug statuses at https://wiki.ubuntu.com/Bugs/Status. Thank you again for taking the time to report this bug and helping to make Ubuntu better. Please submit any future bugs you may find.

Changed in linux (Ubuntu):
status: Incomplete → Invalid
Revision history for this message
Andika Triwidada (andika) wrote :

I got this bug on xenial 16.04.3
kernel 4.4.0-112-generic
CPU model name: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz

$ sudo kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

$ uname -a
Linux vmlab 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

$ dmesg|grep kvm
[21488.889798] kvm: enabling virtualization on CPU3 failed
[21488.889818] kvm: enabling virtualization on CPU1 failed
[21488.889819] kvm: enabling virtualization on CPU2 failed
[21488.889825] kvm: enabling virtualization on CPU0 failed
[21489.575476] kvm: enabling virtualization on CPU1 failed
[21489.575496] kvm: enabling virtualization on CPU0 failed
[21489.575498] kvm: enabling virtualization on CPU2 failed
[21489.575507] kvm: enabling virtualization on CPU3 failed
[21497.617250] kvm: enabling virtualization on CPU0 failed
[21497.617254] kvm: enabling virtualization on CPU2 failed
[21497.617268] kvm: enabling virtualization on CPU3 failed
[21497.617271] kvm: enabling virtualization on CPU1 failed
[21498.314797] kvm: enabling virtualization on CPU3 failed
[21498.314817] kvm: enabling virtualization on CPU1 failed
[21498.314819] kvm: enabling virtualization on CPU2 failed
[21498.314826] kvm: enabling virtualization on CPU0 failed

Please advice if you need more info or test to be done.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.