qcow2 unusably slow when migrating from 8.04LTS to 10.04LTS

[Steven Wagner]

Binary package hint: qemu-kvm

After a clean install, all I get is a blank screen and high cpu for win2k3 guests. Reproducable, ive tried installing a dozen times with the same result every time. I also have tried without using viritio for the hd, but I get the same result. Ive tried tweaking nearly all of the libvirt settings with no difference. I will attach my xml conf as an example.

ProblemType: Bug
DistroRelease: Ubuntu 10.04
Package: qemu-kvm 0.12.3+noroms-0ubuntu9.2
ProcVersionSignature: Ubuntu 2.6.32-25.45-server 2.6.32.21+drm33.7
Uname: Linux 2.6.32-25-server x86_64
Architecture: amd64
Date: Thu Nov 4 13:53:49 2010
InstallationMedia: Ubuntu-Server 10.04 LTS "Lucid Lynx" - Release amd64 (20100427)
MachineType: Supermicro X8STi
ProcCmdLine: BOOT_IMAGE=/vmlinuz-2.6.32-25-server root=/dev/mapper/hostname-root ro quiet
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: qemu-kvm
dmi.bios.date: 03/10/2010
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1.0c
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: X8STi
dmi.board.vendor: Supermicro
dmi.board.version: 1234567890
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 1234567890
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1.0c:bd03/10/2010:svnSupermicro:pnX8STi:pvr1234567890:rvnSupermicro:rnX8STi:rvr1234567890:cvnSupermicro:ct3:cvr1234567890:
dmi.product.name: X8STi
dmi.product.version: 1234567890
dmi.sys.vendor: Supermicro

[Serge Hallyn (serge-hallyn)]

Could you try running the VM without libvirt, using vnc?

Something like

kvm -drive file=/mnt/slices/newtuna.qcow2,if=virtio,index=0,boot=on -m 1G -vnc :1

should suffice. Then connect to it using 'vcnviewer :1'.

[Serge Hallyn (serge-hallyn)]

(Note, I'm not asking you to make this your regular procedure, just trying
to figure out where the problem is)

[Steven Wagner (stevenwagner)]

Serge - I did as you requested. When running the guest using that kvm command, and then accessing it with vncviewer, I still get the same black screen. 'top' on the server shows the process using 99% cpu.

[Serge Hallyn (serge-hallyn)]

I am trying to reproduce this locally - can you tell me exactly which version you are
using? I am planning to try windows server 2003 Standard edition, SP2.

[Serge Hallyn (serge-hallyn)]

Couldn't reproduce with a win2003 32-bit R2 SP2 on 10.10 host. I'll try on
10.04 when I have a chance to reboot.

[Serge Hallyn (serge-hallyn)]

Could not reproduce on 10.04 (amd64) host either.

Please start up the VM with libvirt, and do 'ps -ef | grep kvm' to find the
command being used to start the VM. Then trim things like vnc, serial,
parallel and monitor options, add 'monitor stdio', and start the command
by hand. Try it once with no vnc option, and once with vnc :1 (and
subsequently connect with 'vncviewer :1' from a different terminal).

For instance, in my case the original command was

/usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 1 -name windows -uuid 376b7084-555c-d0b3-a1b9-ae165b56c4ca -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/windows.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/mnt/srv/libvirt-storage-pool-1/win2003-r2-sp2-32.img,if=ide,index=0,boot=on,format=raw -drive if=ide,media=cdrom,index=2,format=raw -net nic,macaddr=52:54:00:33:ed:0b,vlan=0,name=nic.0 -net tap,fd=59,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 127.0.0.1:0 -vga cirrus

so I ran

/usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 512 -smp 1 -name windows -boot c -drive file=/mnt/srv/libvirt-storage-pool-1/win2003-r2-sp2-32.img,if=ide,index=0,boot=on,format=raw -monitor stdio -vga cirrus

[Serge Hallyn (serge-hallyn)]

Actually, I just noticed that your .xml specifies virtio for the block device.
Win 2003 by default does not support that. Did you do anything special
during install? Did you install the virtio drivers, and if so, where did you
get them?

[Steven Wagner (stevenwagner)]

Serge -
Ill jump to the point and tell you that Ive tried without virtio as well, and the problem remains that same. Thanks for your continued help on this issue. Im blocked on migrating any windows hosts for now. The only other time ive seen something like this was when my server has Cpu frequency scaling turned on before..its would cause live hosts to lock and go to 99% cpu usage. But ive already made sure cpu frequency scaling is off.

[Serge Hallyn (serge-hallyn)]

Steven,

so when you ran kvm from the command line without using libvirt, as described
in comment #8, did you still have the failure? What exactly did your command
look like?

The simplest command which should definately work would be:

   qemu-img create win1.img 20G
   kvm -hda win1.img -cdrom win2003*.iso -m 1G -boot d

[Steven Wagner (stevenwagner)]

Serge,
The version of windows I am using is, Microsoft Windows Server 2003 R2 with SP2 Enterprise Edition.
The kvm command I tested was:
    kvm -drive file=/mnt/slices/newtuna.qcow2,if=virtio,index=0,boot=on -m 1G -vnc :1
 and that gave me the same result, blank screen, high cpu usage.

I just now tried this...
I took the running line from ps which is:

/usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 1 -name newtuna -uuid 21a3c860-05b0-5dcf-cc8c-9243604f8e9b -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/newtuna.monitor,server,nowait -monitor chardev:monitor -localtime -boot c -drive file=/mnt/slices/newtuna.qcow2,if=virtio,index=0,boot=on,format=qcow2 -drive file=/mnt/isos/win2003/CRMEVOL_EN.ISO,if=ide,media=cdrom,index=2,format=raw -drive file=/mnt/isos/virtio-win-1.1.11-0.iso,if=ide,media=cdrom,index=3,format=raw -drive file=/mnt/isos/viostor-31-03-2010-floppy.img,if=floppy,index=0,format=raw -net nic,macaddr=52:54:00:68:23:96,vlan=0,model=virtio,name=virtio.0 -net tap,fd=58,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:8 -vga cirrus

and trimmed it down to

sudo /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp 1 -name newtuna -boot c -drive file=/mnt/slices/newtuna.qcow2,if=virtio,index=0,boot=on,format=qcow2 -vnc 127.0.0.1:8 -monitor stdio -vga cirrus

This command ran, and I was able to connect with vncviewer. All I got was a blank screen, but I did not get any high cpu usage for the thread on the server node. Then i figured out that monitor starts in paused status, so I did c for continue, and verified the status was running. vnc continued to show blank screen, and thread showed high cpu use.

then I tried your next suggestion...
    qemu-img create win1.img 20G
    kvm -hda win1.img -cdrom win2003*.iso -m 1G -boot d

my actual line was:
    sudo kvm -hda win1.img -cdrom win2k3.iso -m 1G -boot d
and the returned the error "Could not initialize SDL - exiting"
next I did
    sudo kvm -hda win1.img -cdrom win2k3.iso -m 1G -boot d -vga cirrus
still got the same error.

so next I tried:
     sudo /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 1G -hda win1.img -cdrom win2k3.iso -vnc :10 -monitor stdio -vga cirrus
this worked, I was able to connect with vnc and go through the win2k3 installer. The installer goes through text mode, then reboots at which point i have to reestablish the vnc connection, then goes into GUI install mode, setup admin and pass, then finally finishing installation. And reboot...it worked, windows came up okay. So now that I know that works... how to figure out why it doesnt work when I was trying to install the way I usually do with libvirt.

[Steven Wagner (stevenwagner)]

Okay, so it appears to be a problem with something that is saved in the image. I can use the full cmd and it runs...

sudo /usr/bin/kvm -M pc-0.12 -enable-kvm -m 1024 -smp 1 -name newtuna -uuid 21a3c860-05b0-5dcf-cc8c-9243604f8e9b -localtime -boot c -drive file=win1.img -drive file=/mnt/isos/win2003/CRMEVOL_EN.ISO,if=ide,media=cdrom,index=2,format=raw -drive file=/mnt/isos/virtio-win-1.1.11-0.iso,if=ide,media=cdrom,index=3,format=raw -drive file=/mnt/isos/viostor-31-03-2010-floppy.img,if=floppy,index=0,format=raw -net nic,macaddr=52:54:00:68:23:96,vlan=0,model=virtio,name=virtio.0 -net tap,fd=58,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -usbdevice tablet -vnc :10 -vga cirrus

if I substitute
-drive file=win1.img
with
-drive file=/mnt/slices/newtuna.qcow2,if=virtio,index=0,boot=on,format=qcow2

it no longer works.

[Steven Wagner (stevenwagner)]

I tried to do another win2k3 install, this time via libvirt xml file. I used a minimal config. The install was immediately running very slow, possibly due to using a qcow disk file and/or not having used any virtio config. It ran so slow though with the disk io wait, that it brought the load average way up on the server node, and it caused ALL of my other linux virt instances to get CPU Stuck msgs, forcing me to reload the entire virtualization node. Bad. With all these issues, im going to have to go with windows virtualization does not work on ubuntu lucid. It used to work on ubuntu hardy.

[Serge Hallyn (serge-hallyn)]

>, and thread showed high cpu use.
>
>then I tried your next suggestion...
> qemu-img create win1.img 20G
> kvm -hda win1.img -cdrom win2003*.iso -m 1G -boot d
>
>my actual line was:
> sudo kvm -hda win1.img -cdrom win2k3.iso -m 1G -boot d
>and the returned the error "Could not initialize SDL - exiting"
>next I did
> sudo kvm -hda win1.img -cdrom win2k3.iso -m 1G -boot d -vga cirrus
>still got the same error.

Ok, I see, so you're doing this remotely.

>if I substitute
>-drive file=win1.img
>with
>-drive file=/mnt/slices/newtuna.qcow2,if=virtio,index=0,boot=on,format=qcow2
>
>it no longer works.

Where did you get your virtio drivers from (/mnt/isos/virtio-win-1.1.11-0.iso
and /mnt/isos/viostor-31-03-2010-floppy.img)? Those are pretty delicate,
so thus far it would seem we should look into
  1) the windows virtio drivers as your not-booting problem
  2) either the virtio drivers or qcow causing your slow install

[Steven Wagner (stevenwagner)]

It appears that writeback caching was turned off in qemu for qcow2 in June 2009 because qcow2 was "fast". Then a new patch was applied in July 2010 that made qcow2 slow again! I was able to solve my problems by turning on writeback cache.

<driver name='qemu' type='qcow2' cache='writeback'/>

However, this creates a huge migration issue for anyone upgrading from Ubuntu Server 8.04 LTS to 10.04 LTS. If they use qcow2 files with qemu-kvm, their virtualization guests will fail and start having disk IO failures if they are not aware that they now need to turn on writeback caching manually.

see links about qcow2 changes:

https://lists.ubuntu.com/archives/ubuntu-server/2010-March/003910.html

http://us.generation-nt.com/answer/bug-594069-qemu-kvm-really-poor-write-performance-qcow2-hdd-image-help-199995311.html

[Serge Hallyn (serge-hallyn)]

Marking wontfix - though perhaps changing the default in Ubuntu would make sense (as far as I know the performance impact is still there)... I'm open to the suggestion.