virtio: trying to map MMIO memory

[Rick Vernam]

Qemu host is Core i7, running Linux. Guest is Windows XP sp3.
Often, qemu will crash shortly after starting (1-5 minutes) with a statement "qemu-system-x86_64: virtio: trying to map MMIO memory"
This has occured with qemu-kvm 0.14, qemu-kvm 0.14.1, qemu-0.15.0-rc0 and qemu 0.15.0-rc1.
Qemu is started as such:
qemu-system-x86_64 -cpu host -enable-kvm -pidfile /home/rick/qemu/hds/wxp.pid -drive file=/home/rick/qemu/hds/wxp.raw,if=virtio -m 768 -name WinXP -net nic,model=virtio -net user -localtime -usb -vga qxl -device virtio-serial -chardev spicevmc,name=vdagent,id=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice port=1234,disable-ticketing -daemonize -monitor telnet:localhost:12341,server,nowait
The WXP guest has virtio 1.1.16 drivers for net and scsi, and the most current spice binaries from spice-space.org.

[Stefan Hajnoczi (stefanha)]

On Sun, Jul 31, 2011 at 12:01 AM, Rick Vernam <email address hidden> wrote:
> Public bug reported:
>
> Qemu host is Core i7, running Linux.  Guest is Windows XP sp3.
> Often, qemu will crash shortly after starting (1-5 minutes) with a statement "qemu-system-x86_64: virtio: trying to map MMIO memory"
> This has occured with qemu-kvm 0.14, qemu-kvm 0.14.1, qemu-0.15.0-rc0 and qemu 0.15.0-rc1.
> Qemu is started as such:
> qemu-system-x86_64 -cpu host -enable-kvm -pidfile /home/rick/qemu/hds/wxp.pid -drive file=/home/rick/qemu/hds/wxp.raw,if=virtio -m 768 -name WinXP -net nic,model=virtio -net user -localtime -usb -vga qxl -device virtio-serial -chardev spicevmc,name=vdagent,id=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice port=1234,disable-ticketing -daemonize -monitor telnet:localhost:12341,server,nowait
> The WXP guest has virtio 1.1.16 drivers for net and scsi, and the most current spice binaries from spice-space.org.

This is probably a guest virtio driver bug.

Vadim: Any known issues like this with 1.1.16?

Stefan

[vrozenfe (vrozenfe)]

On Sun, 2011-07-31 at 18:54 +0100, Stefan Hajnoczi wrote:
> On Sun, Jul 31, 2011 at 12:01 AM, Rick Vernam <email address hidden> wrote:
> > Public bug reported:
> >
> > Qemu host is Core i7, running Linux. Guest is Windows XP sp3.
> > Often, qemu will crash shortly after starting (1-5 minutes) with a statement "qemu-system-x86_64: virtio: trying to map MMIO memory"
> > This has occured with qemu-kvm 0.14, qemu-kvm 0.14.1, qemu-0.15.0-rc0 and qemu 0.15.0-rc1.
> > Qemu is started as such:
> > qemu-system-x86_64 -cpu host -enable-kvm -pidfile /home/rick/qemu/hds/wxp.pid -drive file=/home/rick/qemu/hds/wxp.raw,if=virtio -m 768 -name WinXP -net nic,model=virtio -net user -localtime -usb -vga qxl -device virtio-serial -chardev spicevmc,name=vdagent,id=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice port=1234,disable-ticketing -daemonize -monitor telnet:localhost:12341,server,nowait
> > The WXP guest has virtio 1.1.16 drivers for net and scsi, and the most current spice binaries from spice-space.org.
>
> This is probably a guest virtio driver bug.
>
> Vadim: Any known issues like this with 1.1.16?
No, it something new to me.
Will try to reproduce and fix it.
Thank you,
Vadim.
>
> Stefan

[Rick Vernam (rickv-hobi)]

Seems to only crash the first time qemu is started after booting the host machine.
After the first crash, qemu will run solid for days if the host machine is not rebooted.
If I have an opportunity, I'll test if it also crashes after first start when kvm and/or kvm_intel modules are unloaded and reloaded.

[David Rando (david-rando)]

I have the same problem. I'm using the packages from the Sergei ppa with spice enabled on a server with 9 windows xp machines and 1 linux (ubuntu 10.04) one. Ubuntu is rock solid and never crash, but the windows machines do randomnly. I've updated everything i could (using the version from spice-space.org), i've disabled the memory ballooning, disabled spice etc... but it's always the same. It just crash with that message.

I guess it's a driver problem. I've searched on google and found some clues (there's a guy porting spice drivers to BSD and got that problem and could resolve it).

If i can do a test that helps, here i am. I've tried a few things but i'm out of ideas.

With my test, the only thing that all the machines had always enabled is the virtio storage driver. I've tried disabling the network one and the memory one but no luck, so maybe it's related to the harddisk controller. (i can't disable it because windows doesn't like to mess with the hard disk controller, you know).

Thanks a lot ;-)

[Rick Vernam (rickv-hobi)]

Continues to occur with recently updated qxl, vdagent & virtio serial windows binaries from spice-space.org.
Also continues with qemu-kvm-0.15.0-rc1, qemu-0.15.0-rc1 & qemu-0.15.0-rc2

[Rick Vernam (rickv-hobi)]

Vadim,

Have you been able to reproduce this?
Do you require any additional information?

Thanks,
-Rick

[Rick Vernam (rickv-hobi)]

Continues with Qemu 0.15.0 and Qemu-KVM 0.15.0

[David Rando (david-rando)]

It's something related to Windows. I have in the same machine a linux server working with spice enabled and is rock solid. The windows machines crash with that error randomnly.

[Rick Vernam (rickv-hobi)]

So that would point to virtio. This appears to be the place for virtio bugs, correct?
Should I be doing anything to help usher this along?

[Stefan Hajnoczi (stefanha)]

On Sun, Aug 14, 2011 at 7:11 AM, Rick Vernam <email address hidden> wrote:
> So that would point to virtio.  This appears to be the place for virtio bugs, correct?
> Should I be doing anything to help usher this along?

Either we need to help Vadim reproduce this so he can take a look.
Vadim: were you able to reproduce this?

Or someone interested in Windows driver debugging can see if they can
debug this. The symptom is that the guest driver is placing invalid
descriptors into the vring. QEMU tries to map the memory and finds
the address is in a memory-mapped I/O region instead of a RAM region.
Normally the vring descriptors only point into RAM, so this is a guest
driver bug where the vring is being corrupted somehow. If anyone
wants to take a look I can try to help guide them along the
virtio-specifics.

Stefan

[David Rando (david-rando)]

Do you know if it's something related to the virtio net driver? anyone tried going to the e1000 only? i have some machines with e1000 and some of them with virtio-net, but i have crash no matter what driver is using (but the virtio driver is installed anyway, despite i'm using the e1000).

I was searching in the git repository for windows drivers, (http://git.kernel.org/?p=virt/kvm/kvm-guest-drivers-windows.git;a=history;f=NetKVM/Common/ndis56common.h;hb=HEAD ) but couldn't find anything related to this.

Any news? i can't debug the driver, i would do it if i knew how.

David Rando.

[Stefan Hajnoczi (stefanha)]

On Thu, Aug 25, 2011 at 3:53 PM, David Rando <email address hidden> wrote:
> Do you know if it's something related to the virtio net driver? anyone
> tried going to the e1000 only? i have some machines with e1000 and some
> of them with virtio-net, but i have crash no matter what driver is using
> (but the virtio driver is installed anyway, despite i'm using the
> e1000).
>
> I was searching in the git repository for windows drivers,
> (http://git.kernel.org/?p=virt/kvm/kvm-guest-drivers-
> windows.git;a=history;f=NetKVM/Common/ndis56common.h;hb=HEAD ) but
> couldn't find anything related to this.
>
> Any news? i can't debug the driver, i would do it if i knew how.
>
> David Rando.
>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/818673
>
> Title:
>  virtio: trying to map MMIO memory
>
> Status in QEMU:
>  New
>
> Bug description:
>  Qemu host is Core i7, running Linux.  Guest is Windows XP sp3.
>  Often, qemu will crash shortly after starting (1-5 minutes) with a statement "qemu-system-x86_64: virtio: trying to map MMIO memory"
>  This has occured with qemu-kvm 0.14, qemu-kvm 0.14.1, qemu-0.15.0-rc0 and qemu 0.15.0-rc1.
>  Qemu is started as such:
>  qemu-system-x86_64 -cpu host -enable-kvm -pidfile /home/rick/qemu/hds/wxp.pid -drive file=/home/rick/qemu/hds/wxp.raw,if=virtio -m 768 -name WinXP -net nic,model=virtio -net user -localtime -usb -vga qxl -device virtio-serial -chardev spicevmc,name=vdagent,id=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice port=1234,disable-ticketing -daemonize -monitor telnet:localhost:12341,server,nowait
>  The WXP guest has virtio 1.1.16 drivers for net and scsi, and the most current spice binaries from spice-space.org.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/818673/+subscriptions

Vadim,
Have you had any luck reproducing this issue or any advice for David?

Thanks,
Stefan

[vrozenfe (vrozenfe)]

On Thu, 2011-08-25 at 16:54 +0100, Stefan Hajnoczi wrote:
> On Thu, Aug 25, 2011 at 3:53 PM, David Rando <email address hidden> wrote:
> > Do you know if it's something related to the virtio net driver? anyone
> > tried going to the e1000 only? i have some machines with e1000 and some
> > of them with virtio-net, but i have crash no matter what driver is using
> > (but the virtio driver is installed anyway, despite i'm using the
> > e1000).
> >
> > I was searching in the git repository for windows drivers,
> > (http://git.kernel.org/?p=virt/kvm/kvm-guest-drivers-
> > windows.git;a=history;f=NetKVM/Common/ndis56common.h;hb=HEAD ) but
> > couldn't find anything related to this.
> >
> > Any news? i can't debug the driver, i would do it if i knew how.
> >
> > David Rando.
> >
> > --
> > You received this bug notification because you are a member of qemu-
> > devel-ml, which is subscribed to QEMU.
> > https://bugs.launchpad.net/bugs/818673
> >
> > Title:
> > virtio: trying to map MMIO memory
> >
> > Status in QEMU:
> > New
> >
> > Bug description:
> > Qemu host is Core i7, running Linux. Guest is Windows XP sp3.
> > Often, qemu will crash shortly after starting (1-5 minutes) with a statement "qemu-system-x86_64: virtio: trying to map MMIO memory"
> > This has occured with qemu-kvm 0.14, qemu-kvm 0.14.1, qemu-0.15.0-rc0 and qemu 0.15.0-rc1.
> > Qemu is started as such:
> > qemu-system-x86_64 -cpu host -enable-kvm -pidfile /home/rick/qemu/hds/wxp.pid -drive file=/home/rick/qemu/hds/wxp.raw,if=virtio -m 768 -name WinXP -net nic,model=virtio -net user -localtime -usb -vga qxl -device virtio-serial -chardev spicevmc,name=vdagent,id=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -spice port=1234,disable-ticketing -daemonize -monitor telnet:localhost:12341,server,nowait
> > The WXP guest has virtio 1.1.16 drivers for net and scsi, and the most current spice binaries from spice-space.org.
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/qemu/+bug/818673/+subscriptions
>
> Vadim,
> Have you had any luck reproducing this issue or any advice for David?
Guys, I'm sorry. Not yet. I'll try my best to trace this problem on the
following weekend.

Best,
Vadim.

>
> Thanks,
> Stefan

[dx-vmonroig (dx-vmonroig)]

We're affected by this bug, too. Trying to find a workaround, last friday we changed VGA model to cirrus, and the machine is working properly without new entries in log.

[Rick Vernam (rickv-hobi)]

ping...

I understand that Vadim must be very busy that he can't look at this - I can relate.
But is there really only one person in all of Qemu and/or Spice who can be addressed to look into this?

So that I can plan around the viability of Qemu for my users, I need to know if the technologies I seek will be well maintained.

Thanks,
-Rick

[vrozenfe (vrozenfe)]

I've made several unsuccessful attempts to reproduce this problem,
running VMs on top of F14 and RHEL6.2

The only one relatively close problem, reported by our QE, was
https://bugzilla.redhat.com/show_bug.cgi?id=727034
It must be fixed in our internal repository. (Public repository
is out of sync, but I'm going to update it soon)

I put our recent (WHQL candidates) drivers here:
http://people.redhat.com/vrozenfe/virtio-win-prewhql-0.1.zip

Please give them a try and share your experience.

Best,
Vadim.

[Rick Vernam (rickv-hobi)]

Still crashes just the same.
I updated the drivers for virt net, scsi & serial from the XP and WXp folders in the zip file that you referenced.
Then I shutdown the VM.
Because it only seems to happen every other time that Qemu is started, I started it back up and shut it down again.
Then the VM was started a third time and left idle prior to crashing.

Thanks, and sorry that I didn't have better news.
(also, note that I've built qemu-kvm straight from www.linux-kvm.org, and qemu straight from qemu.org).

-Rick

[Rick Vernam (rickv-hobi)]

so if I use -vga std instead of -vga qxl (and of course take out the -spice stuff), I don't crash.
perhaps this is spice/qxl related?

[Rick Vernam (rickv-hobi)]

sorry, scratch that last about -vga std ... it still crashed just the same using -vga std.

[vrozenfe (vrozenfe)]

Thank you, Rick.

Could you help me to narrow this problem down?

As I see, you have three virtio drivers installed on your system - block, net, and virtio serial.
Technically, anyone of them can create "trying to map MMIO memory" problem.
The best way to find a buggy driver ( or drivers) will be to isolate one from the other.
If you can, please try running only one virtio device every time to see which driver
sends incorrect scatter/gather list element to QEMU.

Another question. You said, the problem happens after every second or third restart.
Do you shutdown your VM, or just restart it? How does it work after going through
several hibernate/resume, and/or suspend/resume cycles.

Best regards,
Vadim.

[Rick Vernam (rickv-hobi)]

On Wednesday 14 September 2011 14:42:09 vrozenfe wrote:
> Thank you, Rick.
>
> Could you help me to narrow this problem down?
Absolutely.

>
> As I see, you have three virtio drivers installed on your system - block,
> net, and virtio serial. Technically, anyone of them can create "trying to
> map MMIO memory" problem. The best way to find a buggy driver ( or
> drivers) will be to isolate one from the other. If you can, please try
> running only one virtio device every time to see which driver sends
> incorrect scatter/gather list element to QEMU.
Sure, no problem. I'll have that in the next few days.

>
> Another question. You said, the problem happens after every second or third
> restart. Do you shutdown your VM, or just restart it?
Have to shut down the VM guest so that the qemu process exits.

> How does it work
> after going through several hibernate/resume, and/or suspend/resume
> cycles.
I often will suspend with or without pausing qemu (via monitor commands 'stop'
and 'cont'). I have never experienced any problem with the qemu process that
was running prior to the suspend.

>
> Best regards,
> Vadim.

Thanks,
-Rik

[Rick Vernam (rickv-hobi)]

On Wednesday 14 September 2011 16:30:11 Rick Vernam wrote:
> On Wednesday 14 September 2011 14:42:09 vrozenfe wrote:
> > Thank you, Rick.
> >
> > Could you help me to narrow this problem down?
>
> Absolutely.
>
> > As I see, you have three virtio drivers installed on your system - block,
> > net, and virtio serial. Technically, anyone of them can create "trying to
> > map MMIO memory" problem. The best way to find a buggy driver ( or
> > drivers) will be to isolate one from the other. If you can, please try
> > running only one virtio device every time to see which driver sends
> > incorrect scatter/gather list element to QEMU.
>
> Sure, no problem. I'll have that in the next few days.
I started qemu without any of the virt-serial stuff, specfically:
qemu-system-x86_64 -cpu host -enable-kvm -pidfile /home/rick/qemu/hds/wxp.pid -
drive file=/home/rick/qemu/hds/wxp.raw,if=virtio,aio=native -m 1536 -name WinXP
-net nic,model=virtio -net user -localtime -usb -vga qxl -spice
port=1234,disable-ticketing -monitor stdio

It's been running for around 2 hours and no crash yet.

Thanks,
-Rick

>
> > Another question. You said, the problem happens after every second or
> > third restart. Do you shutdown your VM, or just restart it?
>
> Have to shut down the VM guest so that the qemu process exits.
>
> > How does it work
> > after going through several hibernate/resume, and/or suspend/resume
> > cycles.
>
> I often will suspend with or without pausing qemu (via monitor commands
> 'stop' and 'cont'). I have never experienced any problem with the qemu
> process that was running prior to the suspend.
>
> > Best regards,
> > Vadim.
>
> Thanks,
> -Rik

[Rick Vernam (rickv-hobi)]

On Thursday 15 September 2011 11:23:53 Rick Vernam wrote:
> On Wednesday 14 September 2011 16:30:11 Rick Vernam wrote:
> > On Wednesday 14 September 2011 14:42:09 vrozenfe wrote:
> > > Thank you, Rick.
> > >
> > > Could you help me to narrow this problem down?
> >
> > Absolutely.
> >
> > > As I see, you have three virtio drivers installed on your system -
> > > block, net, and virtio serial. Technically, anyone of them can create
> > > "trying to map MMIO memory" problem. The best way to find a buggy
> > > driver ( or drivers) will be to isolate one from the other. If you
> > > can, please try running only one virtio device every time to see which
> > > driver sends incorrect scatter/gather list element to QEMU.
> >
> > Sure, no problem. I'll have that in the next few days.
>
> I started qemu without any of the virt-serial stuff, specfically:
> qemu-system-x86_64 -cpu host -enable-kvm -pidfile
> /home/rick/qemu/hds/wxp.pid - drive
> file=/home/rick/qemu/hds/wxp.raw,if=virtio,aio=native -m 1536 -name WinXP
> -net nic,model=virtio -net user -localtime -usb -vga qxl -spice
> port=1234,disable-ticketing -monitor stdio
>
> It's been running for around 2 hours and no crash yet.
So without virt-serial, the machine ran until I rebooted the guest OS, then
crashed with the same error message. Without virt-serial it seemed to be
stable so long as it was just left running.

Now I'll run it without virt-net, and let you know how that goes.

>
> Thanks,
> -Rick
>
> > > Another question. You said, the problem happens after every second or
> > > third restart. Do you shutdown your VM, or just restart it?
> >
> > Have to shut down the VM guest so that the qemu process exits.
> >
> > > How does it work
> > > after going through several hibernate/resume, and/or suspend/resume
> > > cycles.
> >
> > I often will suspend with or without pausing qemu (via monitor commands
> > 'stop' and 'cont'). I have never experienced any problem with the qemu
> > process that was running prior to the suspend.
> >
> > > Best regards,
> > > Vadim.
> >
> > Thanks,
> > -Rik

[vrozenfe (vrozenfe)]

Thank you, Rick.

I will start checking virtio-serial driver tomorrow.

Best,
Vadim.

[Rick Vernam (rickv-hobi)]

On Friday 16 September 2011 03:52:34 hkran wrote:
[snip]
>
> I have tried many times with many restarts or shutdown-and-boot xp guest
> but failed to meet the crashing.
> (I am using the virtio drivers referenced in the earlier mail list.)
> my command:
>
> /home/huikai/qemu15/bin/qemu --enable-kvm -m 768 -drive
> file=/home/huikai/winxp_dev.img,if=virtio -net nic,model=virtio -net
> user -usb -usbdevice tablet -localtime -vga qxl -device virtio-serial
> -chardev spicevmc,name=vdagent,id=vdagent -device
> virtserialport,chardev=vdagent,name=spice0 -spice
> port=1234,disable-ticketing -monitor telnet:localhost:12341,server,nowait

Okay, I tried a variation of that:
qemu-system-x86_64 -cpu host -enable-kvm -m 1536 -pidfile
/home/rick/qemu/hds/wxp.pid -drive file=/home/rick/qemu/hds/wxp.raw,if=virtio
-net nic,model=virtio -net user -localtime -usb -vga qxl -device virtio-serial
-chardev spicevmc,name=vdagent,id=vdagent -device
virtserialport,chardev=vdagent,name=spice0 -spice port=1234,disable-ticketing
-monitor telnet:localhost:12341,server,nowait

And it's been running stable all day.
The differences between the command line that crashes and yours are:
- yours doesn't have "aio=native" in the -drive declaration.
- yours has some differences in the virtio-serial device declaration.
- yours has some differences in the virtserialport device declaration.

As time permits I'm going to try each of those differences individually.

Thanks,
-Rick

[Rick Vernam (rickv-hobi)]

On Friday 16 September 2011 12:42:02 Rick Vernam wrote:
> On Friday 16 September 2011 03:52:34 hkran wrote:
> [snip]
>
> > I have tried many times with many restarts or shutdown-and-boot xp guest
> > but failed to meet the crashing.
> > (I am using the virtio drivers referenced in the earlier mail list.)
> > my command:
> >
> > /home/huikai/qemu15/bin/qemu --enable-kvm -m 768 -drive
> > file=/home/huikai/winxp_dev.img,if=virtio -net nic,model=virtio -net
> > user -usb -usbdevice tablet -localtime -vga qxl -device virtio-serial
> > -chardev spicevmc,name=vdagent,id=vdagent -device
> > virtserialport,chardev=vdagent,name=spice0 -spice
> > port=1234,disable-ticketing -monitor
> > telnet:localhost:12341,server,nowait
>
> Okay, I tried a variation of that:
> qemu-system-x86_64 -cpu host -enable-kvm -m 1536 -pidfile
> /home/rick/qemu/hds/wxp.pid -drive
> file=/home/rick/qemu/hds/wxp.raw,if=virtio -net nic,model=virtio -net user
> -localtime -usb -vga qxl -device virtio-serial -chardev
> spicevmc,name=vdagent,id=vdagent -device
> virtserialport,chardev=vdagent,name=spice0 -spice
> port=1234,disable-ticketing -monitor telnet:localhost:12341,server,nowait
>
> And it's been running stable all day.
> The differences between the command line that crashes and yours are:
> - yours doesn't have "aio=native" in the -drive declaration.
> - yours has some differences in the virtio-serial device declaration.
> - yours has some differences in the virtserialport device declaration.

I added "aio=native" and it crashed.
If it helps, I ran config like so:
./configure --target-list=x86_64-softmmu --disable-curses --disable-curl --
audio-drv-list=alsa --audio-card-list=sb16,ac97,hda --enable-vnc-thread --
disable-bluez --enable-vhost-net --enable-spice

and I've attached config.log, as well as the output of configure.

Thanks,
-Rick

[Rick Vernam (rickv-hobi)]

On Friday 16 September 2011 12:42:02 Rick Vernam wrote:
> On Friday 16 September 2011 03:52:34 hkran wrote:
> [snip]
>
> > I have tried many times with many restarts or shutdown-and-boot xp guest
> > but failed to meet the crashing.
> > (I am using the virtio drivers referenced in the earlier mail list.)
> > my command:
> >
> > /home/huikai/qemu15/bin/qemu --enable-kvm -m 768 -drive
> > file=/home/huikai/winxp_dev.img,if=virtio -net nic,model=virtio -net
> > user -usb -usbdevice tablet -localtime -vga qxl -device virtio-serial
> > -chardev spicevmc,name=vdagent,id=vdagent -device
> > virtserialport,chardev=vdagent,name=spice0 -spice
> > port=1234,disable-ticketing -monitor
> > telnet:localhost:12341,server,nowait
>
> Okay, I tried a variation of that:
> qemu-system-x86_64 -cpu host -enable-kvm -m 1536 -pidfile
> /home/rick/qemu/hds/wxp.pid -drive
> file=/home/rick/qemu/hds/wxp.raw,if=virtio -net nic,model=virtio -net user
> -localtime -usb -vga qxl -device virtio-serial -chardev
> spicevmc,name=vdagent,id=vdagent -device
> virtserialport,chardev=vdagent,name=spice0 -spice
> port=1234,disable-ticketing -monitor telnet:localhost:12341,server,nowait
>
> And it's been running stable all day.
> The differences between the command line that crashes and yours are:
> - yours doesn't have "aio=native" in the -drive declaration.
> - yours has some differences in the virtio-serial device declaration.
> - yours has some differences in the virtserialport device declaration.
>
> As time permits I'm going to try each of those differences individually.
Without "aio=native" ...
in the definition of virtserialport, I changed "name=spice0" to
"name=com.redhat.spice.0" - with this change, the guest vdagent works, but it
crashed...

>
> Thanks,
> -Rick

[Rick Vernam (rickv-hobi)]

On Friday 23 September 2011 14:07:17 Alon Levy wrote:
> On Thu, Sep 22, 2011 at 02:10:04PM -0500, Rick Vernam wrote:
> > On Friday 16 September 2011 12:42:02 Rick Vernam wrote:
> > > On Friday 16 September 2011 03:52:34 hkran wrote:
> > > [snip]
> > >
> > > > I have tried many times with many restarts or shutdown-and-boot xp
> > > > guest but failed to meet the crashing.
> > > > (I am using the virtio drivers referenced in the earlier mail list.)
> > > > my command:
> > > >
> > > > /home/huikai/qemu15/bin/qemu --enable-kvm -m 768 -drive
> > > > file=/home/huikai/winxp_dev.img,if=virtio -net nic,model=virtio -net
> > > > user -usb -usbdevice tablet -localtime -vga qxl -device
> > > > virtio-serial -chardev spicevmc,name=vdagent,id=vdagent -device
> > > > virtserialport,chardev=vdagent,name=spice0 -spice
> > > > port=1234,disable-ticketing -monitor
> > > > telnet:localhost:12341,server,nowait
> > >
> > > Okay, I tried a variation of that:
> > > qemu-system-x86_64 -cpu host -enable-kvm -m 1536 -pidfile
> > > /home/rick/qemu/hds/wxp.pid -drive
> > > file=/home/rick/qemu/hds/wxp.raw,if=virtio -net nic,model=virtio -net
> > > user -localtime -usb -vga qxl -device virtio-serial -chardev
> > > spicevmc,name=vdagent,id=vdagent -device
> > > virtserialport,chardev=vdagent,name=spice0 -spice
> > > port=1234,disable-ticketing -monitor
> > > telnet:localhost:12341,server,nowait
> > >
> > > And it's been running stable all day.
> > > The differences between the command line that crashes and yours are:
> > > - yours doesn't have "aio=native" in the -drive declaration.
> > > - yours has some differences in the virtio-serial device declaration.
> > > - yours has some differences in the virtserialport device declaration.
> > >
> > > As time permits I'm going to try each of those differences
> > > individually.
> >
> > Without "aio=native" ...
> > in the definition of virtserialport, I changed "name=spice0" to
> > "name=com.redhat.spice.0" - with this change, the guest vdagent works,
> > but it crashed...
>
> If you provide details on the crash maybe someone can help.
This email thread has details early on the thread, and there is a bug report
here: https://bugs.launchpad.net/bugs/818673
All the details of the crash that are available to me are previously
described.

>
> > > Thanks,
> > > -Rick

[Rick Vernam (rickv-hobi)]

Vadim,

Did you see comment #27? Is that helpful, would you like any additional info? Are there other things you would like for me to try?

Thanks,
-Rick

[Rick Vernam (rickv-hobi)]

So I've built qemu with -enable-debug and tried running with an attached GDB, but got nothing.
I've never tried to debug Qemu before, but I know it's not quite a simple as debugging other apps.
I am honestly clueless about how to further debug this problem.

Should I give up on using virtio-serial for spice vdagent integration?
It seems that nobody really has any interest in this problem.

Are other people using qemu with spice (including functional guest agent support, copy/paste, etc)? How?
I know hkran posted how (s)he uses qemu without hitting this bug, but when I use qemu in that way, I loose guest agent.

I like to fix things myself, and I hate to be asking about this when everybody clearly has more interesting things to do.
I just need some input so that I can have realistic expectations.

Thanks,
-Rick

[Ian Pilcher (arequipeno)]

I've been dealing with this bug for some time on Fedora. Until recently, I was using the VirtIO drivers from RHEV 2.2, which don't suffer from this problem. As of Fedora 16, however, that isn't an option, because they cause the guest to blue-screen early in the boot process.

So ... I've been doing some more testing with the following setup:

  Host:
    Intel DQ67SW motherboard with Q67 chipset (including IOMMU)
    BIOS version SWQ6710H.86A.0050.2011.0401.1409 (release date 04/01/2011)
    Intel Core i7 2600, 4-cores, 8 threads, 3.4 GHz
    16GB memory
    Fedora 15 64-bit, fully updated including updates-testing repo
      qemu-kvm-0.14.0-8.fc15.x86_64
      libvirt-0.8.8-7.fc15.x86_64
      kernel-2.6.41.6-1.fc15.x86_64

  Guest:
    Windows 7 Professional 32-bit, fully updated
    2 VCPUs
    3.5GB memory
    Red Hat VirtIO Ethernet Adapter driver version 6.0.209.605 (9/20/2010)
    Red Hat VirtIO SCSI Controller driver version 6.0.0.10 (9/20/2010)
    (No VirtIO serial ports or channels defined)

(The VirtIO drivers are from http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/.)

I have determined that disabling the Intel IOMMU has no effect; the problem still occurs.

Perhaps more interestingly, it seems that the problem only occurs when I am using the VirtIO SCSI *and* the VirtIO Ethernet drivers. It seems that the problem does not occur if I only use one of the drivers; an IDE disk with a VirtIO NIC seems to be stable, as does a VirtIO disk with an e1000 NIC.

Now to the big question ... what the heck can be done to get this problem fixed? I hope that everyone agrees that it's totally unacceptable for a problem like this to sit unfixed for so long. I am more than willing to test any patches, enable
debugging, etc.; just tell me what to do.

Thanks!

[Ian Pilcher (arequipeno)]

Two other observations:

* The problem is also present in the latest drivers in the RHEL 6.2 virtio-win package (both driver versions 60.62.102.3000, dates 9/12/2011).
* The problem does not seem to occur if the guest has only 1 VCPU.

So the problem only occurs when using 2 VirtIO devices with 2 VCPUs. This leads me to speculate that there is some sort of VirtIO "core" that is shared between the 2 devices, and that there is some sort of race condition or locking problem in that core.

[Rick Vernam (rickv-hobi)]

In reply to comment #32, I encounter this problem with 1VCPU - see the original description of the bug.
Also note that after qemu quits with the error, the subsequent execution of the same qemu invocation will run stable.

[Vitalis (workyz)]

And I have this bug!
Linux test-2 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16 19:52:42 UTC 2010 x86_64 GNU/Linux
In container i have Windows XP SP3

In log:
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -cpu qemu32 -enable-kvm -m 2048 -sm
p 2 -name boss_xp -uuid 9041090d-acee-da4a-921d-238f2a43be64 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/boss_xp.monitor,server,nowait -monitor cha
rdev:monitor -localtime -boot c -drive file=/root/boss_xp.qcow2,if=virtio,index=0,boot=on,format=raw,cache=none -drive file=/home/admino/virtio-win-1.1.16.is
o,if=ide,media=cdrom,index=2,format=raw -net nic,macaddr=00:16:36:49:83:6c,vlan=0,model=virtio,name=virtio.0 -net tap,fd=43,vlan=0,name=tap.0 -chardev pty,id
=serial0 -serial chardev:serial0 -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:2 -k en-us -vga cirrus
char device redirected to /dev/pts/3
pci_add_option_rom: failed to find romfile "pxe-virtio.bin"
virtio_ioport_write: unexpected address 0x13 value 0x1
virtio: trying to map MMIO memory

After this Windows go shutdown.
I tried this (in log):

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -cpu qemu32 -enable-kvm -m 2048 -sm
p 2 -name boss_xp -uuid 9041090d-acee-da4a-921d-238f2a43be64 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/boss_xp.monitor,server,nowait -monitor cha
rdev:monitor -localtime -boot c -drive file=/root/boss_xp.qcow2,if=virtio,index=0,boot=on,format=raw,cache=none -drive file=/home/admino/virtio-win-1.1.16.is
o,if=ide,media=cdrom,index=2,format=raw -net nic,macaddr=00:16:36:49:83:6c,vlan=0,model=rtl8139,name=rtl8139.0 -net tap,fd=43,vlan=0,name=tap.0 -chardev pty,
id=serial0 -serial chardev:serial0 -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:2 -k en-us -vga cirrus
char device redirected to /dev/pts/3
pci_add_option_rom: failed to find romfile "pxe-rtl8139.bin"
virtio: trying to map MMIO memory
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -cpu qemu32 -enable-kvm -m 2048 -sm
p 2 -name boss_xp -uuid 9041090d-acee-da4a-921d-238f2a43be64 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/boss_xp.monitor,server,nowait -monitor cha
rdev:monitor -localtime -boot c -drive file=/root/boss_xp.qcow2,if=virtio,index=0,boot=on,format=raw,cache=none -drive file=/home/admino/virtio-win-1.1.16.is
o,if=ide,media=cdrom,index=2,format=raw -net nic,macaddr=00:16:36:49:83:6c,vlan=0,model=rtl8139,name=rtl8139.0 -net tap,fd=43,vlan=0,name=tap.0 -chardev pty,
id=serial0 -serial chardev:serial0 -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:2 -k en-us -vga cirrus
char device redirected to /dev/pts/3
pci_add_option_rom: failed to find romfile "pxe-rtl8139.bin"
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -cpu qemu32 -enable-kvm -m 2048 -sm
p 1 -name boss_xp -uuid 9041090d-acee-da4a-921d-238f2a43be64 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/boss_xp.monitor,server,nowait -monitor cha
rdev:monitor -localtime -bo...

[Vitalis (workyz)]

And more: i have too more virtual PC with WindowsXP SP3 and with one CPU, but them doesnt have any problems. Maybe this bug depends on 2 and more CPU??

[Rick Vernam (rickv-hobi)]

I experience this on uni-processor.

On Tuesday 24 January 2012 16:48:04 Vitalis wrote:
> And more: i have too more virtual PC with WindowsXP SP3 and with one
> CPU, but them doesnt have any problems. Maybe this bug depends on 2 and
> more CPU??

[Mike Cao (vipmike007)]

Does this Bug similiar with https://bugzilla.redhat.com/show_bug.cgi?id=771390 ?

[Rick Vernam (rickv-hobi)]

Yes, I would say it is the same bug. I will test the driver that Vadim linked in Comment 33 (https://bugzilla.redhat.com/show_bug.cgi?id=771390#c33) and report back.

Thanks, Mike, for posting here.

[Rick Vernam (rickv-hobi)]

well, the link in the redhat bug, comment 33, is no good apparently. I will follow that bug, and test when I see Vadim has posted a new driver to test.

[Vitalis (workyz)]

Now have next config and bug still:
/usr/bin/kvm -S -M pc-0.12 -cpu qemu32 -enable-kvm -m 2048 -smp 1 -name boss_xp -uuid 9041090d-acee-da4a-921d-238f2a43be64 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/boss_xp.monitor,server,nowait -monitor chardev:monitor -localtime -boot c -drive file=/root/boss_xp.qcow2,if=virtio,index=0,boot=on,format=raw,cache=none -drive file=/home/admino/virtio-win-1.1.16.iso,if=ide,media=cdrom,index=2,format=raw -net nic,macaddr=00:16:36:49:83:6c,vlan=0,model=rtl8139,name=rtl8139.0 -net tap,fd=43,vlan=0,name=tap.0 -serial none -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:2 -k en-us -vga cirrus

[vrozenfe (vrozenfe)]

It was a long journey.
But now it seem like we've managed to fix this problem.
https://bugzilla.redhat.com/show_bug.cgi?id=771390#c45

I put new drivers here:
http://people.redhat.com/vrozenfe/vioscsi.vfd

Best regards,
Vadim.

[Vitalis (workyz)]

Thanks! Where can I get ISO of new drivers pack? for Ubuntu 10.04

[Alon Levy (alonlevy1)]

Vadim, Could this be related to the hangs during boot with qxl and virtio-serial in a single windows vm?

Alon

[vrozenfe (vrozenfe)]

I have no idea regarding Ubuntu, but you can find the new drivers
at Fedora project site
http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/virtio-win-0.1-22.iso

[vrozenfe (vrozenfe)]

Hi Alon,
Unfortunately, qxl and virtio-serial
hang is a different problem.

[Vitalis (workyz)]

Hello with bad news! I have:

virtio_ioport_write: unexpected address 0x13 value 0x1

on config:

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -cpu qemu32 -enable-kvm -m 3072 -smp 1 -name nata_xp -uuid da607499-1d8f-e7ef-d1d2-38
1c1839e4ba -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/nata_xp.monitor,server,nowait -monitor chardev:monitor -localtime -boot c -drive file=/root/nata_xp.qcow2,if=virtio,index=0,boot=on,format=raw
,cache=none -drive file=/home/admino/virtio-win-0.1-22.iso,if=ide,media=cdrom,index=2,format=raw -net nic,macaddr=00:16:36:06:02:69,vlan=0,model=virtio,name=virtio.0 -net tap,fd=43,vlan=0,name=tap.0 -serial
none -parallel none -usb -usbdevice tablet -vnc 127.0.0.1:3 -k en-us -vga cirrus
pci_add_option_rom: failed to find romfile "pxe-virtio.bin"

with kernel 2.6.32-40-generic #87-Ubuntu SMP Tue Mar 6 00:56:56 UTC 2012 x86_64 GNU/Linux
qemu drivers are virtio-win-0.1-22.iso

Anybody help me?

[Thomas Huth (th-huth)]

According to comment 41, this bug has been fixed, so I'm setting the status to "Fix released" now ... Vitalis, your problem from comment 46 sounds differently - if it still persists today, please open a new bug ticket for this instead.