guest disk accesses lead to ATA errors + host vcpu0 unhandled wrmsr/rdmsr

[Christoph Anton Mitterer]

Hi.

This is from http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=717724.

Using Debian sid with 1.5.0-5 my Linux VMs (also Debian sid) are broken. When they boot I get gazillions of ATA errors inside the guest, as well as:
[ 242.479951] kvm [7790]: vcpu0 unhandled rdmsr: 0x345
[ 242.483683] kvm [7790]: vcpu0 unhandled wrmsr: 0x680 data 0
[ 242.483687] kvm [7790]: vcpu0 unhandled wrmsr: 0x6c0 data 0
[ 242.483689] kvm [7790]: vcpu0 unhandled wrmsr: 0x681 data 0
[ 242.483691] kvm [7790]: vcpu0 unhandled wrmsr: 0x6c1 data 0
[ 242.483693] kvm [7790]: vcpu0 unhandled wrmsr: 0x682 data 0
[ 242.483696] kvm [7790]: vcpu0 unhandled wrmsr: 0x6c2 data 0
[ 242.483698] kvm [7790]: vcpu0 unhandled wrmsr: 0x683 data 0
[ 242.483700] kvm [7790]: vcpu0 unhandled wrmsr: 0x6c3 data 0
[ 242.483702] kvm [7790]: vcpu0 unhandled wrmsr: 0x684 data 0
[ 242.483704] kvm [7790]: vcpu0 unhandled wrmsr: 0x6c4 data 0
[ 242.988307] kvm [7790]: vcpu0 unhandled rdmsr: 0xe8
[ 242.988312] kvm [7790]: vcpu0 unhandled rdmsr: 0xe7
[ 242.988314] kvm [7790]: vcpu0 unhandled rdmsr: 0xce
[ 242.988316] kvm [7790]: vcpu0 unhandled rdmsr: 0xce
[ 242.988318] kvm [7790]: vcpu0 unhandled rdmsr: 0x1ad
[ 242.988320] kvm [7790]: vcpu0 unhandled rdmsr: 0xce
[ 242.988322] kvm [7790]: vcpu0 unhandled rdmsr: 0xe8
[ 242.988324] kvm [7790]: vcpu0 unhandled rdmsr: 0xe7
[ 242.988598] kvm [7790]: vcpu0 unhandled rdmsr: 0xce

in the host.

Please have a look at the Debian bug, for screenshots and more info.
The problem didn't occur in 1.5.0-4 and there were basically no changes inside the VM (i.e. no kernel upgrade or so).

Thanks,
Chris.

[Michael Tokarev (mjt+launchpad-tls)]

For the record, the difference between debian qemu 1.5.0-4 and 1.5.0-5 is the 1.5.1 patch - I applied it without actually changing the "upstream" version number.

[Christoph Anton Mitterer (calestyo)]

Hi.

Finally I was able to get it running with IDE instead of SATA.
That works,... fine... no ATA errors.

As you already expected,.. I still get the:
[20424.793207] kvm [20947]: vcpu0 unhandled wrmsr: 0x6c1 data 0
[20424.793211] kvm [20947]: vcpu0 unhandled wrmsr: 0x682 data 0
[20424.793215] kvm [20947]: vcpu0 unhandled wrmsr: 0x6c2 data 0
[20424.793218] kvm [20947]: vcpu0 unhandled wrmsr: 0x683 data 0
[20424.793222] kvm [20947]: vcpu0 unhandled wrmsr: 0x6c3 data 0
[20424.793226] kvm [20947]: vcpu0 unhandled wrmsr: 0x684 data 0
[20424.793229] kvm [20947]: vcpu0 unhandled wrmsr: 0x6c4 data 0
[20425.946617] kvm [20947]: vcpu0 unhandled rdmsr: 0xe8
[20425.946623] kvm [20947]: vcpu0 unhandled rdmsr: 0xe7
[20425.946625] kvm [20947]: vcpu0 unhandled rdmsr: 0xce
[20425.946628] kvm [20947]: vcpu0 unhandled rdmsr: 0xce

So that seems to be unrelated... nevertheless.. I haven't seen these before with 1.5.0.

Anyway...I will try to get it running with virtio as well, as you suggested... but I have troubles to convince libvirt of using it.

btw: Might be unrealted... but with 1.5.1 and IDE now... when I do an update-initramfs -u inside the guest I gues these:
[ 276.130974] traps: ld-linux-x32.so[5642] general protection ip:f7788e8d sp:ffe12528 error:0 in ld-2.17.so[f7772000+21000]
[ 276.141129] traps: ld-linux-x32.so[5656] general protection ip:f77b8e8d sp:ff978458 error:0 in ld-2.17.so[f77a2000+21000]
[ 276.151444] traps: ld-linux-x32.so[5670] general protection ip:f7721e8d sp:ffd44cc8 error:0 in ld-2.17.so[f770b000+21000]
[ 276.207037] traps: ld-linux-x32.so[5714] general protection ip:f7770e8d sp:ff92cce8 error:0 in ld-2.17.so[f775a000+21000]
[ 276.278651] traps: ld-linux-x32.so[5775] general protection ip:f76fbe8d sp:ffe73048 error:0 in ld-2.17.so[f76e5000+21000]
[ 276.319277] traps: ld-linux-x32.so[5805] general protection ip:f76fee8d sp:fff486f8 error:0 in ld-2.17.so[f76e8000+21000]
[ 276.329552] traps: ld-linux-x32.so[5819] general protection ip:f77a8e8d sp:ffc9b448 error:0 in ld-2.17.so[f7792000+21000]
[ 276.338671] traps: ld-linux-x32.so[5833] general protection ip:f77bfe8d sp:ffe597b8 error:0 in ld-2.17.so[f77a9000+21000]
[ 276.347835] traps: ld-linux-x32.so[5847] general protection ip:f77b9e8d sp:ffcb1f08 error:0 in ld-2.17.so[f77a3000+21000]
[ 276.356548] traps: ld-linux-x32.so[5861] general protection ip:f77d1e8d sp:ffee16d8 error:0 in ld-2.17.so[f77bb000+21000]
[ 313.662052] do_general_protection: 24 callbacks suppressed
[ 313.662371] traps: ld-linux-x32.so[8926] general protection ip:f77e9e8d sp:ffa443a8 error:0 in ld-2.17.so[f77d3000+21000]
[ 313.670615] traps: ld-linux-x32.so[8940] general protection ip:f7796e8d sp:ff879cf8 error:0 in ld-2.17.so[f7780000+21000]
[ 313.677952] traps: ld-linux-x32.so[8954] general protection ip:f7791e8d sp:ffa25948 error:0 in ld-2.17.so[f777b000+21000]
[ 313.708061] traps: ld-linux-x32.so[8998] general protection ip:f7724e8d sp:ffe6b858 error:0 in ld-2.17.so[f770e000+21000]
[ 313.750866] traps: ld-linux-x32.so[9059] general protection ip:f77c7e8d sp:ff8a32a8 error:0 in ld-2.17.so[f77b1000+21000]
[ 313.771839] traps: ld-linux-x32.so[9089] general protection ip:f77b0e8d sp:ffdfa038 error:0...

[Christoph Anton Mitterer (calestyo)]

Also, when using SATA for the virtual disk,... it seems that it takes much longer till it get's to the point where it (the initrd) tries to mount root.

Till that point I see no errors... but then, after some time hanging at the root mount... it starts with the ATA errors.

[Michael Tokarev (mjt+launchpad-tls)]

So, what changed so you were able to boot from IDE without errors? As far as I understand, your initial prob was I/O errors on IDE (virtual) drive, but now you can boot it without errors. Or was it always SATA not IDE?

The rdmsr/wrmsr are unrelated, I already told you that, it is accessing optional CPU features which has nothing to do with disk access.

The segfaults during mkinitrd are unrelated too _unless_ you're running it from "failing" virtual drive. If you run it from failing virtual drive, it is remotely possible to have errors due to some file not read properly due to errors, but even this is a very remote possibility.

The time it takes to start mounting root is most likely also unrelated, it depends on the guest.

It _looks_ like the whole prob is due to FLUSH CASHE command not working correctly with SATA and IDE, but you confused me here - I don't understand where you were and are getting errors and where you don't.

Besides, you never answered to this: do you have enough free space in the host filesystem where your disk image(s) are located?

[Christoph Anton Mitterer (calestyo)]

Sorry for the confusion.
The initial problem was _always_ with SATA... I couldn't try IDE at first, as the guest initrd haven't had and the IDE drivers.

Yesterday then, I tried IDE and it worked with that.

Sure, I just wanted to confirm that you were really right ;-)

Sorry I have missed the question about enough space before.. there is plenty of free host space left (133 GB).

[Christoph Anton Mitterer (calestyo)]

Having tried now with virtio as well: works

So:
virtio, ide = OK
sata = broken

[Michael Tokarev (mjt+launchpad-tls)]

The fix is in a62eaa26c1d6d48fbdc3ac1d32bd1314f5fdc8c9 "ahci: Fix FLUSH command", should be in 1.6 and 1.5.3.

[Christoph Anton Mitterer (calestyo)]

From a first test of the Debian package in experimental, I'd say that the issue has been fixed :)
Thanks a lot for all your help!