KVM machines became corrupted after reboot (i386 only)

[Tomasz Nowak]

Binary package hint: qemu-kvm

Hi,
  I've just upgraded my Linux to Kubuntu 9.10 (i386).
  I'm using KVM and virt-manager on it.
  Unfortunately my VM boxes became corrupted after I install new software on these or create some files.

  Here is the VM definition:

$ virsh dumpxml ubuntu
Connecting to uri: qemu:///system
<domain type='kvm' id='27'>
  <name>ubuntu</name>
  <uuid>00000000-0000-4000-0000-111111111111</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='i686' machine='pc-0.11'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <source file='/dev/mapper/vm-ubuntu'/>
      <target dev='hda' bus='ide'/>
    </disk>
    <disk type='file' device='cdrom'>
      <source file='/home/tomnowak/Desktop/Software/linux/ubuntu/ubuntu-9.10-server-i386.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
    </disk>
    <interface type='network'>
      <mac address='00:50:56:11:11:11'/>
      <source network='default'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5911' autoport='no' listen='127.0.0.1'/>
    <video>
      <model type='vga' vram='8192' heads='1'/>
    </video>
  </devices>
  <seclabel type='dynamic' model='apparmor'>
    <label>libvirt-00000000-0000-4000-0000-111111111111</label>
    <imagelabel>libvirt-00000000-0000-4000-0000-111111111111</imagelabel>
  </seclabel>
</domain>

I'm trying to install Ubuntu Server Edition 9.10 but after a few reset there is a problem with swap partition on the screen while booting:

>>>
One or more of the mounts listed in /etc/fstab cannot yet be mounted:
(ESC for recovery shell)
swap: waiting for UUID=903d9649-0587-45b1-a501-8c02d71f43fc
<<<

Then when trying to recreate swap on /dev/sda5 I get:

>>>
root@ubuntu:~# fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000dd054

   Device Boot Start End Blocks Id System
/dev/sda1 * 1 1243 9984366 83 Linux
/dev/sda2 1244 1305 498015 5 Extended
/dev/sda5 1244 1305 497983+ 82 Linux swap / Solaris
root@ubuntu:~# swapon -a
swapon: cannot find the device for UUID=903d9649-0587-45b1-a501-8c02d71f43fc
root@ubuntu:~# swapon /dev/sda5
swapon: /dev/sda5: read swap header failed: Invalid argument
root@ubuntu:~# mkswap /dev/sda5
Setting up swapspace version 1, size = 497976 KiB
no label, UUID=38153005-9024-4eee-b1b9-99c3b0cfee62
root@ubuntu:~# swapon /dev/sda5
swapon: /dev/sda5: read swap header failed: Invalid argument
<<<<

Later I tried to install some software:
# apt-get install azureus
and during the package installation I saw strange characters on the screen and everything stopped working.
After a reset system was dead - the grub stopped booting - it seems my entire virtual disk became totally corrupted.
>>>
GRUB loading.
error: invalid arch independent ELF magic
grub rescue>
<<<

The above message is almost every time different

I've tried different disk types: ide, scsi, virtio - always the same.
Also the same when trying to install Kubuntu 9.10 under KVM.

[Aslak Knutsen (aslak-4fs)]

Upgrading a Jaunty vm to Karmic works for me, but a new Karmic install fails with the same error.

At some point I manage to update fstab after a mkswap && swapon and it worked on reboot. But after a reboot of the host the problem was back.

[Maciek Sujkowski (maciejus)]

I'm afraid I have the same problem but on windows xp vm on Karmic kvm i386.
VM installs correctly and seems to work correctly however when I map physical partition into vm and try to read files from that partition then I notice that most of the files are corrupted however they are correct when read under the host OS so this "fake corruption" is caused somehow by VM. If you need more details just let me know.

[Bitti (bitaround)]

Your hda should be:

<disk type='block' device='disk'>
 <source dev='/dev/vm/ubuntu'/>
 <target dev='hda' bus='ide'/>
</disk>

There is an example:
http://libvirt.org/drvqemu.html

[Tomasz Nowak (tomnowak)]

I've checked that and it doesn't make any difference.
I did a fresh Ubuntu 9.10 install on virtual machine and after reboot I've installed firefox on it and the filesystem became corrupted.
Even if I don't use raw partitions I have the same issue.
Now I really thinking about giving up with qemu-kvm as I lost a lot of time with it on 9.10.
Maybe it is time to return to VirtualBox as it always working well for me.
I wonder if x86_64 Karmic 9.10 has the same issue.

[Tomasz Nowak (tomnowak)]

I can't reproduce the issue on x86_64 karmic version.
So it seems it is only 32-bit related issue.

[Dustin Kirkland  (kirkland)]

Updating the title to note that this affecting 32-bit only.

Can anyone reproduce this problem on Lucid kvm? I'm marking incomplete for now. Please mark confirmed if you can reproduce on Lucid. Thanks!

[David Ayers (ayers)]

Yes this bug also affects Lucid beta2. I'll attempt to add a screenshot of what happens after installing Lucid B2 within Lucid B2 on i386.

[Scott Moser (smoser)]

David, can you please provide your kvm installation command line, and also what type of disk format you're using (ie, from qemu-create). If you're using from libvirt, virsh dumpxml output would be nice.

[David Ayers (ayers)]

Hello Scott,

After upgrading the host to the latest packages I was now able to install Lucid-Beta2 and run it afterwards.
I must admit that I'm now unsure if I remembered to upgrade after the initial Installation since I re installed a few times due to grub issues when installing on a USB drive yet fiddling with the grub installation of the internal drive (but I'll open separate report for that.

For the record though:
I used whatever the default for virt-manager is:
Applications->System Tools->Virtual Machine Manager
(here I had to create a new domain "localhost (QEMU)" due to missing privileges in the "localhost (QEMU Usermode)" domain)
mouse right click->New
Name: Lucid
* Local install media (ISO image or CDROM)
Use ISO Image: <Browse to nfs mounted ubuntu-10.04-beta2-desktop.i386.iso image>
OS type: Linux
Version: Ubuntu 10.04 (Lucid Lynx)
RAM 512
CPU 1
* Enable storage for this machine
 * create a disk image on the computer's hard drive
20.0 GB
 * Allocate entire disk now
Finish
- Open vm-console to install Lucid in VM
  (the Installation was done in German in Austrian locale)
using full vda block device.
... (continue default installation)

I've attached the virsh dumpxml Lucid > Lucid.xml output anyway.

Thanks for looking into the issue!

[Anders Olsson (anders-anderso)]

I have also encountered what seems to be data corruption with a karmic i386 kvm host after rebooting (actually I stopped the guest, rebooted the host and now I can't start the guest again). I get a kernel panic because of a bad ELF header. The guest is also karmic i386. It has worked fine for a while, I have done maybe 10 reboots earlier with this guest. I can provide more details if requested.

Are there any workarounds? Do I have a chance of saving my guest somehow by running an fsck or similar on the image?

[Anders Olsson (anders-anderso)]

Actually I remember the disk being full so the host did not boot correctly until I made some space available. This is the most likely reason for the corruption of the guest, though it shouldn't happen anyway.

[David Ayers (ayers)]

I have now upgraded my production system to lucid i686 2.6.32-23-generic-pae and I was able to install a new VM again.
This issue is no longer affecting me.

[Serge Hallyn (serge-hallyn)]

Per comment #12, I will mark this fix released.