kvm can not boot Ubuntu guest

[peterdv]

Binary package hint: kvm

Kvm fails to boot an Ubuntu 7.10 guest installed from ubuntu-7.10-alternate-i386.iso as a command line system.

The guest simply hangs after detecting the network card at the first reboot after what appeared to be a successful install.

This happens on a freshly installed host (from both ubuntu-7.10-alternate-amd64.iso and ubuntu-7.10-alternate-i386.iso)

The guest was installed using:

qemu -cdrom /root/mirror/ubuntu-7.10-alternate-i386.iso -boot d -no-acpi -m 256 -hda /srv/kvm/images/ubuntu-7.10-cmdline-i386.qcow2 -vnc :1 -k da -net nic,model=rtl8139,macaddr=00:11:22:33:44:a1,vlan=0 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup

and working around the cdrom detection problem using https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/120316/comments/7

Booting the guest using:

kvm -boot c -no-acpi -m 256 -hda /srv/kvm/images/ubuntu-7.10-cmdline-i386.qcow2 -vnc :1 -k da -net nic,model=rtl8139,macaddr=00:11:22:33:44:a1,vlan=0 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup

results in a boot sequence ending in (transcribed by hand from the vnc console):
Begin: Waiting for root file system ... ...
  SCSI subsystem initialized
  8139cp: 10/100 PCI Ethernet driver v1.3 (Mar 22, 2004)
  eth0: RTL-8139c+ at 0xd080e000, 00:11:22:33:44:a1, IRQ 11
  8839too Fast Ethernet driver 0.9.28

... and then just silence.

[peterdv (peter-dahl-vestergaard)]

I forgot to mention that I use kvm 1:28-4ubuntu2

[peterdv (peter-dahl-vestergaard)]

... patience. After quite a while, the guest consoles bails out into BusyBox.

apparently the root disk is not detected:

 ALERT! /dev/disk/by-uuid/468f57ac-f59f-49af-8421-3b0a31ed7ee1 does not exist. Dropping to a shell !

Also I notice that the host console reports:

 kvm: unhandled wrmsr: 0xc1

[peterdv (peter-dahl-vestergaard)]

Googeling for " kvm: unhandled wrmsr: 0xc1" gives a possibly related hit over at the kvm-devel list:
  https://sourceforge.net/tracker/?func=detail&atid=893831&aid=1769884&group_id=180599

This bug reports failure to boot linux guest with 2.6.22 kernel.

Seems similar, I'm using
  Linux vm02 2.6.22-14-generic #1 SMP Tue Dec 18 08:02:57 UTC 2007 i686 GNU/Linux
The bug report suggest a workaround adding nolapic to the guest kernel. Does not work for me.

0xc1 is appears to be related to a performance counter: http://<email address hidden>/msg05404.html

I notice the latest release is 1:59+dfsg-0ubuntu5 having binary packages available for Ubuntu Hardy only. Any chance for binary packages for 7.10 ?

[Oleksij Rempel (olerem)]

> Any chance for binary packages for 7.10 ?

Not really. kvm-59 is designed and included in kernel 2.6.24.

[Oleksij Rempel (olerem)]

If you need blending age kvm with latest bugs and bugfixes, try to use git source:
kernel git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm.git
userspace git://git.kernel.org/pub/scm/linux/kernel/git/avi/kvm-userspace.git

[Oleksij Rempel (olerem)]

> The bug report suggest a workaround adding nolapic to the guest kernel. Does not work for me.

you should use "nmi_watchdog=0" instead

[Oleksij Rempel (olerem)]

Here is the fix/workaround relased for linux kernel. This can be included to linux-source-2.6.22
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=57c22f49e4c7d27a03e109a317c30b12f1bae0b2

[peterdv (peter-dahl-vestergaard)]

Unfortunately "nmi_watchdog=0" did not work for me.

Thanks for the link in comment 7.

I see my options as either build a custom (guest) kernel or move to Hardy. I am testing for a potential production environment, so I am slightly hesitant moving to Hardy (yet), Since I happen to have the opportunity in terms of available hardware right now, I do however think that I will give it a try (tomorrow).

However the problem still remains for the current stable Ubuntu release.

[Oleksij Rempel (olerem)]

Can you please attach your cpuinfo.

[peterdv (peter-dahl-vestergaard)]

I am testing on a Dell PowerEdge 2950, 4 G ram, lvm2 on top on the Dell raid.

Output from cat /proc/cpuinfo attached.

[peterdv (peter-dahl-vestergaard)]

FYI: it appears that linux-source-2.6.22-14.47 already includes the patch mentioned in comment 7. I shall try to find the time to build this source tomorrow.

[Oleksij Rempel (olerem)]

And please attach debug.log from kvm, you can get it with:
kvm test.img -serial file:debug.log

and add to grub:
kernel /vmlinuz .... earlycon=ttyS0 console=ttyS0

[peterdv (peter-dahl-vestergaard)]

Console output from the guest booting as per comment 12.

Thanks for the recipe, it would have taken me some time to dig out these options from the doc.

[Oleksij Rempel (olerem)]

Hmm.. Seems like this is not kvm issue. ata_piix modules was not loaded
so it can't find you hdd( image ).

[Oleksij Rempel (olerem)]

So far i can't reproduce the issue you have... But i use newest qemu
version.

[peterdv (peter-dahl-vestergaard)]

On Jan 17, 2008 3:52 PM, fishor <email address hidden> wrote:
> Hmm.. Seems like this is not kvm issue. ata_piix modules was not loaded
> so it can't find you hdd( image ).

OK, this rings a bell. As I mentioned at the very top, I was "working
around the cdrom detection problem using
https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.22/+bug/120316/comments/7"

This means that prior to booting the guest install image, I added
"ata_piix.blacklist=yes" to the kernel options - and proceeded with a
normal command line installation.

As I was able to see the hd during installation I possibly wrongly
assumed that the driver was available, I did not take the time to
investigate the actual semantics of this option ... does this make
sense ?

"Peter Dahl Vestergaard" <email address hidden>

[Oleksij Rempel (olerem)]

You can see in debug.log options used to start installed kernel. But
there is no "ata_piix.blacklist=yes", so it was blacklisted some where
in /etc/modules.d/.. or ???

try to run "grep -R "ata_piix" /etc/*"

[peterdv (peter-dahl-vestergaard)]

On Jan 17, 2008 6:35 PM, fishor <email address hidden> wrote:
> You can see in debug.log options used to start installed kernel. But
> there is no "ata_piix.blacklist=yes", so it was blacklisted some where
> in /etc/modules.d/.. or ???

The debug.log is from the first reboot after the install.
Unfortunately I am physically quite a distance from the server, so I
can not test today.

But the persistent work by Aaron Waite over at Bug 19749
(https://bugs.launchpad.net/ubuntu/+source/linux-source-2.6.15/+bug/19749/comments/89)
seems to indicate that:

"""
Section "5.2.1.3. Blacklisting kernel modules", you can read:

"You can blacklist a module using the following syntax:
module_name.blacklist=yes. This will cause the module to be
blacklisted in /etc/modprobe.d/blacklist.local both during the
installation and for the installed system.

Note that a module may still be loaded by the installation system
itself. You can prevent that from happening by running the
installation in expert mode and unselecting the module from the list
of modules displayed during the hardware detection phases."
"""

--
"Peter Dahl Vestergaard" <email address hidden>

[Oleksij Rempel (olerem)]

The rease why ata_piix module not loaded described here:
http://<email address hidden>/msg10716.html

Probably this mean you _need_to_add_ ata_piix.blacklist=yes to your bootloader... becouse this module will not work with this kvm/qemu version. Instead use old ide_piix.

So now we are at the same place were it begon.
This issue affect qemu/kvm and kernel.. this all was fixed in nevest versions.

[peterdv (peter-dahl-vestergaard)]

I can confirm that using a clean install from hardy-alternate-alpha-3-i386.iso as host OS fixes the problem(s) mentioned above.
7.10 boots, installs, and runs cleanly (without the cdrom detection problem and without problems with ata_piix).

To summarize:

1) Recompiling a custom kernel for the host OS from linux-source-2.6.22-14.47 (which includes the patch mentioned in comment 7) did not help in my case.
2) Juggling kernel modules at various stages of the boot and install process (comments 16 to 18) never really worked for me. Quite likely my fault - it should be possible from what I read in the various problem reports.

It appears to me that the source of my problems are fairly well understood and documented here and in the bugs and mails referenced. And they are fixed in Hardy.

Although I do not need the latest bells and whistles, I will continue my test based on Hardy as host OS. Unfortunately I do not think I personally can contribute directly to a fix on 7.10 (and April is not that far off, so maybe it is not worth the effort).

Thank You for Your patience and effort in illuminating this problem, and pinpointing the root cause.

[Launchpad Janitor (janitor)]

Beginning with the Hardy Heron 8.04 development cycle, all open Ubuntu kernel bugs need to be reported against the "linux" kernel package. We are automatically migrating this bug to the new "linux" package. However, development has already began for the upcoming Intrepid Ibex 8.10 release. It would be helpful if you could test the upcoming release and verify if this is still an issue - http://www.ubuntu.com/testing . If the issue still exists, please update this report by changing the Status of the "linux" task from "Incomplete" to "New". We appreciate your patience and understanding as we make this transition. Thanks!

[Leann Ogasawara (leannogasawara)]

The Ubuntu Kernel Team is planning to move to the 2.6.27 kernel for the upcoming Intrepid Ibex 8.10 release. As a result, the kernel team would appreciate it if you could please test this newer 2.6.27 Ubuntu kernel. There are one of two ways you should be able to test:

1) If you are comfortable installing packages on your own, the linux-image-2.6.27-* package is currently available for you to install and test.

--or--

2) The upcoming Alpha5 for Intrepid Ibex 8.10 will contain this newer 2.6.27 Ubuntu kernel. Alpha5 is set to be released Thursday Sept 4. Please watch http://www.ubuntu.com/testing for Alpha5 to be announced. You should then be able to test via a LiveCD.

Please let us know immediately if this newer 2.6.27 kernel resolves the bug reported here or if the issue remains. More importantly, please open a new bug report for each new bug/regression introduced by the 2.6.27 kernel and tag the bug report with 'linux-2.6.27'. Also, please specifically note if the issue does or does not appear in the 2.6.26 kernel. Thanks again, we really appreicate your help and feedback.