xen domU don't boot 2.6.39+

[Mark Dokter]

2.6.39 boots fine on debian-unstable and gentoo, but on natty and oneiric I have to use linux-image-2.6.38 because all flavours of linux-image-2.6.39 i tried (generic, server, virtual and the 3.0rc1 from kernel ppa) just freeze.
Unfortunately I can't provide any helpful error messages and the screenshot I made isn't particularly interesting either.
---
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 2011-08-03 16:23 seq
 crw-rw---- 1 root audio 116, 33 2011-08-03 16:23 timer
AplayDevices: Error: [Errno 2] No such file or directory
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: Error: [Errno 2] No such file or directory
CurrentDmesg: [ 18.890095] eth0: no IPv6 routers present
DistroRelease: Ubuntu 11.10
HibernationDevice: RESUME=UUID=ad28d075-5294-4b0a-b1f3-7fa64487f51c
InstallationMedia: Ubuntu-Server 10.10 "Maverick Meerkat" - Release amd64 (20101007)
IwConfig: Error: [Errno 2] No such file or directory
Lsusb: Error: command ['lsusb'] failed with exit code 1: unable to initialize libusb: -99
MachineType: Xen HVM domU
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-2.6.38-8-server root=/dev/mapper/data-root ro console=tty1 console=ttyS0
ProcVersionSignature: Ubuntu 2.6.38-8.42-server 2.6.38.2
RelatedPackageVersions:
 linux-restricted-modules-2.6.38-8-server N/A
 linux-backports-modules-2.6.38-8-server N/A
 linux-firmware 1.56
RfKill: Error: [Errno 2] No such file or directory
Tags: oneiric
Uname: Linux 2.6.38-8-server x86_64
UpgradeStatus: Upgraded to oneiric on 2011-05-31 (63 days ago)
UserGroups:

dmi.bios.date: 07/24/2011
dmi.bios.vendor: Xen
dmi.bios.version: 4.1.1
dmi.chassis.type: 1
dmi.chassis.vendor: Xen
dmi.modalias: dmi:bvnXen:bvr4.1.1:bd07/24/2011:svnXen:pnHVMdomU:pvr4.1.1:cvnXen:ct1:cvr:
dmi.product.name: HVM domU
dmi.product.version: 4.1.1
dmi.sys.vendor: Xen

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in dianosing the problem. From a terminal window please run:

apport-collect 791364

and then change the status of the bug back to 'New'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Mark Dokter (dokter)]

apport information

[Mark Dokter (dokter)]

apport information

[Mark Dokter (dokter)]

apport information

[Mark Dokter (dokter)]

apport information

[Mark Dokter (dokter)]

apport information

[Mark Dokter (dokter)]

apport information

[Mark Dokter (dokter)]

apport information

[Mark Dokter (dokter)]

apport information

[Mark Dokter (dokter)]

apport information

[Mark Dokter (dokter)]

I updated the domU and tried to boot the current linux-image-3.0.0-7-server which unfortunately did not succeed.
The update and the apport-collect messages were done by booting linux-image-2.6.38-8-server.
When booting 3.0.0-7, the machine hangs with the following output:

[...udev error repeated lots of times...]
udevd[85]: timeout: killing '/sbin/modprobe -bv acpi:PNP0700:' [147]

udevd[85]: timeout: killing '/sbin/modprobe -bv acpi:PNP0700:' [147]

[ 723.020104] INFO: task modprobe:147 blocked for more than 120 seconds.
[ 723.023759] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 723.031340] modprobe D ffffffff81805120 0 147 85 0x00000004
[ 723.038795] ffff88001e1c3ce8 0000000000000086 ffff88001e1c3cd8 ffffffff8139f02c
[ 723.047501] ffff88001e1c3fd8 ffff88001e1c3fd8 ffff88001e1c3fd8 0000000000012a40
[ 723.056203] ffff88001ca08000 ffff88001e2516f0 000000000000632c 7fffffffffffffff
[ 723.064872] Call Trace:
[ 723.067572] [<ffffffff8139f02c>] ? vt_console_print+0x2bc/0x3a0
[ 723.073365] [<ffffffff815faca5>] schedule_timeout+0x2a5/0x320
[ 723.079017] [<ffffffff810329a9>] ? default_spin_lock_flags+0x9/0x10
[ 723.085111] [<ffffffff810869b2>] ? up+0x32/0x50
[ 723.089672] [<ffffffff810329a9>] ? default_spin_lock_flags+0x9/0x10
[ 723.095784] [<ffffffff815f9cff>] wait_for_common+0xdf/0x180
[ 723.101280] [<ffffffff81057330>] ? try_to_wake_up+0x200/0x200
[ 723.106916] [<ffffffff815f9e7d>] wait_for_completion+0x1d/0x20
[ 723.112634] [<ffffffff8107c9ad>] flush_work_sync+0x4d/0x60
[ 723.118044] [<ffffffff81078d80>] ? do_work_for_cpu+0x30/0x30
[ 723.123635] [<ffffffffa003bba8>] floppy_init+0x52f/0x5ce [floppy]
[ 723.129568] [<ffffffffa003b000>] ? 0xffffffffa003afff
[ 723.134624] [<ffffffffa003bc47>] ? floppy_init+0x5ce/0x5ce [floppy]
[ 723.140754] [<ffffffffa003bc95>] floppy_module_init+0x4e/0x3b9 [floppy]
[ 723.147166] [<ffffffff81002042>] do_one_initcall+0x42/0x180
[ 723.152675] [<ffffffff8109f3de>] sys_init_module+0xbe/0x230
[ 723.158176] [<ffffffff81604882>] system_call_fastpath+0x16/0x1b
[ 723.164015] INFO: task sh:187 blocked for more than 120 seconds.
[ 723.169820] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 723.177396] sh D ffffffff81805120 0 187 1 0x00000000
[ 723.184897] ffff88001e2e18a8 0000000000000086 0000000000000000 ffff88001f7fcc00
[ 723.193594] ffff88001e2e1fd8 ffff88001e2e1fd8 ffff88001e2e1fd8 0000000000012a40
[ 723.202263] ffff88001ca08000 ffff88001e2e96f0 0000000000000002 7fffffffffffffff
[ 723.210953] Call Trace:
[ 723.213620] [<ffffffff815faca5>] schedule_timeout+0x2a5/0x320
[ 723.219237] [<ffffffff8115d040>] ? __mem_cgroup_commit_charge+0x70/0xc0
[ 723.225642] [<ffffffff810329a9>] ? default_spin_lock_flags+0x9/0x10
[ 723.231740] [<ffffffff815f9cff>] wait_for_common+0xdf/0x180
[ 723.237218] [<ffffffff81057330>] ? try_to_wake_up+0x200/0x200
[ 723.242867] [<ffffffff815f9e7d>] wait_for_completion+0x1d/0x20
[ 723.248592] [<ffffffff8107ca4e>] flush_work+0x2e/0x40
[ 723.253636] [<ffffffff81078d80>] ? do_work_for_cpu+0x30/0x30
[ 723.259184] [<ffffffff81392f95>] tty_flu...

[Mark Dokter (dokter)]

The dom0 used is ubuntu oneiric with kernel 3.0.0-7-server and xen 4.1.1 from https://launchpad.net/~bderzhavets/+archive/xen-hg

[Mark Dokter (dokter)]

Situation is still the same with:
Dom0 as of today with xen-hypervisor-4.1-amd64 4.1.1-2ubuntu1 and linux-headers-3.0.0-9-server 3.0.0-9.15

oneiric hvm domU with 1 CPU and 1GB RAM, linux-image-server 3.0.0-9.15 or linux-image-generic 3.1.0-rc4 don't boot.
I've added kernel parameters debug loglevel=8 initcall_debug and these are the last messages after which only rcu_sched_state detected stall on CPU0 messages are printed:

[ 10.048487] Freeing unused kernel memory: 1376k freed
Loading, please wait...
[ 10.110763] udevd[81]: starting version 173
Begin: Loading essential drivers ... [ 10.145811] calling platform_pci_module_init+0x0/0x1000 [xen_platform_pci] @ 118
[ 10.154464] xen: --> irq=28, pirq=22
[ 10.158913] xen-platform-pci 0000:00:03.0: PCI INT A -> GSI 28 (level, low) -> IRQ 28
[ 10.170597] Grant table initialized
[ 10.204926] initcall platform_pci_module_init+0x0/0x1000 [xen_platform_pci] returned 0 after 49400 usecs
[ 10.226935] calling xenbus_probe_frontend_init+0x0/0x1000 [xenbus_probe_frontend] @ 132
[ 10.248089] initcall xenbus_probe_frontend_init+0x0/0x1000 [xenbus_probe_frontend] returned 0 after 11533 usecs
[ 10.269725] calling netif_init+0x0/0x1000 [xen_netfront] @ 132
[ 10.276463] Initialising Xen virtual ethernet driver.
[ 10.304325] initcall netif_init+0x0/0x1000 [xen_netfront] returned 0 after 27187 usecs
[ 10.331757] calling xlblk_init+0x0/0x1000 [xen_blkfront] @ 147
[ 10.349331] blkfront device/vbd/51712 num-ring-pages 1 nr_ents 32.
[ 10.364719] initcall xlblk_init+0x0/0x1000 [xen_blkfront] returned 0 after 26133 usecs
done.
Begin: Running /scripts/init-premount ... [ 70.390060] INFO: rcu_sched_state detected stall on CPU 0 (t=6000 jiffies)
[ 250.690061] INFO: rcu_sched_state detected stall on CPU 0 (t=24030 jiffies)

BTW: A debian domU with linux-image 3.0.0-1 boots fine.

[Mark Dokter (dokter)]

Installed linux-image-3.0.0-10 on domU and dom0. The hvm domU now boots fine.
The oneiric beta1 dvd image is based on linux-image-3.0.0-9 and therefore hangs after the kernel is booted (same behaviour as described above). So the cure for the malfunction must be somewhere in the changelog of linux-image from 3.0.0-9 to 3.0.0-10 and ubuntu should be installable from beta2 or daily image in a hvm domU.

[Brad Figg (brad-figg)]

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel currently in the release pocket than the one you tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

Thank you for your help.

[Driss Boudrar (corleone)]

I've got this bug too !

[Brad Figg (brad-figg)]

Thank you for taking the time to file a bug report on this issue.

However, given the number of bugs that the Kernel Team receives during any development cycle it is impossible for us to review them all. Therefore, we occasionally resort to using automated bots to request further testing. This is such a request.

We have noted that there is a newer version of the development kernel than the one you last tested when this issue was found. Please test again with the newer kernel and indicate in the bug if this issue still exists or not.

If the bug still exists, change the bug status from Incomplete to Confirmed. If the bug no longer exists, change the bug status from Incomplete to Fix Released.

Thank you for your help, we really do appreciate it.

[Mark Dokter (dokter)]

As I wrote before, this bug should be gone since linux-image-3.0.0-10. So you might try an oneiric release install medium. Be aware though, that oneiric release has problems when you try to install in hvm mode, since the xen-platform-pci seems to be missing and therefore you won't see any disks or ethernet devices once those are unplugged by xen.
Your options are:
- upgrading from earlier versions and putting the xen-platrofm-pci, xen-blkfront, xen-netfront modules in the initial ramdisk.
- boot from the oneiric install medium with kernel parameter xen_emul_unplug=never, install from there with the qemu emulated devices, put the modules mentioned above in your initrd and then boot without the parameter to have pv-on-hvm functionality.