Seabios causing hang when adding a virtio disk

[Richard W.M. Jones]

Attaching a completely blank disk image to a virtual machine causes the following stack trace when loading the virtio block driver:

[ 1.106728] loop: module loaded
[ 1.125680] vda: unknown partition table
[ 1.789721] Switching to clocksource tsc
[ 8.373409] Clocksource tsc unstable (delta = 87849991 ns)
[ 8.374642] Switching to clocksource jiffies
[ 241.037694] INFO: task swapper/0:1 blocked for more than 120 seconds.
[ 241.037966] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.038424] swapper/0 D ffffffff81806240 0 1 0 0x00000000
[ 241.039028] ffff88001ed7d870 0000000000000046 0000000000000000 000000004dbb1e54
[ 241.039460] ffff88001ed7dfd8 ffff88001ed7dfd8 ffff88001ed7dfd8 0000000000013780
[ 241.039839] ffffffff81c0d020 ffff88001ed80000 ffff88001ed7d850 ffff88001f014040
[ 241.040220] Call Trace:
[ 241.041349] [<ffffffff811157a0>] ? __lock_page+0x70/0x70
[ 241.041712] [<ffffffff8165574f>] schedule+0x3f/0x60
[ 241.041798] [<ffffffff816557ff>] io_schedule+0x8f/0xd0
[ 241.041943] [<ffffffff811157ae>] sleep_on_page+0xe/0x20
[ 241.045386] [<ffffffff81655eca>] __wait_on_bit_lock+0x5a/0xc0
[ 241.045467] [<ffffffff81115797>] __lock_page+0x67/0x70
[ 241.049105] [<ffffffff81089e60>] ? autoremove_wake_function+0x40/0x40
[ 241.049416] [<ffffffff81116a20>] do_read_cache_page+0x160/0x180
[ 241.049656] [<ffffffff811ad630>] ? blkdev_write_begin+0x30/0x30
[ 241.049783] [<ffffffff81116a89>] read_cache_page_async+0x19/0x20
[ 241.049931] [<ffffffff81116a9e>] read_cache_page+0xe/0x20
[ 241.053417] [<ffffffff811e14bd>] read_dev_sector+0x2d/0x90
[ 241.053629] [<ffffffff811e2604>] adfspart_check_ICS+0x74/0x2d0
[ 241.053834] [<ffffffff81311f64>] ? snprintf+0x34/0x40
[ 241.053987] [<ffffffff811e2590>] ? rescan_partitions+0x300/0x300
[ 241.054210] [<ffffffff811e1c38>] check_partition+0xf8/0x200
[ 241.057501] [<ffffffff811e236a>] rescan_partitions+0xda/0x300
[ 241.057746] [<ffffffff811ae68c>] __blkdev_get+0x2bc/0x420
[ 241.057888] [<ffffffff810899f7>] ? bit_waitqueue+0x17/0xc0
[ 241.058000] [<ffffffff811ae84e>] blkdev_get+0x5e/0x1e0
[ 241.058080] [<ffffffff812f6b62>] register_disk+0x162/0x180
[ 241.058177] [<ffffffff812f6c24>] add_disk+0xa4/0x1b0
[ 241.061902] [<ffffffff8162925a>] virtblk_probe+0x43d/0x4e2
[ 241.064195] [<ffffffff814083f0>] ? virtblk_config_changed+0x30/0x30
[ 241.065436] [<ffffffff813a3020>] ? vp_find_vqs+0xc0/0xc0
[ 241.065529] [<ffffffff813a15e3>] virtio_dev_probe+0xe3/0x140
[ 241.065617] [<ffffffff813f08d8>] really_probe+0x68/0x190
[ 241.065701] [<ffffffff813f0b65>] driver_probe_device+0x45/0x70
[ 241.065789] [<ffffffff813f0c3b>] __driver_attach+0xab/0xb0
[ 241.065873] [<ffffffff813f0b90>] ? driver_probe_device+0x70/0x70
[ 241.065962] [<ffffffff813f0b90>] ? driver_probe_device+0x70/0x70
[ 241.066051] [<ffffffff813ef9cc>] bus_for_each_dev+0x5c/0x90
[ 241.066137] [<ffffffff813f069e>] driver_attach+0x1e/0x20
[ 241.069422] [<ffffffff813f02f0>] bus_add_driver+0x1a0/0x270
[ 241.073426] [<ffffffff81d2fcb8>] ? loop_init+0x12f/0x12f
[ 241.073641] [<ffffffff813f11a6>] driver_register+0x76/0x140
[ 241.073801] [<ffffffff81d2fcb8>] ? loop_init+0x12f/0x12f
[ 241.073951] [<ffffffff813a1840>] register_virtio_driver+0x20/0x30
[ 241.077413] [<ffffffff81d2fd0a>] init+0x52/0x7c
[ 241.077611] [<ffffffff81002040>] do_one_initcall+0x40/0x180
[ 241.077760] [<ffffffff81cf9ce9>] kernel_init+0xcf/0x14e
[ 241.077913] [<ffffffff81661d74>] kernel_thread_helper+0x4/0x10
[ 241.078061] [<ffffffff81cf9c1a>] ? start_kernel+0x3c7/0x3c7
[ 241.081352] [<ffffffff81661d70>] ? gs_change+0x13/0x13
[ 361.089003] INFO: task swapper/0:1 blocked for more than 120 seconds.
[ 361.089312] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 361.090036] swapper/0 D ffffffff81806240 0 1 0 0x00000000
[ 361.090322] ffff88001ed7d870 0000000000000046 0000000000000000 000000004dbb1e54
[ 361.090716] ffff88001ed7dfd8 ffff88001ed7dfd8 ffff88001ed7dfd8 0000000000013780
[ 361.091024] ffffffff81c0d020 ffff88001ed80000 ffff88001ed7d850 ffff88001f014040
[ 361.091332] Call Trace:
[ 361.091570] [<ffffffff811157a0>] ? __lock_page+0x70/0x70
[ 361.091797] [<ffffffff8165574f>] schedule+0x3f/0x60
[ 361.091990] [<ffffffff816557ff>] io_schedule+0x8f/0xd0
[ 361.092189] [<ffffffff811157ae>] sleep_on_page+0xe/0x20
[ 361.092392] [<ffffffff81655eca>] __wait_on_bit_lock+0x5a/0xc0
[ 361.092702] [<ffffffff81115797>] __lock_page+0x67/0x70
[ 361.092852] [<ffffffff81089e60>] ? autoremove_wake_function+0x40/0x40
[ 361.092852] [<ffffffff81116a20>] do_read_cache_page+0x160/0x180
[ 361.092852] [<ffffffff811ad630>] ? blkdev_write_begin+0x30/0x30
[ 361.092945] [<ffffffff81116a89>] read_cache_page_async+0x19/0x20
[ 361.093390] [<ffffffff81116a9e>] read_cache_page+0xe/0x20
[ 361.093600] [<ffffffff811e14bd>] read_dev_sector+0x2d/0x90
[ 361.097020] [<ffffffff811e2604>] adfspart_check_ICS+0x74/0x2d0
[ 361.097194] [<ffffffff81311f64>] ? snprintf+0x34/0x40
[ 361.097328] [<ffffffff811e2590>] ? rescan_partitions+0x300/0x300
[ 361.097476] [<ffffffff811e1c38>] check_partition+0xf8/0x200
[ 361.097617] [<ffffffff811e236a>] rescan_partitions+0xda/0x300
[ 361.100852] [<ffffffff811ae68c>] __blkdev_get+0x2bc/0x420
[ 361.100956] [<ffffffff810899f7>] ? bit_waitqueue+0x17/0xc0
[ 361.101103] [<ffffffff811ae84e>] blkdev_get+0x5e/0x1e0
[ 361.101226] [<ffffffff812f6b62>] register_disk+0x162/0x180
[ 361.101328] [<ffffffff812f6c24>] add_disk+0xa4/0x1b0
[ 361.101461] [<ffffffff8162925a>] virtblk_probe+0x43d/0x4e2
[ 361.101677] [<ffffffff814083f0>] ? virtblk_config_changed+0x30/0x30
[ 361.108947] [<ffffffff813a3020>] ? vp_find_vqs+0xc0/0xc0
[ 361.109136] [<ffffffff813a15e3>] virtio_dev_probe+0xe3/0x140
[ 361.109238] [<ffffffff813f08d8>] really_probe+0x68/0x190
[ 361.109373] [<ffffffff813f0b65>] driver_probe_device+0x45/0x70
[ 361.109519] [<ffffffff813f0c3b>] __driver_attach+0xab/0xb0
[ 361.112917] [<ffffffff813f0b90>] ? driver_probe_device+0x70/0x70
[ 361.113131] [<ffffffff813f0b90>] ? driver_probe_device+0x70/0x70
[ 361.113290] [<ffffffff813ef9cc>] bus_for_each_dev+0x5c/0x90
[ 361.113604] [<ffffffff813f069e>] driver_attach+0x1e/0x20
[ 361.113733] [<ffffffff813f02f0>] bus_add_driver+0x1a0/0x270
[ 361.117064] [<ffffffff81d2fcb8>] ? loop_init+0x12f/0x12f
[ 361.117238] [<ffffffff813f11a6>] driver_register+0x76/0x140
[ 361.117398] [<ffffffff81d2fcb8>] ? loop_init+0x12f/0x12f
[ 361.117537] [<ffffffff813a1840>] register_virtio_driver+0x20/0x30
[ 361.117689] [<ffffffff81d2fd0a>] init+0x52/0x7c
[ 361.121097] [<ffffffff81002040>] do_one_initcall+0x40/0x180
[ 361.121325] [<ffffffff81cf9ce9>] kernel_init+0xcf/0x14e
[ 361.121538] [<ffffffff81661d74>] kernel_thread_helper+0x4/0x10
[ 361.121762] [<ffffffff81cf9c1a>] ? start_kernel+0x3c7/0x3c7
[ 361.124914] [<ffffffff81661d70>] ? gs_change+0x13/0x13

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Richard W.M. Jones (rich-annexia)]

Kernel version is:
Linux tmpubuntu1204 3.2.0-12-generic #21-Ubuntu SMP Tue Jan 31 18:48:57 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

This is Ubuntu 12.04 alpha 2.

[Fred van Zwieten (fvzwieten)]

This report is related to this bug: https://bugs.launchpad.net/ubuntu/+source/libguestfs/+bug/930634

[Joseph Salisbury (jsalisbury)]

Do you know if this issue happened in a previous version of Ubuntu, or is this a new issue?

Would it be possible for you to test the latest upstream kernel? It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v3.3 kernel[1] (Not a kernel in the daily directory). Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag(Only that one tag, please leave the other tags). This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text.

If this bug is fixed by the mainline kernel, please add the following tag 'kernel-fixed-upstream-KERNEL-VERSION'. For example, if kernel version 3.3-rc2 fixed the issue, the tag would be: 'kernel-fixed-upstream-v3.3-rc2'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

If you are unable to test the mainline kernel, for example it will not boot, please add the tag: 'kernel-unable-to-test-upstream'.
Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[1] http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-rc2-precise/

[Richard W.M. Jones (rich-annexia)]

This didn't happen on older versions of Ubuntu (eg. 11.10).

I installed the following kernel:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v3.3-rc3-precise/
and it *does* happen on this kernel so the flag should be
'kernel-bug-exists-upstream'.

[Richard W.M. Jones (rich-annexia)]

FYI is the call stack from the mainline kernel. It's basically the same
as for the precise kernel.

[ 241.283078] INFO: task swapper/0:1 blocked for more than 120 seconds.
[ 241.283078] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 241.283078] swapper/0 D ffffffff8180c6c0 0 1 0 0x00000000
[ 241.283078] ffff88001ed69880 0000000000000046 ffff88001ed69fd8 0000000000013600
[ 241.283078] ffff88001ed68010 0000000000013600 0000000000013600 0000000000013600
[ 241.287078] ffff88001ed69fd8 0000000000013600 ffffffff81c0d020 ffff88001ed60000
[ 241.291079] Call Trace:
[ 241.291079] [<ffffffff8111e090>] ? __lock_page+0x70/0x70
[ 241.291079] [<ffffffff8165e82f>] schedule+0x3f/0x60
[ 241.291079] [<ffffffff8165e8dc>] io_schedule+0x8c/0xd0
[ 241.291079] [<ffffffff8111e09e>] sleep_on_page+0xe/0x20
[ 241.291079] [<ffffffff8165cc0a>] __wait_on_bit_lock+0x5a/0xc0
[ 241.291079] [<ffffffff8111e087>] __lock_page+0x67/0x70
[ 241.291079] [<ffffffff81075970>] ? autoremove_wake_function+0x40/0x40
[ 241.295079] [<ffffffff811b8630>] ? blkdev_write_begin+0x30/0x30
[ 241.295079] [<ffffffff811202f4>] do_read_cache_page+0x94/0x110
[ 241.295079] [<ffffffff811203b9>] read_cache_page_async+0x19/0x20
[ 241.295079] [<ffffffff811203ce>] read_cache_page+0xe/0x20
[ 241.299079] [<ffffffff8130230d>] read_dev_sector+0x2d/0x90
[ 241.299079] [<ffffffff813033aa>] adfspart_check_ICS+0x7a/0x290
[ 241.299079] [<ffffffff813228d4>] ? snprintf+0x34/0x40
[ 241.299079] [<ffffffff81303330>] ? check_partition+0x210/0x210
[ 241.299079] [<ffffffff81303224>] check_partition+0x104/0x210
[ 241.299079] [<ffffffff81302aba>] rescan_partitions+0xda/0x310
[ 241.299079] [<ffffffff8165f54e>] ? _raw_spin_lock+0xe/0x20
[ 241.303079] [<ffffffff811b9184>] __blkdev_get+0x2d4/0x450
[ 241.303079] [<ffffffff811b935c>] blkdev_get+0x5c/0x210
[ 241.303079] [<ffffffff81301017>] register_disk+0x177/0x1a0
[ 241.303079] [<ffffffff813010e6>] add_disk+0xa6/0x1b0
[ 241.307080] [<ffffffff8164879a>] virtblk_probe+0x44c/0x4f1
[ 241.307080] [<ffffffff813b5423>] virtio_dev_probe+0xd3/0x120
[ 241.307080] [<ffffffff81407bb8>] really_probe+0x68/0x190
[ 241.307080] [<ffffffff81407d25>] driver_probe_device+0x45/0x70
[ 241.315080] [<ffffffff81407deb>] __driver_attach+0x9b/0xa0
[ 241.315080] [<ffffffff81407d50>] ? driver_probe_device+0x70/0x70
[ 241.315080] [<ffffffff81406308>] bus_for_each_dev+0x68/0x90
[ 241.319080] [<ffffffff81407a0e>] driver_attach+0x1e/0x20
[ 241.323081] [<ffffffff81407560>] bus_add_driver+0xd0/0x270
[ 241.323081] [<ffffffff81d200f5>] ? max_loop_setup+0x1a/0x1a
[ 241.323081] [<ffffffff814084a0>] driver_register+0x80/0x150
[ 241.323081] [<ffffffff81d200f5>] ? max_loop_setup+0x1a/0x1a
[ 241.327081] [<ffffffff813b5590>] register_virtio_driver+0x20/0x30
[ 241.327081] [<ffffffff81d2014c>] init+0x57/0x81
[ 241.327081] [<ffffffff81002042>] do_one_initcall+0x42/0x180
[ 241.331081] [<ffffffff81ce9681>] kernel_init+0xd2/0x156
[ 241.335081] [<ffffffff81668fe4>] kernel_thread_helper+0x4/0x10
[ 241.335081] [<ffffffff81ce95af>] ? parse_early_options+0x20/0x20
[ 241.335081] [<ffffffff81668fe0>] ...

[Joseph Salisbury (jsalisbury)]

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report at bugzilla.kernel.org [1]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

If you are comfortable with opening a bug upstream, It would be great if you can report back the upstream bug number in this bug report. That will allow us to link this bug to the upstream report.

[1] https://wiki.ubuntu.com/Bugs/Upstream/kernel

[Richard W.M. Jones (rich-annexia)]

Done:
https://bugzilla.kernel.org/show_bug.cgi?id=42778

[Richard W.M. Jones (rich-annexia)]

I think this is a bug in the Ubuntu qemu-kvm package, not in the kernel. (How to change the package that this bug is reported against?)

I tried compiling the Ubuntu qemu-kvm package without any patches (except define_AT_EMPTY_PATH.patch which is required in order for the package to compile). However the bug was still reproducible. So the bug is not caused by any Ubuntu patch.

I compiled qemu from git tag 'v1.0' => bug was reproducible.

I compiled qemu from git HEAD (currently da12872a097) => bug was NOT reproducible.

This proves that some commit in that range fixes the problem, but I haven't yet worked out which one it is.

[Richard W.M. Jones (rich-annexia)]

There are 1912 commits to check!

None of them is obviously a fix for a hanging bug in virtio/block code.

[Richard W.M. Jones (rich-annexia)]

I've identified the commit which fixes the bug, which is:

commit 41bd360325168b3c1db78eb7311420a1607d521f
Author: Jan Kiszka <email address hidden>
Date: Sun Jan 15 17:48:25 2012 +0100

    seabios: Update to release 1.6.3.1

    User visible changes in seabios:
     - Probe HPET existence (fix for -no-hpet)
     - Probe PCI existence (fix for -machine isapc)
     - usb: fix boot paths

    Signed-off-by: Jan Kiszka <email address hidden>

diff --git a/pc-bios/bios.bin b/pc-bios/bios.bin
index bd9ad0e..41e2b38 100644
Binary files a/pc-bios/bios.bin and b/pc-bios/bios.bin differ
diff --git a/roms/seabios b/roms/seabios
index 8e30147..80d11e8 160000
--- a/roms/seabios
+++ b/roms/seabios
@@ -1 +1 @@
-Subproject commit 8e301472e324b6d6496d8b4ffc66863e99d7a505
+Subproject commit 80d11e8577bf03e98f2eb1b0cb3a281ab2879c9e

So in fact the bug is in seabios, not in qemu-kvm.

[Richard W.M. Jones (rich-annexia)]

Luckily my first guess was correct. The following patch needs to be applied to SeaBIOS:

commit 3c5fcec00ce1317cda56d549259550fcc018c834
Author: Kevin O'Connor <email address hidden>
Date: Sat Oct 1 12:35:32 2011 -0400

    Fix alignment bug in pci_bios_init_root_regions().

    If there are no memory allocations for a given type then the "max" bar
    size is zero. However, ALIGN_DOWN does not handle an alignment of
    zero properly. Catch and handle the zero case.

    Signed-off-by: Kevin O'Connor <email address hidden>

diff --git a/src/pciinit.c b/src/pciinit.c
index a857da0..0d8758e 100644
--- a/src/pciinit.c
+++ b/src/pciinit.c
@@ -536,7 +536,7 @@ static void pci_bios_init_bus_bases(struct pci_bus *bus)
     }
 }

-#define ROOT_BASE(top, sum, align) ALIGN_DOWN((top)-(sum),(align))
+#define ROOT_BASE(top, sum, max) ALIGN_DOWN((top)-(sum),(max) ?: 1)

 static int pci_bios_init_root_regions(u32 start, u32 end)
 {

I tested this by applying this patch to seabios 0.6.2-0ubuntu2 and it completely cures the problem.

[Richard W.M. Jones (rich-annexia)]

Can we fix this? It's causing lots of people to have problems with libguestfs, and will cause random failures for *anyone* using virtio disks. I would say this bug should be urgent.