setting maxvcpus larger than vcpus in xen domu config triggers kernel bug in domu

[Tim Evers]

Dom0 is Debian Lenny 2.6.26 / Xen 4.0.1, DomU is lucid PV with current 2.6.35-31 kernel (happens also to 2.6.35-30 but not to 2.6.35.22).

No pygrub involved. Config is like:

kernel = "/home/xen/boot/ubuntu_amd64_vmlinuz-2.6.35-31-virtual-2011-11-21"
ramdisk = "/home/xen/boot/ubuntu_amd64_initrd.img-2.6.35-31-virtual-2011-11-21"
maxmem = 16384
memory = 2048

maxvcpus = 8
vcpus = 1
cpus = "15"

Booting the domu leads to a crash (excerpt):

[ 0.010000] CPU: Unsupported number of siblings 32
[ 0.010000] Performance Events: unsupported p6 CPU model 44 no PMU driver, software events only.
[ 0.010000] SMP alternatives: switching to UP code
[ 0.022859] ftrace: converting mcount calls to 0f 1f 44 00 00
[ 0.022881] ftrace: allocating 23051 entries in 91 pages
[ 0.030076] cpu 0 spinlock event irq 1
[ 0.030390] installing Xen timer for CPU 1
[ 0.030409] cpu 1 spinlock event irq 7
[ 0.030442] SMP alternatives: switching to SMP code
[ 0.010000] CPU: Unsupported number of siblings 32
[ 0.045228] installing Xen timer for CPU 2
[ 0.045254] cpu 2 spinlock event irq 13
[ 0.010000] CPU: Unsupported number of siblings 32
[ 0.045634] installing Xen timer for CPU 3
[ 0.045670] cpu 3 spinlock event irq 19
[ 0.010000] CPU: Unsupported number of siblings 32
[ 0.046056] installing Xen timer for CPU 4
[ 0.046092] cpu 4 spinlock event irq 25
[ 0.010000] CPU: Unsupported number of siblings 32
[ 0.046480] installing Xen timer for CPU 5
[ 0.046516] cpu 5 spinlock event irq 31
[ 0.010000] CPU: Unsupported number of siblings 32
[ 0.046891] installing Xen timer for CPU 6
[ 0.046926] cpu 6 spinlock event irq 37
[ 0.010000] CPU: Unsupported number of siblings 32
[ 0.047307] installing Xen timer for CPU 7
[ 0.047354] cpu 7 spinlock event irq 43
[ 0.010000] CPU: Unsupported number of siblings 32
[ 0.047642] Brought up 8 CPUs
[ 0.048434] devtmpfs: initialized
[ 0.050968] Grant table initialized
[ 0.050979] regulator: core version 0.5
[ 0.060008] Time: 165:165:165 Date: 165/165/65
[ 0.060008] NET: Registered protocol family 16
[ 0.060099] Trying to unpack rootfs image as initramfs...
[ 0.087325] Freeing initrd memory: 24672k freed
[ 0.200000] Cannot set affinity for irq 0
[ 0.200000] Broke affinity for irq 6
[ 0.200000] Broke affinity for irq 7
[ 0.200000] Broke affinity for irq 8
[ 0.200000] Broke affinity for irq 9
[ 0.200000] Broke affinity for irq 10
[ 0.200000] Broke affinity for irq 11
[ 0.210000] ------------[ cut here ]------------
[ 0.210000] kernel BUG at /build/buildd/linux-lts-backport-maverick-2.6.35/arch/x86/xen/spinlock.c:344!
[ 0.210000] invalid opcode: 0000 [#1] SMP
[ 0.210000] last sysfs file:
[ 0.210000] CPU 5
[ 0.210000] Modules linked in:
[ 0.210000]
[ 0.210000] Pid: 18, comm: migration/5 Not tainted 2.6.35-31-virtual #62~lucid1-Ubuntu /
[ 0.210000] RIP: e030:[<ffffffff81007cb4>] [<ffffffff81007cb4>] dummy_handler+0x4/0x10
[ 0.210000] RSP: e02b:ffff880003a8dea8 EFLAGS: 00010046
[ 0.210000] RAX: ffffffffff57b000 RBX: ffff88007fc0cba0 RCX: 0000000000000000
[ 0.210000] RDX: 0000000000400200 RSI: 0000000000000000 RDI: 000000000000001f
[ 0.210000] RBP: ffff880003a8dea8 R08: 0000000000000000 R09: 0000000000000000
[ 0.210000] R10: ffff880003a95028 R11: 0000000000012d70 R12: 0000000000000000
[ 0.210000] R13: 0000000000000000 R14: 000000000000001f R15: 0000000000000100
[ 0.210000] FS: 0000000000000000(0000) GS:ffff880003a8a000(0000) knlGS:0000000000000000
[ 0.210000] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 0.210000] CR2: 0000000000000000 CR3: 0000000001a50000 CR4: 0000000000002660
[ 0.210000] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.210000] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 0.210000] Process migration/5 (pid: 18, threadinfo ffff88007d454000, task ffff88007d42db80)
[ 0.210000] Stack:
[ 0.210000] ffff880003a8def8 ffffffff810c9df0 0000000000000000 0000000000000000
[ 0.210000] <0> ffff880003a9ae20 ffff88007fc03500 000000000000001f 0000000000000a00
[ 0.210000] <0> 0000000000000001 0000000000000100 ffff880003a8df18 ffffffff810cc442
[ 0.210000] Call Trace:
[ 0.210000] <IRQ>
[ 0.210000] [<ffffffff810c9df0>] handle_IRQ_event+0x50/0x160
[ 0.210000] [<ffffffff810cc442>] handle_percpu_irq+0x42/0x80
[ 0.210000] [<ffffffff813481fe>] xen_evtchn_do_upcall+0x1de/0x210
[ 0.210000] [<ffffffff810b1e00>] ? cpu_stopper_thread+0xf0/0x1d0
[ 0.210000] [<ffffffff8100b02e>] xen_do_hypervisor_callback+0x1e/0x30
[ 0.210000] <EOI>
[ 0.210000] [<ffffffff810b1e00>] ? cpu_stopper_thread+0xf0/0x1d0
[ 0.210000] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1010
[ 0.210000] [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1010
[ 0.210000] [<ffffffff81006b4d>] ? xen_force_evtchn_callback+0xd/0x10
[ 0.210000] [<ffffffff810072f2>] ? check_events+0x12/0x20
[ 0.210000] [<ffffffff81007299>] ? xen_irq_enable_direct_end+0x0/0x7
[ 0.210000] [<ffffffff810b1f8f>] ? stop_machine_cpu_stop+0xaf/0xe0
[ 0.210000] [<ffffffff810b1ee0>] ? stop_machine_cpu_stop+0x0/0xe0
[ 0.210000] [<ffffffff810b1e06>] ? cpu_stopper_thread+0xf6/0x1d0
[ 0.210000] [<ffffffff815a4b71>] ? schedule+0x3e1/0x850
[ 0.210000] [<ffffffff815a6f8e>] ? _raw_spin_unlock_irqrestore+0x1e/0x30
[ 0.210000] [<ffffffff810b1d10>] ? cpu_stopper_thread+0x0/0x1d0
[ 0.210000] [<ffffffff8107f526>] ? kthread+0x96/0xa0
[ 0.210000] [<ffffffff8100aee4>] ? kernel_thread_helper+0x4/0x10
[ 0.210000] [<ffffffff8100a313>] ? int_ret_from_sys_call+0x7/0x1b
[ 0.210000] [<ffffffff815a751d>] ? retint_restore_args+0x5/0x6
[ 0.210000] [<ffffffff8100aee0>] ? kernel_thread_helper+0x0/0x10
[ 0.210000] Code: 89 e5 c9 0f 95 c0 c3 55 b8 01 00 00 00 86 07 84 c0 48 89 e5 0f 94 c0 c9 0f b6 c0 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 <0f> 0b eb fe 0f 1f 84 00 00 00 00 00 55 ba ff ff ff ff 48 89 e5
[ 0.210000] RIP [<ffffffff81007cb4>] dummy_handler+0x4/0x10
[ 0.210000] RSP <ffff880003a8dea8>
[ 0.210000] ---[ end trace f17e946d22a56015 ]---

(full trace in attachment).

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 893177

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Tim Evers (te-8)]

This is to confirm that running the bug collect script is not possible since the machine is crashing before it get's to a shell.

[Joseph Salisbury (jsalisbury)]

Have you tested Natty or Oneiric to see if they have the issue as well?

[Tim Evers (te-8)]

Kernel 2.6.36-8 shows exactly the same behaviour, oneiric is untested.

Beside that I've tracked down the thing a bit:

- Installed 2.6.35-31 source packet
- compiled with lucid .config, except disabling CONFIG_PARAVIRT_SPINLOCKS
- runs!

So maybe this is a bug in the dom0 pv drivers. I'll follow this path and report back...

[Tim Evers (te-8)]

Update:

this happens on hypervisor / dom0 versions:

xen-4.0.4-rc1-pre / 3.1.2
xen-4.1.3-rc1-pre / 3.1.2
xen-4.0.4-rc1-pre / 2.6.32.31

[Tim Evers (te-8)]

Last update for the moment:

Debian seems to have disabled CONFIG_PARAVIRT_SPINLOCKS in it's pv_ops kernel (2.6.32-5-686-*).

I'm stuck.

[Tim Evers (te-8)]

Another finding: works with 2.6.35.22 and CONFIG_PARAVIRT_SPINLOCKS on so this is a regression.

[Joseph Salisbury (jsalisbury)]

This bug was filed against a series that is no longer supported and so is being marked as Won't Fix. If this issue still exists in a supported series, please file a new bug.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.