Bringing up a new Xen instance causes a kernel panic

[Graeme Mathieson]

I've just reinstalled a server and decided that virtualisation is the way forward. However, after having created three images with the following configuration (identical apart from IP addresses):

#
# Configuration file for the Xen instance orkell-app-02.orkell.co.uk, created on
# Thu Apr 12 14:51:18 2007.
#

#
# Kernel + memory size
#
kernel = '/boot/vmlinuz-2.6.19-4-server'

ramdisk = '/boot/initrd.img-2.6.19-4-server'

memory = '256'

#
# Disk device(s).
#
root = '/dev/sda1 ro'

disk = [ 'phy:orkell-host-02/orkell-app-02.orkell.co.uk-disk,sda1,w', 'phy:orkell-host-02/orkell-app-02.orkell.co.uk-swap,sda2,w' ]

#
# Hostname
#
name = 'orkell-app-02.orkell.co.uk'

#
# Networking
#
vif = [ 'ip=194.36.163.208' ]

#
# Behaviour
#
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'

and starting it, I get a kernel panic. Here's the relevant bit from syslog, hopefully:

Apr 12 18:00:13 orkell-host-02 logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/1/0
Apr 12 18:00:13 orkell-host-02 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/2049
Apr 12 18:00:13 orkell-host-02 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/1/2050
Apr 12 18:00:13 orkell-host-02 kernel: [ 23.472608] device vif1.0 entered promiscuous mode
Apr 12 18:00:13 orkell-host-02 kernel: [ 23.472620] audit(1176397213.139:4): dev=vif1.0 prom=256 old_prom=0 auid=4294967295
Apr 12 18:00:13 orkell-host-02 kernel: [ 23.488627] ADDRCONF(NETDEV_UP): vif1.0: link is not ready
Apr 12 18:00:13 orkell-host-02 kernel: [ 23.553829] ip_tables: (C) 2000-2006 Netfilter Core Team
Apr 12 18:00:13 orkell-host-02 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2050/physical-device fe:5 to xenstore.
Apr 12 18:00:13 orkell-host-02 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2050/hotplug-status connected to xenstore.
Apr 12 18:00:13 orkell-host-02 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2049/physical-device fe:4 to xenstore.
Apr 12 18:00:13 orkell-host-02 logger: /etc/xen/scripts/block: Writing backend/vbd/1/2049/hotplug-status connected to xenstore.
Apr 12 18:00:13 orkell-host-02 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif1.0, bridge xenbr0.
Apr 12 18:00:13 orkell-host-02 logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/1/0/hotplug-status connected to xenstore.
Apr 12 18:00:13 orkell-host-02 kernel: [ 24.320015] ADDRCONF(NETDEV_CHANGE): vif1.0: link becomes ready
Apr 12 18:00:14 orkell-host-02 kernel: [ 24.420017] xenbr0: port 3(vif1.0) entering learning state
Apr 12 18:00:14 orkell-host-02 kernel: [ 24.430006] xenbr0: topology change detected, propagating
Apr 12 18:00:14 orkell-host-02 kernel: [ 24.430010] xenbr0: port 3(vif1.0) entering forwarding state
Apr 12 18:00:16 orkell-host-02 ntpdate[4200]: step time server 82.211.81.145 offset -0.144984 sec
Apr 12 18:00:19 orkell-host-02 logger: /etc/xen/scripts/vif-bridge: online XENBUS_PATH=backend/vif/2/0
Apr 12 18:00:19 orkell-host-02 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/2/2049
Apr 12 18:00:19 orkell-host-02 logger: /etc/xen/scripts/block: add XENBUS_PATH=backend/vbd/2/2050
Apr 12 18:00:20 orkell-host-02 kernel: [ 30.467988] device vif2.0 entered promiscuous mode
Apr 12 18:00:20 orkell-host-02 kernel: [ 30.468005] audit(1176397219.994:5): dev=vif2.0 prom=256 old_prom=0 auid=4294967295
Apr 12 18:00:20 orkell-host-02 kernel: [ 30.470753] ADDRCONF(NETDEV_UP): vif2.0: link is not ready
Apr 12 18:00:20 orkell-host-02 logger: /etc/xen/scripts/vif-bridge: Successful vif-bridge online for vif2.0, bridge xenbr0.
Apr 12 18:00:20 orkell-host-02 logger: /etc/xen/scripts/vif-bridge: Writing backend/vif/2/0/hotplug-status connected to xenstore.
Apr 12 18:00:20 orkell-host-02 logger: /etc/xen/scripts/block: Writing backend/vbd/2/2050/physical-device fe:7 to xenstore.
Apr 12 18:00:20 orkell-host-02 logger: /etc/xen/scripts/block: Writing backend/vbd/2/2050/hotplug-status connected to xenstore.
Apr 12 18:00:20 orkell-host-02 logger: /etc/xen/scripts/block: Writing backend/vbd/2/2049/physical-device fe:6 to xenstore.
Apr 12 18:00:20 orkell-host-02 logger: /etc/xen/scripts/block: Writing backend/vbd/2/2049/hotplug-status connected to xenstore.
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.166738] ------------[ cut here ]------------
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.166863] kernel BUG at drivers/xen/core/evtchn.c:481!
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.166960] invalid opcode: 0000 [#1]
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.167054] SMP
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.167278] Modules linked in: xt_tcpudp xt_physdev iptable_filter ip_tables x_tables af_packet bridge ipv6 parport_pc lp parport i2c_piix4 i2c_core sworks_agp agpgart serio_raw evdev psmouse shpchp pci_hotplug pcspkr k8temp ext3 jbd mb
cache sg sd_mod ide_cd cdrom ata_generic 3w_xxxx tg3 sata_promise libata scsi_mod serverworks generic thermal processor fan dm_mod
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.170329] CPU: 1
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.170330] EIP: 0061:[retrigger_vector+62/80] Not tainted VLI
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.170331] EFLAGS: 00010046 (2.6.19-4-server #2)
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.170617] EIP is at retrigger_vector+0x3e/0x50
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.170713] eax: 00000000 ebx: 02080000 ecx: f5746000 edx: 0000001d
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.170814] esi: c03dda80 edi: 0000010d ebp: 00000000 esp: c0523e60
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.170915] ds: 007b es: 007b ss: 0069
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.171010] Process xenwatch (pid: 13, ti=c0522000 task=c147a560 task.ti=c0522000)
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.171111] Stack: c014b691 c03dda80 0000010d c03ddaa8 c014b355 c0375fb8 ed7da6a0 00000014
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.171837] c0d4b400 c0d4b400 00000000 c0523efc c0d4b400 c0257ecb 00000000 c025834f
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.172561] c0d4b000 c0d4b400 ec8bd500 00000002 00000000 00000000 c0523efc c0523ec0
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.173285] Call Trace:
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.173864] [check_irq_resend+97/112] check_irq_resend+0x61/0x70
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.174132] [enable_irq+197/208] enable_irq+0xc5/0xd0
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.174393] [__netif_up+11/32] __netif_up+0xb/0x20
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.174954] [netif_map+623/736] netif_map+0x26f/0x2e0
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.175520] [maybe_connect+509/1312] maybe_connect+0x1fd/0x520
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.176086] [otherend_changed+175/192] otherend_changed+0xaf/0xc0
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.176645] [xenwatch_handle_callback+26/96] xenwatch_handle_callback+0x1a/0x60
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.177202] [xenwatch_thread+285/304] xenwatch_thread+0x11d/0x130
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.177758] [kthread+246/256] kthread+0xf6/0x100
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.177996] [kernel_thread_helper+7/16] kernel_thread_helper+0x7/0x10
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.178171] ===================
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.178263] Code: b8 01 00 00 00 c3 8d b4 26 00 00 00 00 0f b7 d0 0f a3 91 80 08 00 00 19 c0 85 c0 74 0e f0 0f ab 91 00 08 00 00 b8 01 00 00 00 c3 <0f> 0b e1 01 e9 8f 32 c0 eb e8 90 8d b4 26 00 00 00 00 0f b7 04
Apr 12 18:00:20 orkell-host-02 kernel: [ 31.182785] EIP: [retrigger_vector+62/80] retrigger_vector+0×3e/0×50 SS:ESP 0069:c0523e60

Any ideas?
Apr 12 18:00:21 orkell-host-02 kernel: [ 31.183022] <7>eth0: no IPv6 routers present

[Graeme Mathieson (mathie)]

Oh, I meant to say. If there's any other information I can provide that would be useful, please let me know!

Cheers,

Graeme

[Chuck Short (zulcss)]

Which version of xen are you using?

[Graeme Mathieson (mathie)]

mathie@orkell-host-02:~$ dpkg -l|grep xen
ii libc6-xen 2.5-0ubuntu14 GNU C Library: Shared libraries [Xen version
ii libxen3.0 3.0.3-0ubuntu10 library interface for Xen, a Virtual Machine
ii python-xen3.0 3.0.3-0ubuntu10 python bindings for Xen, a Virtual Machine M
ii ubuntu-xen-server 0.0.1-2ubuntu3 Xen software for running on servers.
ii xen-docs-3.0 3.0.3-0ubuntu10 documentation for XEN, a Virtual Machine Mon
ii xen-hypervisor-3.0-i386-pae 3.0.3-0ubuntu10 The Xen Hypervisor for i386 (pae enabled ver
ii xen-image-2.6.19-4-server 2.6.19-2ubuntu7 Linux xen 2.6.19 image on x86.
ii xen-ioemu-3.0 3.0.3-0ubuntu10 XEN administrative tools
ii xen-tools 3.1-1ubuntu1 Tools to manage debian XEN virtual servers
ii xen-utils-3.0 3.0.3-0ubuntu10 XEN administrative tools

[denver (gaby-rohost)]

Just change the dom0 CPU's from 0 to 1

(dom0-cpus 0) --> (dom0-cpus 1)

in /etc/xen/xend-config.sxp

fixed it for me at least.

[Daniel T Chen (crimsun)]

Is this symptom still reproducible in 8.10 beta?

[Graeme Mathieson (mathie)]

I am unable to verify if the bug is still reproducible because, after encountering this problem, we switched to a completely different production deployment strategy.

[Dimitrios Symeonidis (azimout)]

closing this bug, as per your last comment. feel free to switch back to new if you upgrade to the latest version and still face this issue. thank you, and good luck.