Kernel BUG at drivers/xen/core/evtchn.c:481

[Erik de Castro Lopo]

Binary package hint: xen-image-2.6.19-4-generic-amd64

Using Gutsy with Ubuntu kernel xen-image-2.6.19-4-generic-amd64 on a quad core amd64 machine.

I've now seen it crash like this twice (doesn't happen every time). See dmesg output below.

Result of this is a machine that doesn't work and must be rebooted.

[ 42.182170] device vif1.0 entered promiscuous mode
[ 42.182186] audit(1191414038.334:2): dev=vif1.0 prom=256 old_prom=0 auid=4294967295
[ 42.185784] ADDRCONF(NETDEV_UP): vif1.0: link is not ready
[ 50.160007] xenbr0: no IPv6 routers present
[ 53.372088] ----------- [cut here ] --------- [please bite here ] ---------
[ 53.372098] Kernel BUG at drivers/xen/core/evtchn.c:481
[ 53.372102] invalid opcode: 0000 [1] SMP
[ 53.372106] CPU 1
[ 53.372108] Modules linked in: af_packet loop ppdev tun ip6table_filter ip6_tables iptable_raw xt_comment xt_policy xt_multiport ipt_ULOG ipt_TTL ipt_ttl ipt_TOS ipt_tos ipt_TCPMSS ipt_SAME ipt_REJECT ipt_REDIRECT ipt_recent ipt_owner ipt_NETMAP ipt_MASQUERADE ipt_LOG ipt_iprange ipt_hashlimit ipt_ECN ipt_ecn ipt_CLUSTERIP ipt_ah ipt_addrtype ip_nat_tftp ip_nat_snmp_basic ip_nat_pptp ip_nat_irc ip_nat_h323 ip_nat_ftp ip_nat_amanda ip_conntrack_tftp ip_conntrack_pptp ip_conntrack_netbios_ns ip_conntrack_irc ip_conntrack_h323 ip_conntrack_ftp ts_kmp ip_conntrack_amanda xt_tcpmss xt_pkttype xt_physdev bridge xt_NFQUEUE xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_tcpudp xt_state iptable_nat ip_nat ip_conntrack iptable_mangle nfnetlink iptable_filter ip_tables x_tables parport_pc lp parport ide_cd cdrom joydev i2c_nforce2 usbhid i2c_core k8temp evdev shpchp pci_hotplug pcspkr ipv6 ext3 jbd mbcache sg sd_mod amd74xx generic ata_generic ehci_hcd tg3 forcedeth ohci_hcd usbcore sata_nv libata scsi_mod raid10 raid456 xor raid1 raid0 multipath linear md_mod
[ 53.372211] Pid: 21, comm: xenwatch Not tainted 2.6.19-4-generic-amd64 #2
[ 53.372215] RIP: e030:[<ffffffff8038fd07>] [<ffffffff8038fd07>] retrigger_vector+0x27/0x40
[ 53.372227] RSP: e02b:ffff880001ac7d38 EFLAGS: 00010046
[ 53.372231] RAX: 0000000000000000 RBX: 0000000000008800 RCX: ffffffffff578000
[ 53.372235] RDX: 000000000000001f RSI: ffff880001ac7cd0 RDI: 0000000000000110
[ 53.372239] RBP: 0000000000000110 R08: 0000000000000000 R09: 0000000000000001
[ 53.372244] R10: ffff8800758da2c0 R11: ffffffff8038fce0 R12: ffffffff80544080
[ 53.372249] R13: ffffffff805440bc R14: 0000000000000000 R15: ffff880001ac7da0
[ 53.372257] FS: 0000000000000000(0000) GS:ffffffff8053b080(0000) knlGS:0000000000000000
[ 53.372261] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 53.372264] CR2: 0000000000000000 CR3: 000000006c5ce000 CR4: 0000000000000660
[ 53.372269] Process xenwatch (pid: 21, threadinfo ffff880001ac6000, task ffff880001b00180)
[ 53.372273] Stack: ffffffff802b70c0 ffff88000131b500 ffff88000131b500 0000000000000000
[ 53.372282] 000000000000020b 0000000000000008 ffffffff803a0cec 0000000000000000
[ 53.372289] ffffffff803a121e ffff880001ac7e7c 0000000000000000 0000000000000000
[ 53.372295] Call Trace:
[ 53.372301] [<ffffffff802b70c0>] enable_irq+0xb0/0x100
[ 53.372314] [<ffffffff803a0cec>] __netif_up+0xc/0x20
[ 53.372320] [<ffffffff803a121e>] netif_map+0x30e/0x370
[ 53.372336] [<ffffffff802680f3>] thread_return+0x5d/0x11a
[ 53.372342] [<ffffffff80398200>] cmp_dev+0x0/0x60
[ 53.372347] [<ffffffff80427fb2>] klist_iter_exit+0x12/0x20
[ 53.372362] [<ffffffff802a2430>] keventd_create_kthread+0x0/0x80
[ 53.372368] [<ffffffff803a04e1>] maybe_connect+0x2a1/0x4b0
[ 53.372388] [<ffffffff80396660>] xenwatch_thread+0x0/0x180
[ 53.372394] [<ffffffff802a2430>] keventd_create_kthread+0x0/0x80
[ 53.372400] [<ffffffff80395815>] xenwatch_handle_callback+0x15/0x50
[ 53.372406] [<ffffffff803967b9>] xenwatch_thread+0x159/0x180
[ 53.372413] [<ffffffff802a2670>] autoremove_wake_function+0x0/0x30
[ 53.372421] [<ffffffff802a2430>] keventd_create_kthread+0x0/0x80
[ 53.372428] [<ffffffff80236749>] kthread+0xd9/0x120
[ 53.372435] [<ffffffff8022a814>] schedule_tail+0x44/0xc0
[ 53.372444] [<ffffffff80266418>] child_rip+0xa/0x12
[ 53.372450] [<ffffffff802a2430>] keventd_create_kthread+0x0/0x80
[ 53.372469] [<ffffffff80236670>] kthread+0x0/0x120
[ 53.372475] [<ffffffff8026640e>] child_rip+0x0/0x12
[ 53.372482]
[ 53.372484]
[ 53.372485] Code: 0f 0b 68 11 34 46 80 c2 e1 01 f0 0f ab 91 00 08 00 00 b8 01
[ 53.372500] RIP [<ffffffff8038fd07>] retrigger_vector+0x27/0x40
[ 53.372507] RSP <ffff880001ac7d38>
[ 53.372510] <6>Capability LSM initialized

[Simon Huerlimann (huerlisi)]

Got the same on kernel 2.6.19-4-server #2 SMP i686. Here's what I did:
shuerlimann@prolo1:~$ sudo xm create /etc/xen/sysadmin.cfg
Using config file "/etc/xen/sysadmin.cfg".
Started domain sysadmin
shuerlimann@prolo1:~$ sudo xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 1893 2 r----- 39.1
sysadmin 2 128 1 r----- 4.1
shuerlimann@prolo1:~$ sudo /etc/init.d/xend
xend xendomains
shuerlimann@prolo1:~$ sudo /etc/init.d/xendomains restart
Shutting down Xen domains: sysadmin(save)...Error: Device 0 not connected
 SHUTDOWN_ALL .All domains terminated
/etc/init.d/xendomains: line 264: 5217 Terminated watchdog_xm shutdown 1
Restoring Xen domains: sysadminl prolo1 kernel: [ 1100.910599] ------------[ cut here ]------------
 prolo1 kernel: [ 1100.910983] kernel BUG at drivers/xen/core/evtchn.c:481!
 prolo1 kernel: [ 1100.911218] invalid opcode: 0000 [#1]
 prolo1 kernel: [ 1100.911426] SMP
 prolo1 kernel: [ 1100.912389] CPU: 1
 prolo1 kernel: [ 1100.912391] EIP: 0061:[<c0247cce>] Not tainted VLI
 prolo1 kernel: [ 1100.912394] EFLAGS: 00010046 (2.6.19-4-server #2)
 prolo1 kernel: [ 1100.913007] EIP is at retrigger_vector+0x3e/0x50
 prolo1 kernel: [ 1100.913223] eax: 00000000 ebx: 02080000 ecx: f5746000 edx: 00000017
 prolo1 kernel: [ 1100.913330] esi: c03dd900 edi: 0000010a ebp: 00000000 esp: ed7bfe60
 prolo1 kernel: [ 1100.913436] ds: 007b es: 007b ss: 0069
 prolo1 kernel: [ 1100.913524] Process xenwatch (pid: 13, ti=ed7be000 task=c14fb560 task.ti=ed7be000)
 prolo1 kernel: [ 1100.913632] Stack: c014b691 c03dd900 0000010a c03dd928 c014b355 c0131aee 00000000 00000001
 prolo1 kernel: [ 1100.913820] 00000001 e5890c00 00000000 ed7bfefc e5890c00 c0257ecb 00000000 c025834f
 prolo1 kernel: [ 1100.914203] e5890800 e5890c00 e6cb9ba0 00000002 00000000 00000000 ed7bfefc ed7bfec0
 prolo1 kernel: [ 1100.914601] Call Trace:
 prolo1 kernel: [ 1100.915824] [<c014b691>] check_irq_resend+0x61/0x70
 prolo1 kernel: [ 1100.916293] [<c014b355>] enable_irq+0xc5/0xd0
 prolo1 kernel: [ 1100.916751] [<c0257ecb>] __netif_up+0xb/0x20
 prolo1 kernel: [ 1100.917839] [<c025834f>] netif_map+0x26f/0x2e0
 prolo1 kernel: [ 1100.918907] [<c02573bd>] maybe_connect+0x1fd/0x520
 prolo1 kernel: [ 1100.919994] [<c024f2cf>] otherend_changed+0xaf/0xc0
 prolo1 kernel: [ 1100.921066] [<c024d28a>] xenwatch_handle_callback+0x1a/0x60
 prolo1 kernel: [ 1100.922160] [<c024e00d>] xenwatch_thread+0x11d/0x130
 prolo1 kernel: [ 1100.923242] [<c0135336>] kthread+0xf6/0x100
 prolo1 kernel: [ 1100.923637] [<c0105367>] kernel_thread_helper+0x7/0x10
 prolo1 kernel: [ 1100.923908] =======================
 prolo1 kernel: [ 1100.924142] Code: b8 01 00 00 00 c3 8d b4 26 00 00 00 00 0f b7 d0 0f a3 91 80 08 00 00 19 c0 85 c0 74 0e f0 0f ab 91 00 08 00 00 b8 01 00 00 00 c3 <0f> 0b e1 01 e9 8f 32 c0 eb e8 90 8d b4 26 00 00 00 00 0f b7 04
 prolo1 kernel: [ 1100.925282] EIP: [<c0247cce>] retrigger_vector+0x3e/0x50 SS:ESP 0069:ed7bfe60
 webmaster1Error: Restore failed
Usa...

[Adam Ashley (adamashley)]

I am also getting this problem. Unfortunately it's on every single reboot of the server.

Jan 10 09:24:44 vali kernel: [ 81.094131] ------------[ cut here ]------------
Jan 10 09:24:44 vali kernel: [ 81.094195] kernel BUG at drivers/xen/core/evtchn.c:481!
Jan 10 09:24:44 vali kernel: [ 81.094246] invalid opcode: 0000 [#1]
Jan 10 09:24:44 vali kernel: [ 81.094296] SMP
Jan 10 09:24:44 vali kernel: [ 81.094434] Modules linked in: xt_tcpudp xt_physdev iptable_filter ip_tables x_tables af_packet bridge video sbs i2c_ec button
battery container ac asus_acpi loop lp serial_core parport_pc parport ipv6 i2c_piix4 i2c_core k8temp ati_agp agpgart psmouse serio_raw pcspkr shpchp pci_hotplu
g evdev ext3 jbd mbcache sg sd_mod floppy tg3 atiixp ohci_hcd ehci_hcd usbcore sata_sil ata_generic libata scsi_mod generic thermal processor fan dm_mod
Jan 10 09:24:44 vali kernel: [ 81.096858] CPU: 1
Jan 10 09:24:44 vali kernel: [ 81.096859] EIP: 0061:[retrigger_vector+62/80] Not tainted VLI
Jan 10 09:24:44 vali kernel: [ 81.096860] EFLAGS: 00010046 (2.6.19-4-server #2)
Jan 10 09:24:44 vali kernel: [ 81.097014] EIP is at retrigger_vector+0x3e/0x50
Jan 10 09:24:44 vali kernel: [ 81.097065] eax: 00000000 ebx: 02080000 ecx: f5746000 edx: 0000003b
Jan 10 09:24:44 vali kernel: [ 81.097117] esi: c03de380 edi: 0000011f ebp: 00000000 esp: c1703e60
Jan 10 09:24:44 vali kernel: [ 81.097169] ds: 007b es: 007b ss: 0069
Jan 10 09:24:44 vali kernel: [ 81.097220] Process xenwatch (pid: 13, ti=c1702000 task=c1778560 task.ti=c1702000)
Jan 10 09:24:44 vali kernel: [ 81.097272] Stack: c014b691 c03de380 0000011f c03de3a8 c014b355 c0131aee 00000000 00000001
Jan 10 09:24:44 vali kernel: [ 81.097714] 00000001 d5691400 00000000 c1703efc d5691400 c0257ecb 00000000 c025834f
Jan 10 09:24:44 vali kernel: [ 81.098155] d5691000 d5691400 d5a91b40 00000002 00000000 00000000 c1703efc c1703ec0
Jan 10 09:24:44 vali kernel: [ 81.098596] Call Trace:
Jan 10 09:24:44 vali kernel: [ 81.099061] [check_irq_resend+97/112] check_irq_resend+0x61/0x70
Jan 10 09:24:44 vali kernel: [ 81.099248] [enable_irq+197/208] enable_irq+0xc5/0xd0
Jan 10 09:24:44 vali kernel: [ 81.099431] [__netif_up+11/32] __netif_up+0xb/0x20
Jan 10 09:24:44 vali kernel: [ 81.099887] [netif_map+623/736] netif_map+0x26f/0x2e0
Jan 10 09:24:44 vali kernel: [ 81.100349] [maybe_connect+509/1312] maybe_connect+0x1fd/0x520
Jan 10 09:24:44 vali kernel: [ 81.100811] [otherend_changed+175/192] otherend_changed+0xaf/0xc0
Jan 10 09:24:44 vali kernel: [ 81.101265] [xenwatch_handle_callback+26/96] xenwatch_handle_callback+0x1a/0x60
Jan 10 09:24:44 vali kernel: [ 81.101721] [xenwatch_thread+285/304] xenwatch_thread+0x11d/0x130
Jan 10 09:24:44 vali kernel: [ 81.102176] [kthread+246/256] kthread+0xf6/0x100
Jan 10 09:24:44 vali kernel: [ 81.102339] [kernel_thread_helper+7/16] kernel_thread_helper+0x7/0x10
Jan 10 09:24:44 vali kernel: [ 81.102443] =======================
Jan 10 09:24:44 vali kernel: [ 81.102493] Code: b8 01 00 00 00 c3 8d b4 26 00 00 00 00 0f b7 d0 0f a3 91 80 08 00 00 19 c0 85 c0 74 0e f0 0f ab 91 00 08 00 0
0 b8 01 00 00 00 c3 <...

[Arnd (arnd-arndnet)]

Same here...

Jan 30 22:56:57 moonraker kernel: [ 3586.748458] device vif4.0 entered promiscuous mode
Jan 30 22:56:57 moonraker kernel: [ 3586.748467] audit(1201730217.790:9): dev=vif4.0 prom=256 old_prom=0 auid=4294967295
Jan 30 22:56:57 moonraker kernel: [ 3586.750219] ADDRCONF(NETDEV_UP): vif4.0: link is not ready
Jan 30 22:56:58 moonraker kernel: [ 3587.195559] ----------- [cut here ] --------- [please bite here ] ---------
Jan 30 22:56:58 moonraker kernel: [ 3587.195616] Kernel BUG at drivers/xen/core/evtchn.c:481
Jan 30 22:56:58 moonraker kernel: [ 3587.195644] invalid opcode: 0000 [1] SMP
Jan 30 22:56:58 moonraker kernel: [ 3587.195675] CPU 1
Jan 30 22:56:58 moonraker kernel: [ 3587.195699] Modules linked in: xt_physdev iptable_filter ip_tables x_tables dm_mirror autofs4 binfmt_misc rfcomm l2cap bluetooth af_packet bridge capability commoncap eeprom lm92 w83781d hwmon_vid i2c_isa i2c_i801 i2c_core parport_pc lp parport tsdev ide_cd cdrom pcspkr iTCO_wdt ata_generic ata_piix psmouse serio_raw shpchp pci_hotplug ipv6 evdev ext3 jbd mbcache sg sd_mod piix 3w_9xxx ahci libata scsi_mod ehci_hcd generic e1000 uhci_hcd usbcore raid1 md_mod dm_mod
Jan 30 22:56:58 moonraker kernel: [ 3587.196035] Pid: 13, comm: xenwatch Not tainted 2.6.19-4-generic-amd64 #2
Jan 30 22:56:58 moonraker kernel: [ 3587.196067] RIP: e030:[<ffffffff8038fcd7>] [<ffffffff8038fcd7>] retrigger_vector+0x27/0x40
Jan 30 22:56:58 moonraker kernel: [ 3587.196149] RSP: e02b:ffff880001bb3d38 EFLAGS: 00010046
Jan 30 22:56:58 moonraker kernel: [ 3587.196188] RAX: 0000000000000000 RBX: 0000000000008580 RCX: ffffffffff578000
Jan 30 22:56:58 moonraker kernel: [ 3587.196223] RDX: 0000000000000019 RSI: ffff880001bb3cd0 RDI: 000000000000010b
Jan 30 22:56:58 moonraker kernel: [ 3587.196254] RBP: 000000000000010b R08: 0000000000000071 R09: 0000000000000000
Jan 30 22:56:58 moonraker kernel: [ 3587.196287] R10: ffff8800ed3a0c80 R11: ffffffff8038fcb0 R12: ffffffff80543e00
Jan 30 22:56:58 moonraker kernel: [ 3587.196319] R13: ffffffff80543e3c R14: 0000000000000000 R15: ffff880001bb3da0
Jan 30 22:56:58 moonraker kernel: [ 3587.196354] FS: 00002afb77361e30(0000) GS:ffffffff8053b080(0000) knlGS:0000000000000000
Jan 30 22:56:58 moonraker kernel: [ 3587.196402] CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
Jan 30 22:56:58 moonraker kernel: [ 3587.196431] CR2: 0000000000000000 CR3: 00000000c9a5e000 CR4: 0000000000002660
Jan 30 22:56:58 moonraker kernel: [ 3587.196464] Process xenwatch (pid: 13, threadinfo ffff880001bb2000, task ffff880001bfd040)
Jan 30 22:56:58 moonraker kernel: [ 3587.196512] Stack: ffffffff802b7090 ffff8800ca318500 ffff8800ca318500 0000000000000000
Jan 30 22:56:58 moonraker kernel: [ 3587.196572] 000000000000020c 0000000000000009 ffffffff803a0cbc 0000000000000000
Jan 30 22:56:58 moonraker kernel: [ 3587.196629] ffffffff803a11ee ffff880001bb3e7c 0000000000000000 0000000000000000
Jan 30 22:56:58 moonraker kernel: [ 3587.196668] Call Trace:
Jan 30 22:56:58 moonraker kernel: [ 3587.196710] [<ffffffff802b7090>] enable_irq+0xb0/0x100
Jan 30 22:56:58 moonraker kernel: [ 3587.196741] [<ffffffff803a0cbc>] __netif_up+0xc/0x20
Jan 30 22:56:58 moonraker kernel: [ 3587.196769] ...

[Patrick Tescher (pat2man)]

Does anyone know of a fix? This bug has been open for a while...

[Adam Ashley (adamashley)]

I found switching to the official xen 2.6.18 kernel stopped the problem from occurring. As mine is purely a server I could get by without the other niceties of the ubuntu kernels

[Craig Ward (craig-ward)]

Did this ever get fixed? I'm seeing the same problems on a Dell PowerEdge 1850 when starting some domU's, causing the entire box to hang:

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: ------------[ cut here ]------------

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: kernel BUG at drivers/xen/core/evtchn.c:481!

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: invalid opcode: 0000 [#1]

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: SMP

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: CPU: 3

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: EIP is at retrigger+0x1f/0x35

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: eax: 00000000 ebx: 02080000 ecx: 00000024 edx: f55f6000

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: esi: c03186a0 edi: 00000113 ebp: 00000000 esp: c0399eb0

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: ds: 007b es: 007b ss: 0069

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: Process xenwatch (pid: 17, ti=c0398000 task=c0ea4550 task.ti=c0398000)

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: Stack: c013b225 c03186a0 00000113 c03186c8 c013af79 e00e5ec0 00000000 00000000

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: e00e5ec0 c02173c4 00000000 c02177a4 c0210933 00000010 00000000 0000020b

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: 0000020a 00000000 00000000 ed5c1236 c02e67a4 ee31e000 00000000 00000002

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: Call Trace:

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: Code: ee 85 f6 75 96 58 5a 5b 5e 5f 5d c3 0f b7 0c 85 40 b8 37 c0 8b 15 84 19 2d c0 85 c9 74 1d 0f a3 8a 80 08 00 00 19 c0 85 c0 75 08 <0f> 0b e1 01 92 1a 2b c0 f0 0f ab 8a 00 08 00 00 b8 01 00 00 00

Message from syslogd@badger at Mon Feb 25 18:00:35 2008 ...
badger kernel: EIP: [<c020c41e>] retrigger+0x1f/0x35 SS:ESP 0069:c0399eb0

[Dimitrios Symeonidis (azimout)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. You reported this bug a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue for you. Can you try with the latest Ubuntu release? Thanks in advance.

[Erik de Castro Lopo (erikd)]

I can't test this because I don't use Xen any more.

This can be closed.

[Dimitrios Symeonidis (azimout)]

closing as won't fix