Gutsy xen-3.1/kernel-2.6.22 dom0 kernel panic

[Géza Gémes]

Binary package hint: linux-image-2.6.22-10-xen

Runing xen-3.1 and kernel 2.6.22-xen on Gutsy (current) and Feisty (backported) causes kernel panics especially when compliling kernel modules.
For example:
...../linux-ubuntu-modules-2.6.22-2.6.22$dpkg-buildpackage -rfakeroot
gives:
.....
  CC [M] /home/administrator/linux/linux-ubuntu-modules-2.6.22-2.6.22/debian/build/build-386/wireless/iwlwifi/base-3945.o

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.855443] ------------[ cut here ]------------

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.855864] invalid opcode: 0000 [#1]

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.856058] SMP

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.864782] CPU: 2

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.864783] EIP: 0061:[xen_l2_entry_update+148/160] Not tainted VLI

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.864785] EFLAGS: 00010282 (2.6.22-10-xen #1)

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.865318] EIP is at xen_l2_entry_update+0x94/0xa0

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.865495] eax: ffffffea ebx: eb5ebdf8 ecx: 00000001 edx: 00000000

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.865675] esi: 00007ff0 edi: 00000000 ebp: ebd54c80 esp: eb5ebdf0

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.865855] ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0069

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.866035] Process as (pid: 10120, ti=eb5ea000 task=c2998550 task.ti=eb5ea000)

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.866218] Stack: 11c04067 00000000 53aed210 00000000 11c04067 00000000 00000000 11c04067

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.867768] 00000000 c015fb99 eb4ed210 ebd54c80 6d204000 00000000 00000000 00000000

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.869350] 00000000 ebd54c80 c0163241 00000061 80000000 ed7c5a04 46d6a62d ed796094

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.870900] Call Trace:

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.871247] [__pte_alloc+665/784] __pte_alloc+0x299/0x310

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.871589] [__handle_mm_fault+913/6928] __handle_mm_fault+0x391/0x1b10

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.871940] [scheduler_tick+243/256] scheduler_tick+0xf3/0x100

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.872314] [timer_interrupt+890/1824] timer_interrupt+0x37a/0x720

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.873004] [do_page_fault+0/3744] do_page_fault+0x0/0xea0

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.872657] [release_pages+390/496] release_pages+0x186/0x1f0

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.873342] [do_page_fault+769/3744] do_page_fault+0x301/0xea0

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.873676] [vma_merge+326/464] vma_merge+0x146/0x1d0

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.874025] [do_munmap+390/480] do_munmap+0x186/0x1e0

Message from syslogd@xen0 at Thu Aug 30 13:12:45 2007 ...
xen0 kernel: [ 505.874375] [do_page_fault+0/3744] do_page_fault+0x0/0xea0

Message from syslogd@xen0 at Thu Aug 30 13:12:46 2007 ...
xen0 kernel: [ 505.874708] [error_code+53/64] error_code+0x35/0x40

Message from syslogd@xen0 at Thu Aug 30 13:12:46 2007 ...
xen0 kernel: [ 505.875050] [svc_bind+128/496] svc_bind+0x80/0x1f0

Message from syslogd@xen0 at Thu Aug 30 13:12:46 2007 ...
xen0 kernel: [ 505.875414] =======================

Message from syslogd@xen0 at Thu Aug 30 13:12:46 2007 ...
xen0 kernel: [ 505.875602] Code: 89 74 24 08 be f0 7f 00 00 89 54 24 14 31 d2 89 44 24 10 e8 5f 7f fe ff 85 c0 78 0f 83 c4 18 5b 5e 5f c3 39 c2 72 9b 0f 0b eb fe <0f> 0b eb fe 90 8d b4 26 00 00 00 00 56 89 c2 53 83 ec 0c c1 ea

Message from syslogd@xen0 at Thu Aug 30 13:12:46 2007 ...
xen0 kernel: [ 505.885826] EIP: [xen_l2_entry_update+148/160] xen_l2_entry_update+0x94/0xa0 SS:ESP 0069:eb5ebdf0

In the syslog there is:

Aug 30 13:12:45 xen0 kernel: [ 505.855443] ------------[ cut here ]------------
Aug 30 13:12:45 xen0 kernel: [ 505.855658] kernel BUG at /home/administrator/linux/linux-source-2.6.22-2.6.22/debian/build/custom-source-xen/arch/i386/mm/hypervisor.c:71!
Aug 30 13:12:45 xen0 kernel: [ 505.855864] invalid opcode: 0000 [#1]
Aug 30 13:12:45 xen0 kernel: [ 505.856058] SMP
Aug 30 13:12:45 xen0 kernel: [ 505.856567] Modules linked in: button battery container ac dock sbs video ipv6 bridge lp inte
l_rng parport_pc ipg serial_core iTCO_wdt iTCO_vendor_support parport pcspkr shpchp psmouse serio_raw pci_hotplug e7xxx_edac
edac_mc tsdev evdev ext3 jbd mbcache sg sd_mod ide_cd cdrom ata_piix ata_generic libata floppy e100 mii e1000 sym53c8xx scsi_
transport_spi scsi_mod uhci_hcd piix usbcore thermal processor fan capability commoncap
Aug 30 13:12:45 xen0 kernel: [ 505.864782] CPU: 2
Aug 30 13:12:45 xen0 kernel: [ 505.864783] EIP: 0061:[xen_l2_entry_update+148/160] Not tainted VLI
Aug 30 13:12:45 xen0 kernel: [ 505.864785] EFLAGS: 00010282 (2.6.22-10-xen #1)
Aug 30 13:12:45 xen0 kernel: [ 505.865318] EIP is at xen_l2_entry_update+0x94/0xa0
Aug 30 13:12:45 xen0 kernel: [ 505.865495] eax: ffffffea ebx: eb5ebdf8 ecx: 00000001 edx: 00000000
Aug 30 13:12:45 xen0 kernel: [ 505.865675] esi: 00007ff0 edi: 00000000 ebp: ebd54c80 esp: eb5ebdf0
Aug 30 13:12:45 xen0 kernel: [ 505.865855] ds: 007b es: 007b fs: 00d8 gs: 0033 ss: 0069
Aug 30 13:12:45 xen0 kernel: [ 505.866035] Process as (pid: 10120, ti=eb5ea000 task=c2998550 task.ti=eb5ea000)
Aug 30 13:12:45 xen0 kernel: [ 505.866218] Stack: 11c04067 00000000 53aed210 00000000 11c04067 00000000 00000000 11c04067
Aug 30 13:12:45 xen0 kernel: [ 505.867768] 00000000 c015fb99 eb4ed210 ebd54c80 6d204000 00000000 00000000 00000000
Aug 30 13:12:45 xen0 kernel: [ 505.869350] 00000000 ebd54c80 c0163241 00000061 80000000 ed7c5a04 46d6a62d ed796094
Aug 30 13:12:45 xen0 kernel: [ 505.870900] Call Trace:
Aug 30 13:12:45 xen0 kernel: [ 505.871247] [__pte_alloc+665/784] __pte_alloc+0x299/0x310
Aug 30 13:12:45 xen0 kernel: [ 505.871589] [__handle_mm_fault+913/6928] __handle_mm_fault+0x391/0x1b10
Aug 30 13:12:45 xen0 kernel: [ 505.871940] [scheduler_tick+243/256] scheduler_tick+0xf3/0x100
Aug 30 13:12:45 xen0 kernel: [ 505.872314] [timer_interrupt+890/1824] timer_interrupt+0x37a/0x720
Aug 30 13:12:45 xen0 kernel: [ 505.872657] [release_pages+390/496] release_pages+0x186/0x1f0
Aug 30 13:12:45 xen0 kernel: [ 505.873004] [do_page_fault+0/3744] do_page_fault+0x0/0xea0
Aug 30 13:12:45 xen0 kernel: [ 505.873342] [do_page_fault+769/3744] do_page_fault+0x301/0xea0
Aug 30 13:12:45 xen0 kernel: [ 505.873676] [vma_merge+326/464] vma_merge+0x146/0x1d0
Aug 30 13:12:45 xen0 kernel: [ 505.874025] [do_munmap+390/480] do_munmap+0x186/0x1e0
Aug 30 13:12:45 xen0 kernel: [ 505.874375] [do_page_fault+0/3744] do_page_fault+0x0/0xea0
Aug 30 13:12:46 xen0 kernel: [ 505.874708] [error_code+53/64] error_code+0x35/0x40
Aug 30 13:12:46 xen0 kernel: [ 505.875050] [svc_bind+128/496] svc_bind+0x80/0x1f0
Aug 30 13:12:46 xen0 kernel: [ 505.875414] =======================
Aug 30 13:12:46 xen0 kernel: [ 505.875602] Code: 89 74 24 08 be f0 7f 00 00 89 54 24 14 31 d2 89 44 24 10 e8 5f 7f fe ff 85
c0 78 0f 83 c4 18 5b 5e 5f c3 39 c2 72 9b 0f 0b eb fe <0f> 0b eb fe 90 8d b4 26 00 00 00 00 56 89 c2 53 83 ec 0c c1 ea
Aug 30 13:12:46 xen0 kernel: [ 505.885826] EIP: [xen_l2_entry_update+148/160] xen_l2_entry_update+0x94/0xa0 SS:ESP 0069:eb5e
bdf0

while sudo xm dmesg reports:

(XEN) mm.c:1752:d0 Bad type (saw e8000001 != exp 20000000) for mfn 11c04 (pfn 6d204)
(XEN) mm.c:644:d0 Attempt to create linear p.t. with write perms

[Chuck Short (zulcss)]

There is a newer kernel coming soon can you try that when it comes out.

Thanks
chuck

[Géza Gémes (geza-kzsdabas)]

Is the new kernel accessible (downloadable) somewhere?
I would be happy to test it.

Thanks

Geza

[Chuck Short (zulcss)]

Please try this kernel:

http://kernel.ubuntu.com/~zul/linux-image-2.6.22-10-xen_2.6.22-10.30_i386.deb

thanks
chuck

[Géza Gémes (geza-kzsdabas)]

At least it boots, however I couldn't stress test it because it has intermittent problems with the ext3 filesystems I have on the box (they are going to get remounted read only after a few successful writes) in the syslog there is:
PCI-DMA: Out of SW-IOMMU space for 8192 bytes at device 0000:03:01.0
PCI-DMA: Out of SW-IOMMU space for 8192 bytes at device 0000:03:01.0
PCI-DMA: Out of SW-IOMMU space for 8192 bytes at device 0000:03:01.0
PCI-DMA: Out of SW-IOMMU space for 8192 bytes at device 0000:03:01.0
sd 0:0:1:0: [sda] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
end_request: I/O error, dev sda, sector 23451627
Buffer I/O error on device sda3, logical block 1599
lost page write due to I/O error on sda3
sd 0:0:1:0: [sda] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
end_request: I/O error, dev sda, sector 23452395
sd 0:0:1:0: [sda] Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK,SUGGEST_OK
end_request: I/O error, dev sda, sector 23453163
Aborting journal on device sda3.
ext3_abort called.

Thanks

geza
EXT3-fs error (device sda3): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-only

[John Leach (johnleach)]

I've had 50 guests running stress continuously for about 2 or 3 days with no problems like this.

This is with linux-image-2.6.22-12-xen, with Xen 3.1 backported from Gutsy to Feisty.

I've also run a kernel compile in a loop for a number of hours, again with no problems like this.

Sounds like this could well be fixed now. Géza, could you confirm and/or close this bug?

[Géza Gémes (geza-kzsdabas)]

Ok it really seems stable now.
I think the bug can be closed

[Alvin Cura (alvinc)]

I'm afraid I'm still seeing this bug. Or something similar.

I did not get a kernel panic, but my network died and stayed dead. I was, however, able to reboot.

Nov 4 16:44:43 xen1 -- MARK --
Nov 4 17:04:36 xen1 kernel: [19235.154900] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:00.0
Nov 4 17:04:36 xen1 kernel: [19235.157617] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:00.0
Nov 4 17:04:36 xen1 kernel: [19235.160469] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:00.0
Nov 4 17:04:36 xen1 kernel: [19235.163254] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:00.0
Nov 4 17:04:36 xen1 kernel: [19235.166076] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:00.0
Nov 4 17:04:36 xen1 kernel: [19235.168897] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:00.0
Nov 4 17:04:36 xen1 kernel: [19235.171727] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:00.0
Nov 4 17:04:36 xen1 kernel: [19235.174519] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:00.0
Nov 4 17:04:36 xen1 kernel: [19235.177666] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:00.0

7222 bytes is likely dictated by my ethernet interface being set for jumbo frames:

11: eth0: <BROADCAST,MULTICAST,UP,10000> mtu 7200 qdisc pfifo_fast qlen 1000
    link/ether 00:1b:fc:1f:33:40 brd ff:ff:ff:ff:ff:ff

this is on the following controller:

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 01)

I will proceed with some tests of using different NICs to try to isolate the behaviour.

[Alvin Cura (alvinc)]

More bad news. Problem is persisting, even with a different NIC (although also a Realtek, this one a PCI card instead of the onboard):

syslog:Nov 11 23:22:17 xen1 kernel: [ 8113.207809] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:05.0
syslog:Nov 11 23:22:40 xen1 kernel: [ 8136.691486] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:05.0
syslog:Nov 11 23:23:17 xen1 kernel: [ 8173.209884] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:05.0
syslog:Nov 11 23:24:17 xen1 kernel: [ 8233.214080] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:05.0
syslog:Nov 11 23:24:32 xen1 kernel: [ 8248.385552] PCI-DMA: Out of SW-IOMMU space for 7222 bytes at device 0000:03:05.0

03:05.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet (rev 10)

-----

Further research yielded the following thread in xen-devel:

http://lists.xensource.com/archives/html/xen-devel/2007-09/msg00138.html

However, more bad news, even setting swiotlb at boottime, the problem still occurs.

[marcolino (mas-marco)]

I' ve installed gutsy server and xen 3.10 as described here : http://www.howtoforge.com/ubuntu-7.10-server-install-xen-from-ubuntu-repositories
But when I try to create domU image, kernel panics and this is the log:

[ 116.420322] BUG: unable to handle kernel paging request at virtual address 00200204
[ 116.420507] printing eip:
[ 116.420582] ee1a68c9
[ 116.420585] 04a05000 -> *pde = 00000000:3e022001
[ 116.420666] 02c22000 -> *pme = 00000000:00000000
[ 116.420747] Oops: 0000 [#1]
[ 116.420821] SMP
[ 116.420995] Modules linked in: af_packet bridge parport_pc lp parport loop snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq iTCO_wdt snd_timer snd_seq_device intel_agp snd psmouse serial_core pcspkr serio_raw iTCO_vendor_support agpgart soundcore snd_page_alloc ipv6 evdev ext3 jbd mbcache usbhid hid sg sd_mod 8139cp ata_piix ata_generic floppy 8139too mii libata scsi_mod ehci_hcd uhci_hcd usbcore thermal processor fan fuse apparmor commoncap
[ 116.424050] CPU: 0
[ 116.424050] EIP: 0061:[<ee1a68c9>] Not tainted VLI
[ 116.424051] EFLAGS: 00010286 (2.6.22-14-xen #1)
[ 116.424314] EIP is at walk_page_buffers+0x19/0x70 [ext3]
[ 116.424400] eax: 00000000 ebx: fffbb83c ecx: fffbb83d edx: 00200200
[ 116.424486] esi: 00200200 edi: c183e4d4 ebp: ffffffff esp: cb071dfc
[ 116.424572] ds: 007b es: 007b fs: 00d8 gs: 0000 ss: 0069
[ 116.424655] Process pdflush (pid: 4819, ti=cb070000 task=c12f74c0 task.ti=cb070000)
[ 116.424743] Stack: 00000000 c20a08b8 c16634c0 c183e4d4 c20a08b8 c20a08b8 ee1a9ad1 00001000
[ 116.425272] 00000000 ee1a6920 00000000 c16634c0 cb071f68 c2c21cac 0000000d cb071f68
[ 116.425836] 00000000 c0157a38 c16634c0 c0157ef4 00000000 0000000e c0157a30 c2c21cac
[ 116.426377] Call Trace:
[ 116.426521] [<ee1a9ad1>] ext3_ordered_writepage+0xe1/0x1a0 [ext3]
[ 116.426661] [<ee1a6920>] bget_one+0x0/0x10 [ext3]
[ 116.426810] [<c0157a38>] __writepage+0x8/0x30
[ 116.426961] [<c0157ef4>] write_cache_pages+0x214/0x310
[ 116.427092] [<c0157a30>] __writepage+0x0/0x30
[ 116.427220] [<c0106e20>] do_IRQ+0x40/0x70
[ 116.427356] [<c0158010>] generic_writepages+0x20/0x30
[ 116.427492] [<c0158069>] do_writepages+0x49/0x50
[ 116.427620] [<c01980c3>] __writeback_single_inode+0x93/0x3c0
[ 116.427757] [<c019877e>] sync_sb_inodes+0x17e/0x240
[ 116.427888] [<c0198c49>] writeback_inodes+0x99/0xd0
[ 116.428028] [<c0158632>] background_writeout+0x92/0xb0
[ 116.428173] [<c0158ab0>] pdflush+0x0/0x260
[ 116.428307] [<c0158bf8>] pdflush+0x148/0x260
[ 116.428435] [<c01585a0>] background_writeout+0x0/0xb0
[ 116.428565] [<c0136312>] kthread+0x42/0x70
[ 116.428692] [<c01362d0>] kthread+0x0/0x70
[ 116.428820] [<c0105927>] kernel_thread_helper+0x7/0x10
[ 116.428951] =======================
[ 116.429028] Code: 74 f0 31 c0 39 cb 5b 0f 92 c0 c3 8d b4 26 00 00 00 00 55 57 89 d7 56 53 83 ec 08 89 0c 24 31 c9 89 44 24 04 8b 6a 14 8d 5c 0d 00 <8b> 72 04 3b 1c 24 76 2f 89 d8 29 e8 3b 44 24 1c 73 25 8b 44 24
[ 116.432334] EIP: [<ee1a68c9>] walk_page_buffers+0x19/0x70 [ext3] SS:ESP 0069:cb071dfc

I' m trying th...

[Peter Hellman (peter-segfault)]

I am getting the exakt same thing with 2.6.22-14-xen. DomU kraches when compiling kernel. A lot of crashes when hypervisor and Dom0 are 32-bit.
A bit better with 64-bit hypervisor and Dom0. DomU works better with 64-bit.
Very disturbing, Xen does not feel stable at all.
Custom build 2.6.18 from xen source works just fine, but I need ISCSI and GFS2, which deadlocks in 2.6.22-14-xen by the way.
Dell PE840 Quad-Core Xeon X3210 2.13GHz/2x4MB 1066FSB (105W)
Processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 15
model name : Intel(R) Xeon(R) CPU X3210 @ 2.13GHz
stepping : 11
cpu MHz : 2133.418
cache size : 4096 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
fpu : yes
fpu_exception : yes
cpuid level : 10
wp : yes
flags : fpu tsc msr pae mce cx8 apic mtrr mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx lm constant_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr lahf_lm
bogomips : 4268.85
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual

[Scott Wimer (scott-wimer)]

I had very similar problems with when trying to write to a new LVM logical volume using the xen kernel 2.6.22-14 (iirc). My hardware is a Dell PowerEdge 6950 with their PERC 5 RAID controller. It turns out that the IO scheduler changed and the megaraid_sas driver broke under "heavy" use (I've been able to break it using tar). Anyway, the fix for the driver was committed last October, but didn't make it into the latest 7.10 kernels. http://lkml.org/lkml/2007/10/3/281

So, in my case at least, that's what was causing my problem.

Scott

[Alvin Cura (alvinc)]

I was searching through bugs, and found this one un-updated.

I am no longer seeing these errors under Hardy 8.04.1 and its xen kernel. Is the right thing to do in this case to do an SRU for Gutsy? The kernel team needs to do this one, correct?

- alvinc