We're also seing this kernel bug in relation to NFS .. (not using XEN at all);
Aug 4 14:56:00 ko kernel: [263156.166854] PGD 8063 PUD 9063 PMD 0 Aug 4 14:56:00 ko kernel: [263156.166905] CPU 2 Aug 4 14:56:00 ko kernel: [263156.166926] Modules linked in: nfsd auth_rpcgss exportfs nfs lockd nfs_acl sunrpc autofs4 iptable_filter ip_tables x_ tables parport_pc lp parport loop joydev ipv6 psmouse evdev serio_raw pcspkr jedec_probe cfi_probe gen_probe mtd chipreg map_funcs i2c_nforce2 i2c_c ore button shpchp pci_hotplug k8temp ext3 jbd mbcache pata_amd pata_acpi sr_mod cdrom usb_storage libusual usbhid hid sg sd_mod mptsas mptscsih mptb ase scsi_transport_sas qla2xxx scsi_transport_fc e1000 scsi_tgt ata_generic ehci_hcd ohci_hcd libata scsi_mod usbcore thermal processor fan fbcon tileblit font bitblit softcursor fuse Aug 4 14:56:00 ko kernel: [263156.167218] Pid: 7150, comm: nfsd Not tainted 2.6.24-19-server #1 Aug 4 14:56:00 ko kernel: [263156.167246] RIP: 0010:[iov_iter_advance+0x66/0x80] [iov_iter_advance+0x66/0x80] iov_iter_advance+0x66/0x80 Aug 4 14:56:00 ko kernel: [263156.167292] RSP: 0018:ffff8101fa8478e8 EFLAGS: 00010246 Aug 4 14:56:00 ko kernel: [263156.167317] RAX: 0000000000000000 RBX: 0000000000001000 RCX: 0000000000000000 Aug 4 14:56:00 ko kernel: [263156.167360] RDX: 0000000000000000 RSI: 0000000000001000 RDI: ffff8101fa847980 Aug 4 14:56:00 ko kernel: [263156.167402] RBP: 0000000000001000 R08: 0000000000000000 R09: 0000000000000000 Aug 4 14:56:00 ko kernel: [263156.167444] R10: ffff810004000000 R11: 0000000000000000 R12: 00000005ba1f8000 Aug 4 14:56:00 ko kernel: [263156.167487] R13: 0000000000001000 R14: ffff8101760241d8 R15: 0000000000000000 Aug 4 14:56:00 ko kernel: [263156.167530] FS: 00007f9a5e8106e0(0000) GS:ffff8101fb042380(0000) knlGS:0000000000000000 Aug 4 14:56:00 ko kernel: [263156.167574] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b Aug 4 14:56:00 ko kernel: [263156.167600] CR2: ffff810004000008 CR3: 0000000055597000 CR4: 00000000000006e0 Aug 4 14:56:00 ko kernel: [263156.167643] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Aug 4 14:56:00 ko kernel: [263156.167686] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Aug 4 14:56:00 ko kernel: [263156.167730] Process nfsd (pid: 7150, threadinfo ffff8101fa846000, task ffff8101fb8947e0) Aug 4 14:56:00 ko kernel: [263156.167774] Stack: ffffffff80286a9e ffff8101e8df4600 ffff81033ec88680 0000000100000000 Aug 4 14:56:00 ko kernel: [263156.167823] ffff8101fa847b90 00000005ba0f9000 ffff8101fa847b10 ffff8101e8df4600 Aug 4 14:56:00 ko kernel: [263156.167871] ffff8101760241d8 ffffffff882559a0 ffff8101760240c8 ffffffff882559a0 Aug 4 14:56:00 ko kernel: [263156.167903] Call Trace: Aug 4 14:56:00 ko kernel: [263156.167940] [generic_file_buffered_write+0x1de/0x6b0] generic_file_buffered_write+0x1de/0x6b0 Aug 4 14:56:00 ko kernel: [263156.167979] [__generic_file_aio_write_nolock+0x24f/0x400] __generic_file_aio_write_nolock+0x24f/0x400 Aug 4 14:56:00 ko kernel: [263156.168028] [ext3:generic_file_aio_write+0x64/0x450] generic_file_aio_write+0x64/0xd0 Aug 4 14:56:00 ko kernel: [263156.168069] [ext3:ext3_file_write+0x23/0xc0] :ext3:ext3_file_write+0x23/0xc0 Aug 4 14:56:00 ko kernel: [263156.168100] [ext3:ext3_file_write+0x0/0xc0] :ext3:ext3_file_write+0x0/0xc0 Aug 4 14:56:00 ko kernel: [263156.168129] [do_sync_readv_writev+0xcb/0x110] do_sync_readv_writev+0xcb/0x110 Aug 4 14:56:00 ko kernel: [263156.168161] [__alloc_pages+0x9d/0x3d0] __alloc_pages+0x9d/0x3d0 Aug 4 14:56:00 ko kernel: [263156.168189] [<ffffffff80254260>] autoremove_wake_function+0x0/0x30 Aug 4 14:56:00 ko kernel: [263156.168222] [ext3:__get_free_pages+0x1b/0x1320] __get_free_pages+0x1b/0x40 Aug 4 14:56:00 ko kernel: [263156.168251] [do_readv_writev+0xfd/0x230] do_readv_writev+0xfd/0x230 Aug 4 14:56:00 ko kernel: [263156.168293] [<ffffffff8843c9e2>] :nfsd:nfsd_setuser+0x82/0x1f0 Aug 4 14:56:00 ko kernel: [263156.168327] [<ffffffff88434f6c>] :nfsd:nfsd_setuser_and_check_port+0x5c/0x70 Aug 4 14:56:00 ko kernel: [263156.168379] [<ffffffff88436790>] :nfsd:nfsd_vfs_write+0xf0/0x3d0 Aug 4 14:56:00 ko kernel: [263156.168410] [__dentry_open+0x1a7/0x200] __dentry_open+0x1a7/0x200 Aug 4 14:56:00 ko kernel: [263156.168443] [<ffffffff88436fbc>] :nfsd:nfsd_open+0x12c/0x1e0 Aug 4 14:56:00 ko kernel: [263156.168477] [<ffffffff8843731d>] :nfsd:nfsd_write+0x10d/0x120 Aug 4 14:56:00 ko kernel: [263156.168512] [<ffffffff8843f86b>] :nfsd:nfsd3_proc_write+0x10b/0x160 Aug 4 14:56:00 ko kernel: [263156.168547] [<ffffffff88432271>] :nfsd:nfsd_dispatch+0xb1/0x240 Aug 4 14:56:00 ko kernel: [263156.168597] [<ffffffff8838bdad>] :sunrpc:svc_process+0x47d/0x7e0 Aug 4 14:56:00 ko kernel: [263156.168627] [<ffffffff802362c0>] default_wake_function+0x0/0x10 Aug 4 14:56:00 ko kernel: [263156.168658] [__down_read+0x12/0xb1] __down_read+0x12/0xb1 Aug 4 14:56:00 ko kernel: [263156.168689] [<ffffffff88432810>] :nfsd:nfsd+0x0/0x2e0 Aug 4 14:56:00 ko kernel: [263156.168718] [<ffffffff8843299f>] :nfsd:nfsd+0x18f/0x2e0 Aug 4 14:56:00 ko kernel: [263156.168748] [child_rip+0xa/0x12] child_rip+0xa/0x12 Aug 4 14:56:00 ko kernel: [263156.168777] [<ffffffff88432810>] :nfsd:nfsd+0x0/0x2e0 Aug 4 14:56:00 ko kernel: [263156.168810] [<ffffffff88432810>] :nfsd:nfsd+0x0/0x2e0 Aug 4 14:56:00 ko kernel: [263156.168835] [child_rip+0x0/0x12] child_rip+0x0/0x12 Aug 4 14:56:00 ko kernel: [263156.168861] Aug 4 14:56:00 ko kernel: [263156.168879] Aug 4 14:56:00 ko kernel: [263156.168879] Code: 49 8b 52 08 49 89 d3 eb c4 4c 89 17 4c 89 4f 10 eb 99 0f 1f Aug 4 14:56:00 ko kernel: [263156.168991] RSP <ffff8101fa8478e8> Aug 4 14:56:00 ko kernel: [263156.169317] ---[ end trace 016407414603cb8e ]---
This is fixed by the most recent patch to 2.6.26.1: http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.26.1 commit a6b79bb88e6682d2739aa5b4db7184038bbb32ce
We're also seing this kernel bug in relation to NFS .. (not using XEN at all);
Aug 4 14:56:00 ko kernel: [263156.166854] PGD 8063 PUD 9063 PMD 0 iter_advance+ 0x66/0x80] [iov_iter_ advance+ 0x66/0x80] iov_iter_ advance+ 0x66/0x80 8478e8 EFLAGS: 00010246 0(0000) GS:ffff8101fb04 2380(0000) knlGS:000000000 0000000 file_buffered_ write+0x1de/ 0x6b0] generic_ file_buffered_ write+0x1de/ 0x6b0 file_aio_ write_nolock+ 0x24f/0x400] __generic_ file_aio_ write_nolock+ 0x24f/0x400 file_aio_ write+0x64/ 0x450] generic_ file_aio_ write+0x64/ 0xd0 file_write+ 0x23/0xc0] :ext3:ext3_ file_write+ 0x23/0xc0 file_write+ 0x0/0xc0] :ext3:ext3_ file_write+ 0x0/0xc0 readv_writev+ 0xcb/0x110] do_sync_ readv_writev+ 0xcb/0x110 pages+0x9d/ 0x3d0] __alloc_ pages+0x9d/ 0x3d0 260>] autoremove_ wake_function+ 0x0/0x30 _get_free_ pages+0x1b/ 0x1320] __get_free_ pages+0x1b/ 0x40 writev+ 0xfd/0x230] do_readv_ writev+ 0xfd/0x230 9e2>] :nfsd:nfsd_ setuser+ 0x82/0x1f0 f6c>] :nfsd:nfsd_ setuser_ and_check_ port+0x5c/ 0x70 790>] :nfsd:nfsd_ vfs_write+ 0xf0/0x3d0 open+0x1a7/ 0x200] __dentry_ open+0x1a7/ 0x200 fbc>] :nfsd:nfsd_ open+0x12c/ 0x1e0 31d>] :nfsd:nfsd_ write+0x10d/ 0x120 86b>] :nfsd:nfsd3_ proc_write+ 0x10b/0x160 271>] :nfsd:nfsd_ dispatch+ 0xb1/0x240 dad>] :sunrpc: svc_process+ 0x47d/0x7e0 2c0>] default_ wake_function+ 0x0/0x10 read+0x12/ 0xb1] __down_ read+0x12/ 0xb1 810>] :nfsd:nfsd+ 0x0/0x2e0 99f>] :nfsd:nfsd+ 0x18f/0x2e0 rip+0xa/ 0x12] child_rip+0xa/0x12 810>] :nfsd:nfsd+ 0x0/0x2e0 810>] :nfsd:nfsd+ 0x0/0x2e0 rip+0x0/ 0x12] child_rip+0x0/0x12
Aug 4 14:56:00 ko kernel: [263156.166905] CPU 2
Aug 4 14:56:00 ko kernel: [263156.166926] Modules linked in: nfsd auth_rpcgss exportfs nfs lockd nfs_acl sunrpc autofs4 iptable_filter ip_tables x_
tables parport_pc lp parport loop joydev ipv6 psmouse evdev serio_raw pcspkr jedec_probe cfi_probe gen_probe mtd chipreg map_funcs i2c_nforce2 i2c_c
ore button shpchp pci_hotplug k8temp ext3 jbd mbcache pata_amd pata_acpi sr_mod cdrom usb_storage libusual usbhid hid sg sd_mod mptsas mptscsih mptb
ase scsi_transport_sas qla2xxx scsi_transport_fc e1000 scsi_tgt ata_generic ehci_hcd ohci_hcd libata scsi_mod usbcore thermal processor fan fbcon tileblit font bitblit softcursor fuse
Aug 4 14:56:00 ko kernel: [263156.167218] Pid: 7150, comm: nfsd Not tainted 2.6.24-19-server #1
Aug 4 14:56:00 ko kernel: [263156.167246] RIP: 0010:[iov_
Aug 4 14:56:00 ko kernel: [263156.167292] RSP: 0018:ffff8101fa
Aug 4 14:56:00 ko kernel: [263156.167317] RAX: 0000000000000000 RBX: 0000000000001000 RCX: 0000000000000000
Aug 4 14:56:00 ko kernel: [263156.167360] RDX: 0000000000000000 RSI: 0000000000001000 RDI: ffff8101fa847980
Aug 4 14:56:00 ko kernel: [263156.167402] RBP: 0000000000001000 R08: 0000000000000000 R09: 0000000000000000
Aug 4 14:56:00 ko kernel: [263156.167444] R10: ffff810004000000 R11: 0000000000000000 R12: 00000005ba1f8000
Aug 4 14:56:00 ko kernel: [263156.167487] R13: 0000000000001000 R14: ffff8101760241d8 R15: 0000000000000000
Aug 4 14:56:00 ko kernel: [263156.167530] FS: 00007f9a5e8106e
Aug 4 14:56:00 ko kernel: [263156.167574] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Aug 4 14:56:00 ko kernel: [263156.167600] CR2: ffff810004000008 CR3: 0000000055597000 CR4: 00000000000006e0
Aug 4 14:56:00 ko kernel: [263156.167643] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 4 14:56:00 ko kernel: [263156.167686] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Aug 4 14:56:00 ko kernel: [263156.167730] Process nfsd (pid: 7150, threadinfo ffff8101fa846000, task ffff8101fb8947e0)
Aug 4 14:56:00 ko kernel: [263156.167774] Stack: ffffffff80286a9e ffff8101e8df4600 ffff81033ec88680 0000000100000000
Aug 4 14:56:00 ko kernel: [263156.167823] ffff8101fa847b90 00000005ba0f9000 ffff8101fa847b10 ffff8101e8df4600
Aug 4 14:56:00 ko kernel: [263156.167871] ffff8101760241d8 ffffffff882559a0 ffff8101760240c8 ffffffff882559a0
Aug 4 14:56:00 ko kernel: [263156.167903] Call Trace:
Aug 4 14:56:00 ko kernel: [263156.167940] [generic_
Aug 4 14:56:00 ko kernel: [263156.167979] [__generic_
Aug 4 14:56:00 ko kernel: [263156.168028] [ext3:generic_
Aug 4 14:56:00 ko kernel: [263156.168069] [ext3:ext3_
Aug 4 14:56:00 ko kernel: [263156.168100] [ext3:ext3_
Aug 4 14:56:00 ko kernel: [263156.168129] [do_sync_
Aug 4 14:56:00 ko kernel: [263156.168161] [__alloc_
Aug 4 14:56:00 ko kernel: [263156.168189] [<ffffffff80254
Aug 4 14:56:00 ko kernel: [263156.168222] [ext3:_
Aug 4 14:56:00 ko kernel: [263156.168251] [do_readv_
Aug 4 14:56:00 ko kernel: [263156.168293] [<ffffffff8843c
Aug 4 14:56:00 ko kernel: [263156.168327] [<ffffffff88434
Aug 4 14:56:00 ko kernel: [263156.168379] [<ffffffff88436
Aug 4 14:56:00 ko kernel: [263156.168410] [__dentry_
Aug 4 14:56:00 ko kernel: [263156.168443] [<ffffffff88436
Aug 4 14:56:00 ko kernel: [263156.168477] [<ffffffff88437
Aug 4 14:56:00 ko kernel: [263156.168512] [<ffffffff8843f
Aug 4 14:56:00 ko kernel: [263156.168547] [<ffffffff88432
Aug 4 14:56:00 ko kernel: [263156.168597] [<ffffffff8838b
Aug 4 14:56:00 ko kernel: [263156.168627] [<ffffffff80236
Aug 4 14:56:00 ko kernel: [263156.168658] [__down_
Aug 4 14:56:00 ko kernel: [263156.168689] [<ffffffff88432
Aug 4 14:56:00 ko kernel: [263156.168718] [<ffffffff88432
Aug 4 14:56:00 ko kernel: [263156.168748] [child_
Aug 4 14:56:00 ko kernel: [263156.168777] [<ffffffff88432
Aug 4 14:56:00 ko kernel: [263156.168810] [<ffffffff88432
Aug 4 14:56:00 ko kernel: [263156.168835] [child_
Aug 4 14:56:00 ko kernel: [263156.168861]
Aug 4 14:56:00 ko kernel: [263156.168879]
Aug 4 14:56:00 ko kernel: [263156.168879] Code: 49 8b 52 08 49 89 d3 eb c4 4c 89 17 4c 89 4f 10 eb 99 0f 1f
Aug 4 14:56:00 ko kernel: [263156.168991] RSP <ffff8101fa8478e8>
Aug 4 14:56:00 ko kernel: [263156.169317] ---[ end trace 016407414603cb8e ]---
This is fixed by the most recent patch to 2.6.26.1: www.kernel. org/pub/ linux/kernel/ v2.6/ChangeLog- 2.6.26. 1 2739aa5b4db7184 038bbb32ce
http://
commit a6b79bb88e6682d