Looking through the various instances reported here, we have a couple which have a real stack trace attached: Aug 12 11:32:30 quadpc kernel: [101268.516775] Eeek! page_mapcount(page) went negative! (-1) Aug 12 11:32:30 quadpc kernel: [101268.516785] page pfn = 30b48 Aug 12 11:32:30 quadpc kernel: [101268.516787] page->flags = 40080014 Aug 12 11:32:30 quadpc kernel: [101268.516789] page->count = 0 Aug 12 11:32:30 quadpc kernel: [101268.516790] page->mapping = 00000000 Aug 12 11:32:30 quadpc kernel: [101268.516818] vma->vm_ops = 0x0 Aug 12 11:32:30 quadpc kernel: [101268.516836] ------------[ cut here ]------------ Aug 12 11:32:30 quadpc kernel: [101268.516842] kernel BUG at /build/buildd/linux-2.6.24/mm/rmap.c:631! Aug 12 11:32:30 quadpc kernel: [101268.516846] invalid opcode: 0000 [#2] SMP Aug 12 11:32:30 quadpc kernel: [101268.516852] Modules linked in: vmnet(P) vmmon(P) rfcomm l2cap bluetooth parport_pc ppdev ipv6 acpi_cpufreq cpufreq_userspace cpufreq_ondemand cpufreq_powersave cpufreq_conservative cpufreq_stats freq_table sbs sbshc video output container dock battery xt_limit xt_tcpudp iptable_mangle ipt_LOG ipt_MASQUERADE nf_nat ipt_TOS ipt_REJECT nf_conntrack_irc nf_conntrack_ftp nf_conntrack_ipv4 xt_state nf_conntrack iptable_filter ip_tables x_tables nls_iso8859_1 nls_cp437 vfat fat ac coretemp w83627ehf hwmon_vid sbp2 lp parport loop snd_hda_intel snd_pcm_oss snd_mixer_oss snd_pcm snd_page_alloc snd_hwdep snd_seq_dummy snd_seq_oss ov511 sr_mod videodev snd_seq_midi nvidia(P) cdrom v4l2_common snd_rawmidi atl1 i2c_core v4l1_compat snd_seq_midi_event snd_seq snd_timer snd_seq_device intel_agp iTCO_wdt button snd shpchp pci_hotplug agpgart iTCO_vendor_support soundcore pcspkr evdev usbhid hid usb_storage ext3 jbd mbcache sg sd_mod libusual pata_jmicron pata_acpi ata_generic ata_piix ahci oh Aug 12 11:32:30 quadpc kernel: i1394 libata scsi_mod ieee1394 via_rhine mii uhci_hcd ehci_hcd usbcore thermal processor fan fbcon tileblit font bitblit softcursor fuse Aug 12 11:32:30 quadpc kernel: [101268.517058] Aug 12 11:32:30 quadpc kernel: [101268.517063] Pid: 11524, comm: hadam3_um_5.03_ Tainted: P B D (2.6.24-19-generic #1) Aug 12 11:32:30 quadpc kernel: [101268.517067] EIP: 0060:[page_remove_rmap+0xf9/0x110] EFLAGS: 00010246 CPU: 2 Aug 12 11:32:30 quadpc kernel: [101268.517074] EIP is at page_remove_rmap+0xf9/0x110 Aug 12 11:32:30 quadpc kernel: [101268.517079] EAX: 00000000 EBX: c1616900 ECX: 00000046 EDX: 00000000 Aug 12 11:32:30 quadpc kernel: [101268.517083] ESI: e4059dc0 EDI: 78d41000 EBP: dcd381c0 ESP: cb4e9ec8 Aug 12 11:32:30 quadpc kernel: [101268.517088] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Aug 12 11:32:30 quadpc kernel: [101268.517092] Process hadam3_um_5.03_ (pid: 11524, ti=cb4e8000 task=e35c05c0 task.ti=cb4e8000) Aug 12 11:32:30 quadpc kernel: [101268.517097] Stack: c03b377a 00000000 00000020 fff98504 c017c398 00000000 30b48067 c1f99080 Aug 12 11:32:30 quadpc kernel: [101268.517112] c04782a0 c20202b0 79084fff 74bed000 e4059dc0 cb4e9f60 00000000 002ab000 Aug 12 11:32:30 quadpc kernel: [101268.517126] 00000000 79000000 cb07f78c cb07f78c c20202a0 c1616900 fffffebf ffffffff Aug 12 11:32:30 quadpc kernel: [101268.517141] Call Trace: Aug 12 11:32:30 quadpc kernel: [101268.517153] [unmap_vmas+0x348/0x610] unmap_vmas+0x348/0x610 Aug 12 11:32:30 quadpc kernel: [101268.517208] [unmap_region+0x95/0x120] unmap_region+0x95/0x120 Aug 12 11:32:30 quadpc kernel: [101268.517230] [do_munmap+0x147/0x1f0] do_munmap+0x147/0x1f0 Aug 12 11:32:30 quadpc kernel: [101268.517257] [sys_munmap+0x3c/0x60] sys_munmap+0x3c/0x60 Aug 12 11:32:30 quadpc kernel: [101268.517269] [sysenter_past_esp+0x6b/0xa9] sysenter_past_esp+0x6b/0xa9 Aug 12 11:32:30 quadpc kernel: [101268.517293] [unix_stream_sendmsg+0x160/0x390] unix_stream_sendmsg+0x160/0x390 Aug 12 11:32:30 quadpc kernel: [101268.517311] ======================= Aug 12 11:32:30 quadpc kernel: [101268.517314] Code: 8b 46 40 8b 50 08 b8 aa 37 3b c0 e8 92 01 fd ff 8b 46 48 85 c0 74 14 8b 40 10 85 c0 74 0d 8b 50 2c b8 88 27 3b c0 e8 77 01 fd ff <0f> 0b eb fe 8b 53 0c eb 85 8d b4 26 00 00 00 00 8d bc 27 00 00 Aug 12 11:32:30 quadpc kernel: [101268.517389] EIP: [page_remove_rmap+0xf9/0x110] page_remove_rmap+0xf9/0x110 SS:ESP 0068:cb4e9ec8 Stack trace two: Sep 13 00:58:25 blackbox kernel: [ 595.750026] as[10514]: segfault at 49 ip 402e35d1 sp bf99fbc8 error 6 in libc-2.8.90.so[401a6000+158000] Sep 13 00:58:28 blackbox kernel: [ 598.888700] Eeek! page_mapcount(page) went negative! (-1) Sep 13 00:58:28 blackbox kernel: [ 598.888713] page pfn = 98d4b Sep 13 00:58:28 blackbox kernel: [ 598.888716] page->flags = 8000006c Sep 13 00:58:28 blackbox kernel: [ 598.888721] page->count = 1 Sep 13 00:58:28 blackbox kernel: [ 598.888724] page->mapping = f1cdd5b1 Sep 13 00:58:28 blackbox kernel: [ 598.888754] vma->vm_ops = 0x0 Sep 13 00:58:28 blackbox kernel: [ 598.888780] ------------[ cut here ]------------ Sep 13 00:58:28 blackbox kernel: [ 598.888784] kernel BUG at /build/buildd/linux-2.6.27/mm/rmap.c:662! Sep 13 00:58:28 blackbox kernel: [ 598.888789] invalid opcode: 0000 [#1] SMP Sep 13 00:58:28 blackbox kernel: [ 598.888797] Modules linked in: binfmt_misc radeon drm rfcomm l2cap bluetooth ppdev speedstep_lib cpufreq_conservative cpufreq_stats cpufreq_ondemand freq_table cpufreq_userspace cpufreq_powersave video output pci_slot sbs sbshc wmi container battery bridge stp ipv6 iptable_filter ip_tables x_tables ext2 ac tun sbp2 ieee1394 parport_pc lp parport loop snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device snd soundcore iTCO_wdt iTCO_vendor_support snd_page_alloc button intel_agp agpgart shpchp pci_hotplug evdev ext3 jbd mbcache sr_mod cdrom pata_acpi usbhid hid sg sd_mod crc_t10dif ata_piix ata_generic pata_via libata skge scsi_mod dock ehci_hcd uhci_hcd usbcore raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod thermal processor fan fbcon tileblit font bitblit softcursor uvesafb cn fuse [last unloaded: vboxdrv] Sep 13 00:58:28 blackbox kernel: [ 598.888958] Sep 13 00:58:28 blackbox kernel: [ 598.888962] Pid: 10514, comm: as Not tainted (2.6.27-3-generic #1) Sep 13 00:58:28 blackbox kernel: [ 598.888966] EIP: 0060:[] EFLAGS: 00210246 CPU: 1 Sep 13 00:58:28 blackbox kernel: [ 598.888976] EIP is at page_remove_rmap+0x10a/0x110 Sep 13 00:58:28 blackbox kernel: [ 598.888980] EAX: 00000000 EBX: c257de8c ECX: ffffffff EDX: 00200086 Sep 13 00:58:28 blackbox kernel: [ 598.888984] ESI: f79b28f0 EDI: 40304000 EBP: eadbfd9c ESP: eadbfd8c Sep 13 00:58:28 blackbox kernel: [ 598.888988] DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 Sep 13 00:58:28 blackbox kernel: [ 598.888992] Process as (pid: 10514, ti=eadbe000 task=f50f9920 task.ti=eadbe000) Sep 13 00:58:28 blackbox kernel: [ 598.888995] Stack: c04475d3 f1cdd5b1 c257de8c fffa5c10 eadbfde4 c0195a9e 98d4b067 c1000000 Sep 13 00:58:28 blackbox kernel: [ 598.889008] 00000007 c2ac0364 f79b28f0 c2b1e6c0 98d4b067 f5a0ac00 00000000 fffffffd Sep 13 00:58:28 blackbox kernel: [ 598.889008] 00000001 eadbfde0 c2b1e6d0 f5afa400 40400000 40301000 eadbfe3c c0196b30 Sep 13 00:58:28 blackbox kernel: [ 598.889008] Call Trace: Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? zap_pte_range+0x1fe/0x3b0 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? unmap_vmas+0x1a0/0x360 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? exit_mmap+0x8e/0x110 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? mmput+0x2e/0xa0 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? exit_mm+0xf9/0x130 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? do_exit+0x13d/0x350 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? do_group_exit+0x35/0xa0 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? get_signal_to_deliver+0x183/0x3a0 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? do_notify_resume+0x78/0x160 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? alloc_fd+0xe0/0x100 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? putname+0x2a/0x40 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? putname+0x2a/0x40 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? do_sys_open+0xda/0x100 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? sys_open+0x2e/0x40 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? do_page_fault+0x0/0x720 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? work_notifysig+0x13/0x23 Sep 13 00:58:28 blackbox kernel: [ 598.889008] [] ? do_boot_cpu+0x240/0x40a Sep 13 00:58:28 blackbox kernel: [ 598.889008] ======================= Sep 13 00:58:28 blackbox kernel: [ 598.889008] Code: c0 74 0d 8b 50 08 b8 03 76 44 c0 e8 11 fa ff ff 8b 46 48 85 c0 74 14 8b 40 10 85 c0 74 0d 8b 50 2c b8 10 61 44 c0 e8 f6 f9 ff ff <0f> 0b eb fe 66 90 55 89 e5 57 56 53 83 ec 20 e8 82 69 f6 ff 89 Sep 13 00:58:28 blackbox kernel: [ 598.889008] EIP: [] page_remove_rmap+0x10a/0x110 SS:ESP 0068:eadbfd8c Sep 13 00:58:28 blackbox kernel: [ 598.889315] ---[ end trace 43fa6cc85e252223 ]--- It should be noted also that we see this sort of segfault just before a number of the panics: Aug 24 22:00:19 blackbox kernel: [ 92.151743] Xorg[5751]: segfault at f00100e8 eip b7f00643 esp bfa5f58c error 5 Aug 24 22:00:19 blackbox kernel: [ 92.156963] Eeek! page_mapcount(page) went negative! (-1) Again: ep 25 15:04:27 xanimedes kernel: [ 556.216627] kdeinit4[6942]: segfault at 10 rip 7f73a77ff5d0 rsp 7fffb238f2b0 error 6 Sep 25 15:04:27 xanimedes kernel: [ 556.216672] Eeek! page_mapcount(page) went negative! (-1) Sep 25 15:04:27 xanimedes kernel: [ 556.216675] page pfn = 121514 Sep 25 15:04:27 xanimedes kernel: [ 556.216677] page->flags = 200000000000014 Sep 25 15:04:27 xanimedes kernel: [ 556.216679] page->count = 0 Sep 25 15:04:27 xanimedes kernel: [ 556.216681] page->mapping = 0000000000000000 Sep 25 15:04:27 xanimedes kernel: [ 556.216701] vma->vm_ops = 0x0 If Matts report is also the same bug then this seems to be occurring from back in 2.6.20 thru 2.6.28-rc8 at least.