Jan 29 16:43:24 vmhost kernel: [480258.278614] rmap_remove: ffff8801e927dfb0 80000002035b7263 0->BUG Jan 29 16:43:24 vmhost kernel: [480258.278932] ------------[ cut here ]------------ Jan 29 16:43:24 vmhost kernel: [480258.279129] kernel BUG at /build/buildd/linux-2.6.32/arch/x86/kvm/mmu.c:635! Jan 29 16:43:24 vmhost kernel: [480258.279397] invalid opcode: 0000 [#1] SMP Jan 29 16:43:24 vmhost kernel: [480258.279682] last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:05:05.0/local_cpus Jan 29 16:43:24 vmhost kernel: [480258.280012] CPU 0 Jan 29 16:43:24 vmhost kernel: [480258.280170] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs exportfs reiserfs ipt_MASQUERADE iptable_nat nf_na t nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack bonding ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_intel kvm fbcon tileblit font bitblit softcursor vga16fb vgastate radeon ttm drm_kms_helper drm i2c_algo_bit i3000_edac psmouse serio_raw ipmi_msghandler edac_core lp parport floppy e1000 e1000e 3w_9xxx Jan 29 16:43:24 vmhost kernel: [480258.285521] Pid: 1670, comm: kvm Not tainted 2.6.32-27-server #49-Ubuntu Jan 29 16:43:24 vmhost kernel: [480258.285780] RIP: 0010:[] [] rmap_remove+0x1b0/0x1c0 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286210] RSP: 0018:ffff88020f191a68 EFLAGS: 00010292 Jan 29 16:43:24 vmhost kernel: [480258.286428] RAX: 000000000000004c RBX: ffff8801e927dfb0 RCX: 0000000000000000 Jan 29 16:43:24 vmhost kernel: [480258.286526] RDX: 0000000000000000 RSI: 0000000000000082 RDI: 0000000000000246 Jan 29 16:43:24 vmhost kernel: [480258.286526] RBP: ffff88020f191a88 R08: 00000000ffffffff R09: ffff8800000a18b0 Jan 29 16:43:24 vmhost kernel: [480258.286526] R10: ffff88020f191758 R11: ffff880214599800 R12: ffff88020e23c000 Jan 29 16:43:24 vmhost kernel: [480258.286526] R13: 00000000002035b7 R14: ffff88009e6ab9a0 R15: 00000000000001f6 Jan 29 16:43:24 vmhost kernel: [480258.286526] FS: 00007fce604fd700(0000) GS:ffff880009000000(0000) knlGS:0000000000000000 Jan 29 16:43:24 vmhost kernel: [480258.286526] CS: 0010 DS: 002b ES: 002b CR0: 0000000080050033 Jan 29 16:43:24 vmhost kernel: [480258.286526] CR2: 00007fc56c6ae0b0 CR3: 0000000213d03000 CR4: 00000000000026e0 Jan 29 16:43:24 vmhost kernel: [480258.286526] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 29 16:43:24 vmhost kernel: [480258.286526] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 29 16:43:24 vmhost kernel: [480258.286526] Process kvm (pid: 1670, threadinfo ffff88020f190000, task ffff88020f345b80) Jan 29 16:43:24 vmhost kernel: [480258.286526] Stack: Jan 29 16:43:24 vmhost kernel: [480258.286526] ffff88009e6ab9a0 ffff88020e23c000 ffff8801e927dfb0 0000000000000000 Jan 29 16:43:24 vmhost kernel: [480258.286526] <0> ffff88020f191ac8 ffffffffa0220894 ffff88020f191ab8 ffff88020e23c000 Jan 29 16:43:24 vmhost kernel: [480258.286526] <0> ffff88009e6ab580 ffff88020e23f878 0000000000000001 00000000000e0000 Jan 29 16:43:24 vmhost kernel: [480258.286526] Call Trace: Jan 29 16:43:24 vmhost kernel: [480258.286526] [] kvm_mmu_zap_page+0x84/0x380 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] kvm_mmu_zap_all+0x38/0x70 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] kvm_arch_flush_shadow+0x16/0x30 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] __kvm_set_memory_region+0x6e1/0x770 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? paging64_page_fault+0x1f9/0x210 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? kvm_emulate_pio+0x174/0x1c0 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] kvm_set_memory_region+0x43/0x70 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] kvm_vm_ioctl_set_memory_region+0x1d/0x30 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] kvm_vm_ioctl+0x282/0x6e0 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? vmx_handle_exit+0xca/0x250 [kvm_intel] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? __vmx_load_host_state+0x1d2/0x240 [kvm_intel] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? vmx_vcpu_put+0xe/0x10 [kvm_intel] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? kvm_arch_vcpu_put+0x1b/0x50 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? vcpu_put+0x28/0x30 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? kvm_arch_vcpu_ioctl_run+0xb4/0x1d0 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? try_to_wake_up+0x2e6/0x470 Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? wake_up_state+0x10/0x20 Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? wake_futex+0x40/0x60 Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? futex_wake+0x112/0x130 Jan 29 16:43:24 vmhost kernel: [480258.286526] [] vfs_ioctl+0x22/0xa0 Jan 29 16:43:24 vmhost kernel: [480258.286526] [] do_vfs_ioctl+0x81/0x410 Jan 29 16:43:24 vmhost kernel: [480258.286526] [] ? sys_futex+0x7b/0x170 Jan 29 16:43:24 vmhost kernel: [480258.286526] [] sys_ioctl+0x81/0xa0 Jan 29 16:43:24 vmhost kernel: [480258.286526] [] system_call_fastpath+0x16/0x1b Jan 29 16:43:24 vmhost kernel: [480258.286526] Code: 8b 13 48 89 de 48 c7 c7 70 5c 23 a0 31 c0 e8 ff 7c 33 e1 0f 0b eb fe 48 8b 13 48 89 de 48 c7 c7 50 5c 23 a0 31 c0 e8 e7 7c 33 e1 <0f> 0b eb fe 48 8b 17 48 89 10 eb a8 0f 1f 40 00 55 48 89 e5 41 Jan 29 16:43:24 vmhost kernel: [480258.286526] RIP [] rmap_remove+0x1b0/0x1c0 [kvm] Jan 29 16:43:24 vmhost kernel: [480258.286526] RSP Jan 29 16:43:24 vmhost kernel: [480258.816905] ---[ end trace 39ac185951650917 ]--- Jan 29 16:44:33 vmhost kernel: [480327.671241] BUG: soft lockup - CPU#1 stuck for 61s! [ksmd:53] Jan 29 16:44:33 vmhost kernel: [480327.678895] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs exportfs reiserfs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack bonding ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_intel kvm fbcon tileblit font bitblit softcursor vga16fb vgastate radeon ttm drm_kms_helper drm i2c_algo_bit i3000_edac psmouse serio_raw ipmi_msghandler edac_core lp parport floppy e1000 e1000e 3w_9xxx Jan 29 16:44:33 vmhost kernel: [480327.744839] CPU 1: Jan 29 16:44:33 vmhost kernel: [480327.753302] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs exportfs reiserfs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack bonding ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_intel kvm fbcon tileblit font bitblit softcursor vga16fb vgastate radeon ttm drm_kms_helper drm i2c_algo_bit i3000_edac psmouse serio_raw ipmi_msghandler edac_core lp parport floppy e1000 e1000e 3w_9xxx Jan 29 16:44:33 vmhost kernel: [480327.823706] Pid: 53, comm: ksmd Tainted: G D 2.6.32-27-server #49-Ubuntu Jan 29 16:44:33 vmhost kernel: [480327.843983] RIP: 0010:[] [] __ticket_spin_lock+0xe/0x20 Jan 29 16:44:33 vmhost kernel: [480327.864312] RSP: 0018:ffff880216485c10 EFLAGS: 00000297 Jan 29 16:44:33 vmhost kernel: [480327.875527] RAX: 0000000000002b2a RBX: ffff880216485c10 RCX: 80000000d077b065 Jan 29 16:44:33 vmhost kernel: [480327.896984] RDX: 00007fce42dcb000 RSI: ffff88020f97ad80 RDI: ffff88020e23c000 Jan 29 16:44:33 vmhost kernel: [480327.919374] RBP: ffffffff81012cae R08: ffff88020f97b048 R09: 0000000000000040 Jan 29 16:44:33 vmhost kernel: [480327.940773] R10: ffff88020e0f70b0 R11: 0000000000000000 R12: 0000000000000000 Jan 29 16:44:33 vmhost kernel: [480327.962178] R13: ffff880000012080 R14: 0000000000015bc0 R15: 0000000000015bc0 Jan 29 16:44:33 vmhost kernel: [480327.984300] FS: 0000000000000000(0000) GS:ffff880009080000(0000) knlGS:0000000000000000 Jan 29 16:44:33 vmhost kernel: [480328.006146] CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b Jan 29 16:44:33 vmhost kernel: [480328.017186] CR2: 000000000236fd20 CR3: 0000000001001000 CR4: 00000000000026e0 Jan 29 16:44:33 vmhost kernel: [480328.040392] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 Jan 29 16:44:33 vmhost kernel: [480328.063416] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Jan 29 16:44:33 vmhost kernel: [480328.086604] Call Trace: Jan 29 16:44:33 vmhost kernel: [480328.098046] [] ? _spin_lock+0xe/0x20 Jan 29 16:44:33 vmhost kernel: [480328.109634] [] ? kvm_mmu_notifier_change_pte+0x2e/0x60 [kvm] Jan 29 16:44:33 vmhost kernel: [480328.131850] [] ? __mmu_notifier_change_pte+0x3e/0x90 Jan 29 16:44:33 vmhost kernel: [480328.143421] [] ? write_protect_page+0x15b/0x180 Jan 29 16:44:33 vmhost kernel: [480328.154502] [] ? try_to_merge_one_page+0x96/0x120 Jan 29 16:44:33 vmhost kernel: [480328.165366] [] ? try_to_merge_with_ksm_page+0x94/0xa0 Jan 29 16:44:33 vmhost kernel: [480328.176247] [] ? try_to_merge_two_pages+0x181/0x1a0 Jan 29 16:44:33 vmhost kernel: [480328.186744] [] ? cmp_and_merge_page+0x13e/0x1f0 Jan 29 16:44:33 vmhost kernel: [480328.197214] [] ? ksm_do_scan+0x30/0xb0 Jan 29 16:44:33 vmhost kernel: [480328.208074] [] ? ksm_scan_thread+0x72/0x160 Jan 29 16:44:33 vmhost kernel: [480328.218351] [] ? autoremove_wake_function+0x0/0x40 Jan 29 16:44:33 vmhost kernel: [480328.228525] [] ? ksm_scan_thread+0x0/0x160 Jan 29 16:44:33 vmhost kernel: [480328.238402] [] ? kthread+0x96/0xa0 Jan 29 16:44:33 vmhost kernel: [480328.248160] [] ? child_rip+0xa/0x20 Jan 29 16:44:33 vmhost kernel: [480328.257998] [] ? kthread+0x0/0xa0 Jan 29 16:44:33 vmhost kernel: [480328.267262] [] ? child_rip+0x0/0x20 Jan 29 16:45:01 vmhost CRON[32030]: (root) CMD (if [ -x /etc/munin/plugins/apt_all ]; then /etc/munin/plugins/apt_all update 7200 12 >/dev/null; elif [ -x /etc/munin/plugins/apt ]; then /etc/munin/plugins/apt update 7200 12 >/dev/null; fi) Jan 29 16:45:39 vmhost kernel: [480393.171240] BUG: soft lockup - CPU#1 stuck for 61s! [ksmd:53] Jan 29 16:45:39 vmhost kernel: [480393.180456] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs exportfs reiserfs ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack bonding ipt_REJECT xt_tcpudp iptable_filter ip_tables x_tables bridge stp kvm_intel kvm fbcon tileblit font bitblit softcursor vga16fb vgastate radeon ttm drm_kms_helper drm i2c_algo_bit i3000_edac psmouse serio_raw ipmi_msghandler edac_core lp parport floppy e1000 e1000e 3w_9xxx Jan 29 16:45:39 vmhost kernel: [480393.248579] CPU 1: