Description of problem: I have 4 1TB disk in raid10. Everytime I run following fio load, after few hours I got BUG() like below. Btrfs mounted with compress=lzo,space_cache #v+ [write-test] rw=randwrite ioengine=libaio blocksize=4k iodepth=32 size=1512G #v- [64354.305737] ------------[ cut here ]------------ [64354.306009] kernel BUG at fs/btrfs/inode.c:1668! [64354.306009] invalid opcode: 0000 [#1] SMP [64354.306009] CPU 1 [64354.306009] Modules linked in: lockd bnep bluetooth rfkill snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm snd_timer snd soundcore iTCO_wdt tg3 parport_pc iTCO_vendor_support parport snd_page_alloc dcdbas serio_raw i2c_i801 microcode sunrpc uinput aoe btrfs zlib_deflate libcrc32c sata_promise i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [64354.330470] [64354.330470] Pid: 5149, comm: btrfs-endio-wri Tainted: G W 3.3.0-0.rc2.git3.2.fc17.x86_64 #1 Dell Inc. OptiPlex GX620 /0FH884 [64354.330470] RIP: 0010:[] [] insert_reserved_file_extent.constprop.50+0x251/0x260 [btrfs] [64354.330470] RSP: 0018:ffff8800757ffbd0 EFLAGS: 00010286 [64354.330470] RDX: 0000000000000000 RSI: ffffea00033c3700 RDI: 0000000000000246 [64354.330470] RBP: ffff8800757ffc60 R08: ffff8800cf0dc760 R09: 0000000100110002 [64354.330470] R10: ffff8800cf0dc7f0 R11: 0000000000000000 R12: ffff8800a453a290 [64354.330470] R13: ffff8800cf0dc1d8 R14: ffff8800a3a69868 R15: 0000002e8f0b2000 [64354.330470] FS: 0000000000000000(0000) GS:ffff8800db000000(0000) knlGS:0000000000000000 [64354.330470] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [64354.330470] CR2: 0000003238285240 CR3: 0000000023ba8000 CR4: 00000000000006e0 [64354.330470] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [64354.330470] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [64354.330470] Process btrfs-endio-wri (pid: 5149, threadinfo ffff8800757fe000, task ffff880097978000) [64354.330470] Stack: [64354.330470] 00ff8800757ffc00 0000000000001000 0000001d7ab1f000 ffff8800a453a290 [64354.330470] 0000000000001000 ffff8800cf207a90 ffff8800757ffc50 ffffffffa0123455 [64354.330470] ffff8800757ffc30 0000000000000000 00000000000000cc ffff8800a3a69868 [64354.330470] Call Trace: [64354.330470] [] ? start_transaction+0x85/0x3b0 [btrfs] [64354.330470] [] btrfs_finish_ordered_io+0x232/0x350 [btrfs] [64354.330470] [] btrfs_writepage_end_io_hook+0x4c/0x150 [btrfs] [64354.330470] [] end_bio_extent_writepage+0x122/0x170 [btrfs] [64354.330470] [] bio_endio+0x1d/0x40 [64354.330470] [] end_workqueue_fn+0x64/0x150 [btrfs] [64354.330470] [] worker_loop+0x149/0x5d0 [btrfs] [64354.330470] [] ? btrfs_queue_worker+0x310/0x310 [btrfs] [64354.330470] [] kthread+0xb7/0xc0 [64354.330470] [] ? trace_hardirqs_on+0xd/0x10 [64354.330470] [] kernel_thread_helper+0x4/0x10 [64354.330470] [] ? _raw_spin_unlock_irq+0x30/0x50 [64354.330470] [] ? retint_restore_args+0x13/0x13 [64354.330470] [] ? kthread_worker_fn+0x1a0/0x1a0 [64354.330470] [] ? gs_change+0x13/0x13 [64354.330470] Code: ff 84 0f 85 67 fe ff ff e9 5e fe ff ff 0f 1f 44 00 00 41 80 be 78 fb ff ff 84 75 9a eb 94 0f 1f 40 00 b8 f4 ff ff ff eb b9 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 [64354.330470] RIP [] insert_reserved_file_extent.constprop.50+0x251/0x260 [btrfs] [64354.330470] RSP [64354.557410] ---[ end trace 6288379729f3c564 ]--- [64354.650352] ------------[ cut here ]------------ [64354.651007] kernel BUG at fs/btrfs/inode.c:1668! [64354.651007] invalid opcode: 0000 [#2] SMP [64354.651007] CPU 0 [64354.651007] Modules linked in: lockd bnep bluetooth rfkill snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm snd_timer snd soundcore iTCO_wdt tg3 parport_pc iTCO_vendor_support parport snd_page_alloc dcdbas serio_raw i2c_i801 microcode sunrpc uinput aoe btrfs zlib_deflate libcrc32c sata_promise i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [64354.651007] [64354.651007] Pid: 5054, comm: btrfs-endio-wri Tainted: G D W 3.3.0-0.rc2.git3.2.fc17.x86_64 #1 Dell Inc. OptiPlex GX620 /0FH884 [64354.651007] RIP: 0010:[] [] insert_reserved_file_extent.constprop.50+0x251/0x260 [btrfs] [64354.651007] RSP: 0000:ffff88008a601bd0 EFLAGS: 00010286 [64354.651007] RDX: 0000000000000000 RSI: ffffea00033c3700 RDI: 0000000000000246 [64354.651007] RBP: ffff88008a601c60 R08: ffff8800cf0dc760 R09: 0000000100110003 [64354.651007] R10: ffff8800cf0dc7f0 R11: 0000000000000000 R12: ffff8800a453a290 [64354.651007] R13: ffff8800cf0dc3b0 R14: ffff8800a3a69868 R15: 0000002ea0ad9000 [64354.651007] FS: 0000000000000000(0000) GS:ffff8800dae00000(0000) knlGS:0000000000000000 [64354.651007] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [64354.651007] CR2: 0000003238a5d1d0 CR3: 00000000abeaf000 CR4: 00000000000006f0 [64354.651007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [64354.651007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [64354.651007] Process btrfs-endio-wri (pid: 5054, threadinfo ffff88008a600000, task ffff88008a852680) [64354.651007] Stack: [64354.651007] 00ff88008a601c00 0000000000001000 0000001d7ab2f000 ffff8800a453a290 [64354.651007] 0000000000001000 ffff8800cf207450 ffff88008a601c50 ffffffffa0123455 [64354.651007] ffff88008a601c30 0000000000000000 00000000000000cc ffff8800a3a69868 [64354.651007] Call Trace: [64354.651007] [] ? start_transaction+0x85/0x3b0 [btrfs] [64354.651007] [] btrfs_finish_ordered_io+0x232/0x350 [btrfs] [64354.651007] [] btrfs_writepage_end_io_hook+0x4c/0x150 [btrfs] [64354.651007] [] end_bio_extent_writepage+0x122/0x170 [btrfs] [64354.651007] [] bio_endio+0x1d/0x40 [64354.651007] [] end_workqueue_fn+0x64/0x150 [btrfs] [64354.651007] [] worker_loop+0x149/0x5d0 [btrfs] [64354.651007] [] ? btrfs_queue_worker+0x310/0x310 [btrfs] [64354.651007] [] kthread+0xb7/0xc0 [64354.651007] [] ? trace_hardirqs_on+0xd/0x10 [64354.651007] [] kernel_thread_helper+0x4/0x10 [64354.651007] [] ? _raw_spin_unlock_irq+0x30/0x50 [64354.651007] [] ? retint_restore_args+0x13/0x13 [64354.651007] [] ? kthread_worker_fn+0x1a0/0x1a0 [64354.651007] [] ? gs_change+0x13/0x13 [64354.651007] Code: ff 84 0f 85 67 fe ff ff e9 5e fe ff ff 0f 1f 44 00 00 41 80 be 78 fb ff ff 84 75 9a eb 94 0f 1f 40 00 b8 f4 ff ff ff eb b9 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 [64354.651007] RIP [] insert_reserved_file_extent.constprop.50+0x251/0x260 [btrfs] [64354.651007] RSP [64354.820765] ---[ end trace 6288379729f3c565 ]--- [64354.930890] ------------[ cut here ]------------ [64354.931006] kernel BUG at fs/btrfs/inode.c:1668! [64354.931006] invalid opcode: 0000 [#3] SMP [64354.931006] CPU 0 [64354.931006] Modules linked in: lockd bnep bluetooth rfkill snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm snd_timer snd soundcore iTCO_wdt tg3 parport_pc iTCO_vendor_support parport snd_page_alloc dcdbas serio_raw i2c_i801 microcode sunrpc uinput aoe btrfs zlib_deflate libcrc32c sata_promise i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [64354.931006] [64354.931006] Pid: 31070, comm: btrfs-endio-wri Tainted: G D W 3.3.0-0.rc2.git3.2.fc17.x86_64 #1 Dell Inc. OptiPlex GX620 /0FH884 [64354.931006] RIP: 0010:[] [] insert_reserved_file_extent.constprop.50+0x251/0x260 [btrfs] [64354.931006] RSP: 0000:ffff88005e5c1bd0 EFLAGS: 00010286 [64354.931006] RDX: 0000000000000000 RSI: ffffea00033c3700 RDI: 0000000000000246 [64354.931006] RBP: ffff88005e5c1c60 R08: ffff8800cf0dcb10 R09: 0000000100110004 [64354.931006] R10: ffff8800cf0dcba0 R11: 0000000000000000 R12: ffff8800a453a290 [64354.931006] R13: ffff8800cf0dd620 R14: ffff8800a3a69868 R15: 0000002ed4e12000 [64354.931006] FS: 0000000000000000(0000) GS:ffff8800dae00000(0000) knlGS:0000000000000000 [64354.931006] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [64354.931006] CR2: 0000003238a5d1d0 CR3: 00000000abeaf000 CR4: 00000000000006f0 [64354.931006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [64354.931006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [64354.931006] Process btrfs-endio-wri (pid: 31070, threadinfo ffff88005e5c0000, task ffff880000198000) [64354.931006] Stack: [64354.931006] 00ff88005e5c1c00 0000000000001000 000000129838f000 ffff8800a453a290 [64354.931006] 0000000000001000 ffff8800cf2064b0 ffff88005e5c1c50 ffffffffa0123455 [64354.931006] ffff88005e5c1c30 0000000000000000 00000000000000cc ffff8800a3a69868 [64354.931006] Call Trace: [64354.931006] [] ? start_transaction+0x85/0x3b0 [btrfs] [64354.931006] [] btrfs_finish_ordered_io+0x232/0x350 [btrfs] [64354.931006] [] btrfs_writepage_end_io_hook+0x4c/0x150 [btrfs] [64354.931006] [] end_bio_extent_writepage+0x122/0x170 [btrfs] [64354.931006] [] bio_endio+0x1d/0x40 [64354.931006] [] end_workqueue_fn+0x64/0x150 [btrfs] [64354.931006] [] worker_loop+0x149/0x5d0 [btrfs] [64354.931006] [] ? btrfs_queue_worker+0x310/0x310 [btrfs] [64354.931006] [] kthread+0xb7/0xc0 [64354.931006] [] ? trace_hardirqs_on+0xd/0x10 [64354.931006] [] kernel_thread_helper+0x4/0x10 [64354.931006] [] ? _raw_spin_unlock_irq+0x30/0x50 [64354.931006] [] ? retint_restore_args+0x13/0x13 [64354.931006] [] ? kthread_worker_fn+0x1a0/0x1a0 [64354.931006] [] ? gs_change+0x13/0x13 [64354.931006] Code: ff 84 0f 85 67 fe ff ff e9 5e fe ff ff 0f 1f 44 00 00 41 80 be 78 fb ff ff 84 75 9a eb 94 0f 1f 40 00 b8 f4 ff ff ff eb b9 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 [64354.931006] RIP [] insert_reserved_file_extent.constprop.50+0x251/0x260 [btrfs] [64354.931006] RSP [64355.103531] ---[ end trace 6288379729f3c566 ]--- [64355.214974] ------------[ cut here ]------------ [64355.215006] kernel BUG at fs/btrfs/inode.c:1668! [64355.215006] invalid opcode: 0000 [#4] SMP [64355.215006] CPU 0 [64355.215006] Modules linked in: lockd bnep bluetooth rfkill snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm snd_timer snd soundcore iTCO_wdt tg3 parport_pc iTCO_vendor_support parport snd_page_alloc dcdbas serio_raw i2c_i801 microcode sunrpc uinput aoe btrfs zlib_deflate libcrc32c sata_promise i915 drm_kms_helper drm i2c_algo_bit i2c_core video [last unloaded: scsi_wait_scan] [64355.215006] [64355.215006] Pid: 31073, comm: btrfs-endio-wri Tainted: G D W 3.3.0-0.rc2.git3.2.fc17.x86_64 #1 Dell Inc. OptiPlex GX620 /0FH884 [64355.215006] RIP: 0010:[] [] insert_reserved_file_extent.constprop.50+0x251/0x260 [btrfs] [64355.215006] RSP: 0000:ffff88002b969bd0 EFLAGS: 00010286 [64355.215006] RDX: 0000000000000000 RSI: ffffea00033c3700 RDI: 0000000000000246 [64355.215006] RBP: ffff88002b969c60 R08: ffff8800cf0dc938 R09: 0000000100110005 [64355.215006] R10: ffff8800cf0dc9c8 R11: 0000000000000000 R12: ffff8800a453a290 [64355.215006] R13: ffff8800cf0dcb10 R14: ffff8800a3a69868 R15: 0000002eeafd6000 [64355.215006] FS: 0000000000000000(0000) GS:ffff8800dae00000(0000) knlGS:0000000000000000 [64355.215006] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [64355.215006] CR2: 0000003238a5d1d0 CR3: 00000000abeaf000 CR4: 00000000000006f0 [64355.215006] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [64355.215006] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [64355.215006] Process btrfs-endio-wri (pid: 31073, threadinfo ffff88002b968000, task ffff880097044d00) [64355.215006] Stack: [64355.215006] 00ff88002b969c00 0000000000001000 0000001d7ab3f000 ffff8800a453a290 [64355.215006] 0000000000001000 ffff8800cf206190 ffff88002b969c50 ffffffffa0123455 [64355.215006] ffff88002b969c30 0000000000000000 00000000000000cc ffff8800a3a69868 [64355.215006] Call Trace: [64355.215006] [] ? start_transaction+0x85/0x3b0 [btrfs] [64355.215006] [] btrfs_finish_ordered_io+0x232/0x350 [btrfs] [64355.215006] [] btrfs_writepage_end_io_hook+0x4c/0x150 [btrfs] [64355.215006] [] end_bio_extent_writepage+0x122/0x170 [btrfs] [64355.215006] [] bio_endio+0x1d/0x40 [64355.215006] [] end_workqueue_fn+0x64/0x150 [btrfs] [64355.215006] [] worker_loop+0x149/0x5d0 [btrfs] [64355.215006] [] ? btrfs_queue_worker+0x310/0x310 [btrfs] [64355.215006] [] kthread+0xb7/0xc0 [64355.215006] [] ? trace_hardirqs_on+0xd/0x10 [64355.215006] [] kernel_thread_helper+0x4/0x10 [64355.215006] [] ? _raw_spin_unlock_irq+0x30/0x50 [64355.215006] [] ? retint_restore_args+0x13/0x13 [64355.215006] [] ? kthread_worker_fn+0x1a0/0x1a0 [64355.215006] [] ? gs_change+0x13/0x13 [64355.215006] Code: ff 84 0f 85 67 fe ff ff e9 5e fe ff ff 0f 1f 44 00 00 41 80 be 78 fb ff ff 84 75 9a eb 94 0f 1f 40 00 b8 f4 ff ff ff eb b9 0f 0b <0f> 0b 0f 0b 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 [64355.215006] RIP [] insert_reserved_file_extent.constprop.50+0x251/0x260 [btrfs] [64355.215006] RSP [64355.390309] ---[ end trace 6288379729f3c567 ]--- [64355.390512] btrfs-endio-wri used greatest stack depth: 1848 bytes left Version-Release number of selected component (if applicable): It's fedora16 with F17's latest kernel, currently 3.3.0-0.rc2.git3.2.fc17.x86_64 How reproducible: Always after few hours. Label: 'duo' uuid: 94b52e73-92a0-481f-a809-4acd24f83d93 Total devices 4 FS bytes used 1.50TB devid 1 size 931.51GB used 931.51GB path /dev/sdb devid 2 size 931.51GB used 806.02GB path /dev/sdc devid 3 size 931.51GB used 931.51GB path /dev/sdd devid 4 size 931.51GB used 931.51GB path /dev/sde btrfs f df /mnt/raid10test/ Data, RAID1: total=1.75TB, used=1.50TB System, RAID1: total=16.00MB, used=260.00KB System: total=4.00MB, used=0.00 Metadata, RAID1: total=3.50GB, used=3.25GB # LANG=C ls -lh /mnt/raid10test/; LANG=C df -h /mnt/raid10test/ total 1.5T -rw-r--r--. 1 root root 1.5T Feb 7 03:49 write-test.1.0 Filesystem Size Used Avail Use% Mounted on /dev/sdb 3.7T 3.1T 517G 86% /mnt/raid10test