Colin, you're not kidding, there's a lot more debug information in this one. Here's the part that looks important: [ 289.167264] dmu_object_free: object = 0x8319, dn->dn_type = 196 [ 289.167659] dmu_object_free: object = 0x8318, dn->dn_type = 13 [ 289.167922] dmu_object_free: object = 0x8317, dn->dn_type = 15 [ 289.168197] dmu_object_free: object = 0x8316, dn->dn_type = 12 [ 289.168500] dmu_object_free: object = 0x861b, dn->dn_type = 195 [ 289.351934] dmu_object_free: object = 0x5ef6, dn->dn_type = 50 [ 289.353287] dmu_object_free: object = 0x8553, dn->dn_type = 37 [ 289.353704] dmu_object_free: object = 0x65bb, dn->dn_type = 196 [ 289.917945] dmu_object_free: object = 0x210, dn->dn_type = 19 [ 289.918616] dmu_object_free: object = 0x211, dn->dn_type = 32 [ 289.919025] dnone_free_ramge: nblks = 0, trunc = 1, len = 0xffffffffffffffff, blkshift = 0 [ 289.919442] dnode_free_range: nblks == 0, len == 18446744073709551615 , off=0 [ 289.919443] range_tree_clear: size == 0 [ 289.920690] range_tree_find_impl: size == 0 [ 289.921123] VERIFY(size != 0) failed [ 289.921566] PANIC at range_tree.c:319:range_tree_find_impl() [ 289.922043] Showing stack for process 28938 [ 289.922047] CPU: 15 PID: 28938 Comm: receive_writer Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 289.922049] Hardware name: Supermicro SSG-6038R-E1CR16L/X10DRH-iT, BIOS 2.0 12/17/2015 [ 289.922050] Call Trace: [ 289.922064] dump_stack+0x6d/0x8e [ 289.922082] spl_dumpstack+0x42/0x50 [spl] [ 289.922089] spl_panic+0xc8/0x110 [spl] [ 289.922098] ? console_unlock+0x2a1/0x560 [ 289.922102] ? vprintk_emit+0x104/0x2c0 [ 289.922106] ? vprintk_default+0x29/0x50 [ 289.922111] ? vprintk_func+0x47/0xc0 [ 289.922114] ? printk+0x52/0x6e [ 289.922201] range_tree_find_impl+0x94/0xa0 [zfs] [ 289.922205] ? printk+0x52/0x6e [ 289.922268] range_tree_clear+0x54/0x70 [zfs] [ 289.922322] dnode_free_range+0x15b/0x720 [zfs] [ 289.922372] dmu_object_free+0x80/0xd0 [zfs] [ 289.922420] dmu_free_long_object+0x9f/0xc0 [zfs] [ 289.922469] receive_freeobjects.isra.12+0x7a/0x100 [zfs] [ 289.922517] receive_writer_thread+0x6d2/0xa60 [zfs] [ 289.922521] ? set_curr_task_fair+0x2b/0x60 [ 289.922528] ? spl_kmem_free+0x33/0x40 [spl] [ 289.922535] ? kfree+0x168/0x180 [ 289.922581] ? receive_free.isra.13+0xc0/0xc0 [zfs] [ 289.922589] thread_generic_wrapper+0x74/0x90 [spl] [ 289.922594] kthread+0x121/0x140 [ 289.922600] ? __thread_exit+0x20/0x20 [spl] [ 289.922603] ? kthread_create_worker_on_cpu+0x70/0x70 [ 289.922610] ret_from_fork+0x35/0x40 [ 484.358156] INFO: task txg_quiesce:3136 blocked for more than 120 seconds. [ 484.358778] Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 484.359320] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.359898] txg_quiesce D 0 3136 2 0x80000000 [ 484.359902] Call Trace: [ 484.359914] __schedule+0x24e/0x880 [ 484.359918] schedule+0x2c/0x80 [ 484.359935] cv_wait_common+0x11e/0x140 [spl] [ 484.359942] ? wait_woken+0x80/0x80 [ 484.359951] __cv_wait+0x15/0x20 [spl] [ 484.360056] txg_quiesce_thread+0x2cb/0x3d0 [zfs] [ 484.360124] ? txg_delay+0x1b0/0x1b0 [zfs] [ 484.360132] thread_generic_wrapper+0x74/0x90 [spl] [ 484.360138] kthread+0x121/0x140 [ 484.360144] ? __thread_exit+0x20/0x20 [spl] [ 484.360147] ? kthread_create_worker_on_cpu+0x70/0x70 [ 484.360152] ret_from_fork+0x35/0x40 [ 484.360171] INFO: task zfs:28847 blocked for more than 120 seconds. [ 484.360776] Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 484.361403] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.362092] zfs D 0 28847 28844 0x80000080 [ 484.362095] Call Trace: [ 484.362101] __schedule+0x24e/0x880 [ 484.362104] schedule+0x2c/0x80 [ 484.362114] cv_wait_common+0x11e/0x140 [spl] [ 484.362119] ? wait_woken+0x80/0x80 [ 484.362135] __cv_wait+0x15/0x20 [spl] [ 484.362196] dmu_recv_stream+0xa51/0xef0 [zfs] [ 484.362274] zfs_ioc_recv_impl+0x306/0x1100 [zfs] [ 484.362325] ? dbuf_read+0x34a/0x940 [zfs] [ 484.362377] ? dbuf_rele+0x36/0x40 [zfs] [ 484.362386] ? _cond_resched+0x19/0x40 [ 484.362457] zfs_ioc_recv_new+0x33d/0x410 [zfs] [ 484.362472] ? spl_kmem_alloc_impl+0xe5/0x1a0 [spl] [ 484.362485] ? spl_vmem_alloc+0x19/0x20 [spl] [ 484.362501] ? nv_alloc_sleep_spl+0x1f/0x30 [znvpair] [ 484.362511] ? nv_mem_zalloc.isra.0+0x2e/0x40 [znvpair] [ 484.362517] ? nvlist_xalloc.part.2+0x50/0xb0 [znvpair] [ 484.362581] zfsdev_ioctl+0x451/0x610 [zfs] [ 484.362587] do_vfs_ioctl+0xa8/0x630 [ 484.362592] ? __audit_syscall_entry+0xbc/0x110 [ 484.362599] ? syscall_trace_enter+0x1da/0x2d0 [ 484.362602] SyS_ioctl+0x79/0x90 [ 484.362606] do_syscall_64+0x73/0x130 [ 484.362610] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 484.362613] RIP: 0033:0x7f53da8085d7 [ 484.362615] RSP: 002b:00007ffc72d58f68 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 484.362618] RAX: ffffffffffffffda RBX: 0000000000005a46 RCX: 00007f53da8085d7 [ 484.362620] RDX: 00007ffc72d58f80 RSI: 0000000000005a46 RDI: 0000000000000006 [ 484.362621] RBP: 00007ffc72d58f80 R08: 00007f53daadde20 R09: 0000000000000000 [ 484.362623] R10: 000055ac5cec6010 R11: 0000000000000246 R12: 00007ffc72d5c5b0 [ 484.362624] R13: 0000000000000006 R14: 000055ac5cecdf10 R15: 000000000000000c [ 484.362633] INFO: task receive_writer:28938 blocked for more than 120 seconds. [ 484.363327] Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 484.364048] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 484.364805] receive_writer D 0 28938 2 0x80000080 [ 484.364808] Call Trace: [ 484.364812] __schedule+0x24e/0x880 [ 484.364816] schedule+0x2c/0x80 [ 484.364826] spl_panic+0xfa/0x110 [spl] [ 484.364832] ? console_unlock+0x2a1/0x560 [ 484.364837] ? vprintk_emit+0x104/0x2c0 [ 484.364841] ? vprintk_default+0x29/0x50 [ 484.364847] ? vprintk_func+0x47/0xc0 [ 484.364850] ? printk+0x52/0x6e [ 484.364918] range_tree_find_impl+0x94/0xa0 [zfs] [ 484.364922] ? printk+0x52/0x6e [ 484.364985] range_tree_clear+0x54/0x70 [zfs] [ 484.365042] dnode_free_range+0x15b/0x720 [zfs] [ 484.365093] dmu_object_free+0x80/0xd0 [zfs] [ 484.365143] dmu_free_long_object+0x9f/0xc0 [zfs] [ 484.365195] receive_freeobjects.isra.12+0x7a/0x100 [zfs] [ 484.365246] receive_writer_thread+0x6d2/0xa60 [zfs] [ 484.365250] ? set_curr_task_fair+0x2b/0x60 [ 484.365257] ? spl_kmem_free+0x33/0x40 [spl] [ 484.365265] ? kfree+0x168/0x180 [ 484.365313] ? receive_free.isra.13+0xc0/0xc0 [zfs] [ 484.365321] thread_generic_wrapper+0x74/0x90 [spl] [ 484.365325] kthread+0x121/0x140 [ 484.365332] ? __thread_exit+0x20/0x20 [spl] [ 484.365334] ? kthread_create_worker_on_cpu+0x70/0x70 [ 484.365339] ret_from_fork+0x35/0x40 [ 605.188646] INFO: task txg_quiesce:3136 blocked for more than 120 seconds. [ 605.189595] Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 605.190434] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 605.191302] txg_quiesce D 0 3136 2 0x80000000 [ 605.191306] Call Trace: [ 605.191316] __schedule+0x24e/0x880 [ 605.191320] schedule+0x2c/0x80 [ 605.191332] cv_wait_common+0x11e/0x140 [spl] [ 605.191339] ? wait_woken+0x80/0x80 [ 605.191347] __cv_wait+0x15/0x20 [spl] [ 605.191432] txg_quiesce_thread+0x2cb/0x3d0 [zfs] [ 605.191493] ? txg_delay+0x1b0/0x1b0 [zfs] [ 605.191501] thread_generic_wrapper+0x74/0x90 [spl] [ 605.191505] kthread+0x121/0x140 [ 605.191511] ? __thread_exit+0x20/0x20 [spl] [ 605.191514] ? kthread_create_worker_on_cpu+0x70/0x70 [ 605.191519] ret_from_fork+0x35/0x40 [ 605.191544] INFO: task zfs:28847 blocked for more than 120 seconds. [ 605.192434] Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 605.193375] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 605.194330] zfs D 0 28847 28844 0x80000080 [ 605.194333] Call Trace: [ 605.194339] __schedule+0x24e/0x880 [ 605.194343] schedule+0x2c/0x80 [ 605.194352] cv_wait_common+0x11e/0x140 [spl] [ 605.194356] ? wait_woken+0x80/0x80 [ 605.194363] __cv_wait+0x15/0x20 [spl] [ 605.194414] dmu_recv_stream+0xa51/0xef0 [zfs] [ 605.194482] zfs_ioc_recv_impl+0x306/0x1100 [zfs] [ 605.194521] ? dbuf_read+0x34a/0x940 [zfs] [ 605.194561] ? dbuf_rele+0x36/0x40 [zfs] [ 605.194564] ? _cond_resched+0x19/0x40 [ 605.194626] zfs_ioc_recv_new+0x33d/0x410 [zfs] [ 605.194636] ? spl_kmem_alloc_impl+0xe5/0x1a0 [spl] [ 605.194645] ? spl_vmem_alloc+0x19/0x20 [spl] [ 605.194655] ? nv_alloc_sleep_spl+0x1f/0x30 [znvpair] [ 605.194660] ? nv_mem_zalloc.isra.0+0x2e/0x40 [znvpair] [ 605.194666] ? nvlist_xalloc.part.2+0x50/0xb0 [znvpair] [ 605.194723] zfsdev_ioctl+0x451/0x610 [zfs] [ 605.194729] do_vfs_ioctl+0xa8/0x630 [ 605.194733] ? __audit_syscall_entry+0xbc/0x110 [ 605.194739] ? syscall_trace_enter+0x1da/0x2d0 [ 605.194742] SyS_ioctl+0x79/0x90 [ 605.194745] do_syscall_64+0x73/0x130 [ 605.194750] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 605.194753] RIP: 0033:0x7f53da8085d7 [ 605.194754] RSP: 002b:00007ffc72d58f68 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 605.194757] RAX: ffffffffffffffda RBX: 0000000000005a46 RCX: 00007f53da8085d7 [ 605.194759] RDX: 00007ffc72d58f80 RSI: 0000000000005a46 RDI: 0000000000000006 [ 605.194760] RBP: 00007ffc72d58f80 R08: 00007f53daadde20 R09: 0000000000000000 [ 605.194761] R10: 000055ac5cec6010 R11: 0000000000000246 R12: 00007ffc72d5c5b0 [ 605.194763] R13: 0000000000000006 R14: 000055ac5cecdf10 R15: 000000000000000c [ 605.194777] INFO: task receive_writer:28938 blocked for more than 120 seconds. [ 605.195760] Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 605.196789] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 605.197838] receive_writer D 0 28938 2 0x80000080 [ 605.197841] Call Trace: [ 605.197846] __schedule+0x24e/0x880 [ 605.197850] schedule+0x2c/0x80 [ 605.197858] spl_panic+0xfa/0x110 [spl] [ 605.197863] ? console_unlock+0x2a1/0x560 [ 605.197867] ? vprintk_emit+0x104/0x2c0 [ 605.197872] ? vprintk_default+0x29/0x50 [ 605.197876] ? vprintk_func+0x47/0xc0 [ 605.197879] ? printk+0x52/0x6e [ 605.197939] range_tree_find_impl+0x94/0xa0 [zfs] [ 605.197943] ? printk+0x52/0x6e [ 605.198000] range_tree_clear+0x54/0x70 [zfs] [ 605.198049] dnode_free_range+0x15b/0x720 [zfs] [ 605.198094] dmu_object_free+0x80/0xd0 [zfs] [ 605.198136] dmu_free_long_object+0x9f/0xc0 [zfs] [ 605.198180] receive_freeobjects.isra.12+0x7a/0x100 [zfs] [ 605.198222] receive_writer_thread+0x6d2/0xa60 [zfs] [ 605.198226] ? set_curr_task_fair+0x2b/0x60 [ 605.198233] ? spl_kmem_free+0x33/0x40 [spl] [ 605.198238] ? kfree+0x168/0x180 [ 605.198278] ? receive_free.isra.13+0xc0/0xc0 [zfs] [ 605.198285] thread_generic_wrapper+0x74/0x90 [spl] [ 605.198289] kthread+0x121/0x140 [ 605.198295] ? __thread_exit+0x20/0x20 [spl] [ 605.198297] ? kthread_create_worker_on_cpu+0x70/0x70 [ 605.198302] ret_from_fork+0x35/0x40 [ 726.019648] INFO: task txg_quiesce:3136 blocked for more than 120 seconds. [ 726.020809] Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 726.021920] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.023063] txg_quiesce D 0 3136 2 0x80000000 [ 726.023069] Call Trace: [ 726.023078] __schedule+0x24e/0x880 [ 726.023082] schedule+0x2c/0x80 [ 726.023094] cv_wait_common+0x11e/0x140 [spl] [ 726.023100] ? wait_woken+0x80/0x80 [ 726.023108] __cv_wait+0x15/0x20 [spl] [ 726.023205] txg_quiesce_thread+0x2cb/0x3d0 [zfs] [ 726.023275] ? txg_delay+0x1b0/0x1b0 [zfs] [ 726.023283] thread_generic_wrapper+0x74/0x90 [spl] [ 726.023287] kthread+0x121/0x140 [ 726.023293] ? __thread_exit+0x20/0x20 [spl] [ 726.023296] ? kthread_create_worker_on_cpu+0x70/0x70 [ 726.023301] ret_from_fork+0x35/0x40 [ 726.023324] INFO: task zfs:28847 blocked for more than 120 seconds. [ 726.024528] Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 726.025721] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.026956] zfs D 0 28847 28844 0x80000080 [ 726.026959] Call Trace: [ 726.026965] __schedule+0x24e/0x880 [ 726.026969] schedule+0x2c/0x80 [ 726.026979] cv_wait_common+0x11e/0x140 [spl] [ 726.026984] ? wait_woken+0x80/0x80 [ 726.026991] __cv_wait+0x15/0x20 [spl] [ 726.027051] dmu_recv_stream+0xa51/0xef0 [zfs] [ 726.027126] zfs_ioc_recv_impl+0x306/0x1100 [zfs] [ 726.027174] ? dbuf_read+0x34a/0x940 [zfs] [ 726.027222] ? dbuf_rele+0x36/0x40 [zfs] [ 726.027225] ? _cond_resched+0x19/0x40 [ 726.027294] zfs_ioc_recv_new+0x33d/0x410 [zfs] [ 726.027305] ? spl_kmem_alloc_impl+0xe5/0x1a0 [spl] [ 726.027313] ? spl_vmem_alloc+0x19/0x20 [spl] [ 726.027323] ? nv_alloc_sleep_spl+0x1f/0x30 [znvpair] [ 726.027329] ? nv_mem_zalloc.isra.0+0x2e/0x40 [znvpair] [ 726.027336] ? nvlist_xalloc.part.2+0x50/0xb0 [znvpair] [ 726.027400] zfsdev_ioctl+0x451/0x610 [zfs] [ 726.027405] do_vfs_ioctl+0xa8/0x630 [ 726.027409] ? __audit_syscall_entry+0xbc/0x110 [ 726.027414] ? syscall_trace_enter+0x1da/0x2d0 [ 726.027417] SyS_ioctl+0x79/0x90 [ 726.027420] do_syscall_64+0x73/0x130 [ 726.027425] entry_SYSCALL_64_after_hwframe+0x3d/0xa2 [ 726.027428] RIP: 0033:0x7f53da8085d7 [ 726.027430] RSP: 002b:00007ffc72d58f68 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [ 726.027432] RAX: ffffffffffffffda RBX: 0000000000005a46 RCX: 00007f53da8085d7 [ 726.027434] RDX: 00007ffc72d58f80 RSI: 0000000000005a46 RDI: 0000000000000006 [ 726.027436] RBP: 00007ffc72d58f80 R08: 00007f53daadde20 R09: 0000000000000000 [ 726.027437] R10: 000055ac5cec6010 R11: 0000000000000246 R12: 00007ffc72d5c5b0 [ 726.027438] R13: 0000000000000006 R14: 000055ac5cecdf10 R15: 000000000000000c [ 726.027451] INFO: task receive_writer:28938 blocked for more than 120 seconds. [ 726.028733] Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 726.030021] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 726.031350] receive_writer D 0 28938 2 0x80000080 [ 726.031355] Call Trace: [ 726.031360] __schedule+0x24e/0x880 [ 726.031364] schedule+0x2c/0x80 [ 726.031373] spl_panic+0xfa/0x110 [spl] [ 726.031378] ? console_unlock+0x2a1/0x560 [ 726.031382] ? vprintk_emit+0x104/0x2c0 [ 726.031386] ? vprintk_default+0x29/0x50 [ 726.031390] ? vprintk_func+0x47/0xc0 [ 726.031393] ? printk+0x52/0x6e [ 726.031462] range_tree_find_impl+0x94/0xa0 [zfs] [ 726.031480] ? printk+0x52/0x6e [ 726.031543] range_tree_clear+0x54/0x70 [zfs] [ 726.031604] dnode_free_range+0x15b/0x720 [zfs] [ 726.031663] dmu_object_free+0x80/0xd0 [zfs] [ 726.031715] dmu_free_long_object+0x9f/0xc0 [zfs] [ 726.031771] receive_freeobjects.isra.12+0x7a/0x100 [zfs] [ 726.031826] receive_writer_thread+0x6d2/0xa60 [zfs] [ 726.031834] ? set_curr_task_fair+0x2b/0x60 [ 726.031844] ? spl_kmem_free+0x33/0x40 [spl] [ 726.031852] ? kfree+0x168/0x180 [ 726.031899] ? receive_free.isra.13+0xc0/0xc0 [zfs] [ 726.031911] thread_generic_wrapper+0x74/0x90 [spl] [ 726.031918] kthread+0x121/0x140 [ 726.031924] ? __thread_exit+0x20/0x20 [spl] [ 726.031930] ? kthread_create_worker_on_cpu+0x70/0x70 [ 726.031937] ret_from_fork+0x35/0x40 [ 846.850892] INFO: task txg_quiesce:3136 blocked for more than 120 seconds. [ 846.852296] Tainted: P OE 4.15.0-101-generic #102-Ubuntu [ 846.853681] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 846.855129] txg_quiesce D 0 3136 2 0x80000000 [ 846.855133] Call Trace: [ 846.855142] __schedule+0x24e/0x880 [ 846.855146] schedule+0x2c/0x80 [ 846.855158] cv_wait_common+0x11e/0x140 [spl] [ 846.855167] ? wait_woken+0x80/0x80 [ 846.855180] __cv_wait+0x15/0x20 [spl] [ 846.855279] txg_quiesce_thread+0x2cb/0x3d0 [zfs] [ 846.855358] ? txg_delay+0x1b0/0x1b0 [zfs] [ 846.855368] thread_generic_wrapper+0x74/0x90 [spl] [ 846.855372] kthread+0x121/0x140 [ 846.855378] ? __thread_exit+0x20/0x20 [spl] [ 846.855381] ? kthread_create_worker_on_cpu+0x70/0x70 [ 846.855386] ret_from_fork+0x35/0x40 [ 3610.314028] dmu_object_free: object = 0x1e3d01, dn->dn_type = 19 [ 3610.316196] dmu_object_free: object = 0x1e2b0e, dn->dn_type = 19 [ 3610.323363] dmu_object_free: object = 0x1e57e7, dn->dn_type = 19 [ 3610.324468] dmu_object_free: object = 0x1e57e8, dn->dn_type = 19 [ 3610.325620] dmu_object_free: object = 0x1e57e9, dn->dn_type = 19 [ 3610.326732] dmu_object_free: object = 0x1e57ea, dn->dn_type = 19 [ 3610.327690] dmu_object_free: object = 0x1e57eb, dn->dn_type = 19 [ 3610.328713] dmu_object_free: object = 0x1e57ec, dn->dn_type = 19 [ 3610.329682] dmu_object_free: object = 0x1e57ed, dn->dn_type = 19 [ 3610.330666] dmu_object_free: object = 0x1e57ee, dn->dn_type = 19 And the full dmesg is attached. Thanks