zfs recv PANIC at range_tree.c:304:range_tree_find_impl()

[Seth Arnold]

Hello, I believe these errors happened due to a zfs recv command that was executing at the time:

[10823702.582392] VERIFY(size != 0) failed
[10823702.582428] PANIC at range_tree.c:304:range_tree_find_impl()
[10823702.582463] Showing stack for process 693172
[10823702.582466] CPU: 7 PID: 693172 Comm: receive_writer Tainted: P O 4.15.0-60-generic #67-Ubuntu
[10823702.582466] Hardware name: Supermicro SSG-6038R-E1CR16L/X10DRH-iT, BIOS 2.0 12/17/2015
[10823702.582467] Call Trace:
[10823702.582475] dump_stack+0x63/0x8b
[10823702.582489] spl_dumpstack+0x42/0x50 [spl]
[10823702.582494] spl_panic+0xc8/0x110 [spl]
[10823702.582539] ? dbuf_dirty+0x43d/0x850 [zfs]
[10823702.582542] ? getrawmonotonic64+0x43/0xd0
[10823702.582544] ? getrawmonotonic64+0x43/0xd0
[10823702.582581] ? dmu_zfetch+0x49a/0x500 [zfs]
[10823702.582583] ? getrawmonotonic64+0x43/0xd0
[10823702.582619] ? dmu_zfetch+0x49a/0x500 [zfs]
[10823702.582621] ? mutex_lock+0x12/0x40
[10823702.582654] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs]
[10823702.582697] range_tree_find_impl+0x88/0x90 [zfs]
[10823702.582702] ? spl_kmem_zalloc+0xdc/0x1a0 [spl]
[10823702.582743] range_tree_clear+0x4f/0x60 [zfs]
[10823702.582780] dnode_free_range+0x11f/0x5a0 [zfs]
[10823702.582815] dmu_object_free+0x53/0x90 [zfs]
[10823702.582850] dmu_free_long_object+0x9f/0xc0 [zfs]
[10823702.582885] receive_freeobjects.isra.12+0x7a/0x100 [zfs]
[10823702.582918] receive_writer_thread+0x6d2/0xa60 [zfs]
[10823702.582920] ? set_curr_task_fair+0x2b/0x60
[10823702.582925] ? spl_kmem_free+0x33/0x40 [spl]
[10823702.582928] ? kfree+0x165/0x180
[10823702.582961] ? receive_free.isra.13+0xc0/0xc0 [zfs]
[10823702.582967] thread_generic_wrapper+0x74/0x90 [spl]
[10823702.582969] kthread+0x121/0x140
[10823702.582974] ? __thread_exit+0x20/0x20 [spl]
[10823702.582975] ? kthread_create_worker_on_cpu+0x70/0x70
[10823702.582978] ret_from_fork+0x35/0x40
[10823907.445420] INFO: task txg_quiesce:4485 blocked for more than 120 seconds.
[10823907.445486] Tainted: P O 4.15.0-60-generic #67-Ubuntu
[10823907.445535] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[10823907.445589] txg_quiesce D 0 4485 2 0x80000000
[10823907.445594] Call Trace:
[10823907.445608] __schedule+0x24e/0x880
[10823907.445613] schedule+0x2c/0x80
[10823907.445629] cv_wait_common+0x11e/0x140 [spl]
[10823907.445638] ? wait_woken+0x80/0x80
[10823907.445647] __cv_wait+0x15/0x20 [spl]
[10823907.445766] txg_quiesce_thread+0x2cb/0x3d0 [zfs]
[10823907.445835] ? txg_delay+0x1b0/0x1b0 [zfs]
[10823907.445843] thread_generic_wrapper+0x74/0x90 [spl]
[10823907.445848] kthread+0x121/0x140
[10823907.445854] ? __thread_exit+0x20/0x20 [spl]
[10823907.445857] ? kthread_create_worker_on_cpu+0x70/0x70
[10823907.445861] ret_from_fork+0x35/0x40
[10823907.445916] INFO: task zfs:688217 blocked for more than 120 seconds.
[10823907.445962] Tainted: P O 4.15.0-60-generic #67-Ubuntu
[10823907.446010] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[10823907.446063] zfs D 0 688217 688214 0x80000080
[10823907.446066] Call Trace:
[10823907.446071] __schedule+0x24e/0x880
[10823907.446075] schedule+0x2c/0x80
[10823907.446084] cv_wait_common+0x11e/0x140 [spl]
[10823907.446088] ? wait_woken+0x80/0x80
[10823907.446095] __cv_wait+0x15/0x20 [spl]
[10823907.446151] dmu_recv_stream+0xa51/0xef0 [zfs]
[10823907.446227] zfs_ioc_recv_impl+0x306/0x1100 [zfs]
[10823907.446232] ? ttwu_do_activate+0x77/0x80
[10823907.446303] zfs_ioc_recv_new+0x33d/0x410 [zfs]
[10823907.446312] ? spl_kmem_alloc_impl+0xe5/0x1a0 [spl]
[10823907.446320] ? spl_vmem_alloc+0x19/0x20 [spl]
[10823907.446332] ? nv_alloc_sleep_spl+0x1f/0x30 [znvpair]
[10823907.446338] ? nv_mem_zalloc.isra.0+0x2e/0x40 [znvpair]
[10823907.446344] ? nvlist_xalloc.part.2+0x50/0xb0 [znvpair]
[10823907.446409] zfsdev_ioctl+0x451/0x610 [zfs]
[10823907.446415] do_vfs_ioctl+0xa8/0x630
[10823907.446419] ? __audit_syscall_entry+0xbc/0x110
[10823907.446424] ? syscall_trace_enter+0x1da/0x2d0
[10823907.446426] SyS_ioctl+0x79/0x90
[10823907.446430] do_syscall_64+0x73/0x130
[10823907.446434] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[10823907.446437] RIP: 0033:0x7fc4006905d7
[10823907.446438] RSP: 002b:00007ffe7df61dc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[10823907.446441] RAX: ffffffffffffffda RBX: 0000000000005a46 RCX: 00007fc4006905d7
[10823907.446443] RDX: 00007ffe7df61de0 RSI: 0000000000005a46 RDI: 0000000000000006
[10823907.446444] RBP: 00007ffe7df61de0 R08: 00007fc400965ce0 R09: 0000000000000000
[10823907.446446] R10: 0000555c7adad010 R11: 0000000000000246 R12: 00007ffe7df65410
[10823907.446447] R13: 0000000000000006 R14: 0000555c7adb4ae0 R15: 000000000000000c
[10823907.446452] INFO: task receive_writer:693172 blocked for more than 120 seconds.
[10823907.446504] Tainted: P O 4.15.0-60-generic #67-Ubuntu
[10823907.446551] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[10823907.446604] receive_writer D 0 693172 2 0x80000080
[10823907.446607] Call Trace:
[10823907.446612] __schedule+0x24e/0x880
[10823907.446617] schedule+0x2c/0x80
[10823907.446626] spl_panic+0xfa/0x110 [spl]
[10823907.446677] ? dbuf_dirty+0x43d/0x850 [zfs]
[10823907.446683] ? getrawmonotonic64+0x43/0xd0
[10823907.446686] ? getrawmonotonic64+0x43/0xd0
[10823907.446740] ? dmu_zfetch+0x49a/0x500 [zfs]
[10823907.446744] ? getrawmonotonic64+0x43/0xd0
[10823907.446795] ? dmu_zfetch+0x49a/0x500 [zfs]
[10823907.446797] ? mutex_lock+0x12/0x40
[10823907.446844] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs]
[10823907.446907] range_tree_find_impl+0x88/0x90 [zfs]
[10823907.446915] ? spl_kmem_zalloc+0xdc/0x1a0 [spl]
[10823907.446974] range_tree_clear+0x4f/0x60 [zfs]
[10823907.447027] dnode_free_range+0x11f/0x5a0 [zfs]
[10823907.447077] dmu_object_free+0x53/0x90 [zfs]
[10823907.447125] dmu_free_long_object+0x9f/0xc0 [zfs]
[10823907.447174] receive_freeobjects.isra.12+0x7a/0x100 [zfs]
[10823907.447222] receive_writer_thread+0x6d2/0xa60 [zfs]
[10823907.447225] ? set_curr_task_fair+0x2b/0x60
[10823907.447233] ? spl_kmem_free+0x33/0x40 [spl]
[10823907.447240] ? kfree+0x165/0x180
[10823907.447286] ? receive_free.isra.13+0xc0/0xc0 [zfs]
[10823907.447294] thread_generic_wrapper+0x74/0x90 [spl]
[10823907.447297] kthread+0x121/0x140
[10823907.447303] ? __thread_exit+0x20/0x20 [spl]
[10823907.447305] ? kthread_create_worker_on_cpu+0x70/0x70
[10823907.447309] ret_from_fork+0x35/0x40
[10823950.069580] mpt3sas 0000:03:00.0: invalid short VPD tag 00 at offset 1
[10823952.753814] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[10823952.757865] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[10824028.276294] INFO: task txg_quiesce:4485 blocked for more than 120 seconds.
[10824028.276360] Tainted: P O 4.15.0-60-generic #67-Ubuntu
[10824028.276408] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[10824028.276463] txg_quiesce D 0 4485 2 0x80000000
[10824028.276467] Call Trace:
[10824028.276485] __schedule+0x24e/0x880
[10824028.276490] schedule+0x2c/0x80
[10824028.276507] cv_wait_common+0x11e/0x140 [spl]
[10824028.276516] ? wait_woken+0x80/0x80
[10824028.276524] __cv_wait+0x15/0x20 [spl]
[10824028.276636] txg_quiesce_thread+0x2cb/0x3d0 [zfs]
[10824028.276703] ? txg_delay+0x1b0/0x1b0 [zfs]
[10824028.276711] thread_generic_wrapper+0x74/0x90 [spl]
[10824028.276716] kthread+0x121/0x140
[10824028.276722] ? __thread_exit+0x20/0x20 [spl]
[10824028.276725] ? kthread_create_worker_on_cpu+0x70/0x70
[10824028.276731] ret_from_fork+0x35/0x40
[10824028.276817] INFO: task zfs:688217 blocked for more than 120 seconds.
[10824028.276864] Tainted: P O 4.15.0-60-generic #67-Ubuntu
[10824028.276911] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[10824028.276965] zfs D 0 688217 688214 0x80000080
[10824028.276968] Call Trace:
[10824028.276974] __schedule+0x24e/0x880
[10824028.276978] schedule+0x2c/0x80
[10824028.276987] cv_wait_common+0x11e/0x140 [spl]
[10824028.276991] ? wait_woken+0x80/0x80
[10824028.276998] __cv_wait+0x15/0x20 [spl]
[10824028.277054] dmu_recv_stream+0xa51/0xef0 [zfs]
[10824028.277127] zfs_ioc_recv_impl+0x306/0x1100 [zfs]
[10824028.277132] ? ttwu_do_activate+0x77/0x80
[10824028.277202] zfs_ioc_recv_new+0x33d/0x410 [zfs]
[10824028.277212] ? spl_kmem_alloc_impl+0xe5/0x1a0 [spl]
[10824028.277219] ? spl_vmem_alloc+0x19/0x20 [spl]
[10824028.277231] ? nv_alloc_sleep_spl+0x1f/0x30 [znvpair]
[10824028.277237] ? nv_mem_zalloc.isra.0+0x2e/0x40 [znvpair]
[10824028.277244] ? nvlist_xalloc.part.2+0x50/0xb0 [znvpair]
[10824028.277308] zfsdev_ioctl+0x451/0x610 [zfs]
[10824028.277319] do_vfs_ioctl+0xa8/0x630
[10824028.277324] ? __audit_syscall_entry+0xbc/0x110
[10824028.277332] ? syscall_trace_enter+0x1da/0x2d0
[10824028.277334] SyS_ioctl+0x79/0x90
[10824028.277338] do_syscall_64+0x73/0x130
[10824028.277341] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[10824028.277344] RIP: 0033:0x7fc4006905d7
[10824028.277346] RSP: 002b:00007ffe7df61dc8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[10824028.277349] RAX: ffffffffffffffda RBX: 0000000000005a46 RCX: 00007fc4006905d7
[10824028.277351] RDX: 00007ffe7df61de0 RSI: 0000000000005a46 RDI: 0000000000000006
[10824028.277352] RBP: 00007ffe7df61de0 R08: 00007fc400965ce0 R09: 0000000000000000
[10824028.277354] R10: 0000555c7adad010 R11: 0000000000000246 R12: 00007ffe7df65410
[10824028.277355] R13: 0000000000000006 R14: 0000555c7adb4ae0 R15: 000000000000000c
[10824028.277360] INFO: task receive_writer:693172 blocked for more than 120 seconds.
[10824028.277412] Tainted: P O 4.15.0-60-generic #67-Ubuntu
[10824028.277459] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[10824028.277512] receive_writer D 0 693172 2 0x80000080
[10824028.277515] Call Trace:
[10824028.277520] __schedule+0x24e/0x880
[10824028.277525] schedule+0x2c/0x80
[10824028.277535] spl_panic+0xfa/0x110 [spl]
[10824028.277585] ? dbuf_dirty+0x43d/0x850 [zfs]
[10824028.277593] ? getrawmonotonic64+0x43/0xd0
[10824028.277596] ? getrawmonotonic64+0x43/0xd0
[10824028.277652] ? dmu_zfetch+0x49a/0x500 [zfs]
[10824028.277656] ? getrawmonotonic64+0x43/0xd0
[10824028.277708] ? dmu_zfetch+0x49a/0x500 [zfs]
[10824028.277710] ? mutex_lock+0x12/0x40
[10824028.277759] ? dbuf_rele_and_unlock+0x1a8/0x4b0 [zfs]
[10824028.277824] range_tree_find_impl+0x88/0x90 [zfs]
[10824028.277832] ? spl_kmem_zalloc+0xdc/0x1a0 [spl]
[10824028.277893] range_tree_clear+0x4f/0x60 [zfs]
[10824028.277948] dnode_free_range+0x11f/0x5a0 [zfs]
[10824028.278000] dmu_object_free+0x53/0x90 [zfs]
[10824028.278050] dmu_free_long_object+0x9f/0xc0 [zfs]
[10824028.278101] receive_freeobjects.isra.12+0x7a/0x100 [zfs]
[10824028.278150] receive_writer_thread+0x6d2/0xa60 [zfs]
[10824028.278155] ? set_curr_task_fair+0x2b/0x60
[10824028.278163] ? spl_kmem_free+0x33/0x40 [spl]
[10824028.278171] ? kfree+0x165/0x180
[10824028.278217] ? receive_free.isra.13+0xc0/0xc0 [zfs]
[10824028.278225] thread_generic_wrapper+0x74/0x90 [spl]
[10824028.278230] kthread+0x121/0x140
[10824028.278236] ? __thread_exit+0x20/0x20 [spl]
[10824028.278238] ? kthread_create_worker_on_cpu+0x70/0x70
[10824028.278242] ret_from_fork+0x35/0x40

Thanks

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: linux-image-4.15.0-60-generic 4.15.0-60.67
ProcVersionSignature: Ubuntu 4.15.0-60.67-generic 4.15.18
Uname: Linux 4.15.0-60-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Jan 9 16:32 seq
 crw-rw---- 1 root audio 116, 33 Jan 9 16:32 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.9-0ubuntu7.9
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
Date: Tue Jan 28 13:13:09 2020
HibernationDevice: RESUME=UUID=d34df57d-ae32-4002-be2c-a25efa8678e4
InstallationDate: Installed on 2016-04-04 (1394 days ago)
InstallationMedia: Ubuntu-Server 16.04 LTS "Xenial Xerus" - Beta amd64 (20160325)
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Supermicro SSG-6038R-E1CR16L
PciMultimedia:

ProcEnviron:
 TERM=rxvt-unicode-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-60-generic root=UUID=d1ab8bd8-41d5-4142-92d6-d0926539807b ro
RelatedPackageVersions:
 linux-restricted-modules-4.15.0-60-generic N/A
 linux-backports-modules-4.15.0-60-generic N/A
 linux-firmware 1.173.14
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: Upgraded to bionic on 2018-08-16 (530 days ago)
dmi.bios.date: 12/17/2015
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 2.0
dmi.board.asset.tag: Default string
dmi.board.name: X10DRH-iT
dmi.board.vendor: Supermicro
dmi.board.version: 1.01
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 1
dmi.chassis.vendor: Supermicro
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.0:bd12/17/2015:svnSupermicro:pnSSG-6038R-E1CR16L:pvr123456789:rvnSupermicro:rnX10DRH-iT:rvr1.01:cvnSupermicro:ct1:cvrDefaultstring:
dmi.product.family: SMC X10
dmi.product.name: SSG-6038R-E1CR16L
dmi.product.version: 123456789
dmi.sys.vendor: Supermicro

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.