PANIC at zfs_vfsops.c:585:zfs_space_delta_cb / VERIFY3(sa.sa_magic == SA_MAGIC) failed
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Native ZFS for Linux |
New
|
Unknown
|
|||
zfs-linux (Ubuntu) |
New
|
Undecided
|
Unassigned |
Bug Description
When running rsync or tree (haven't tried much else) on much of the filesystem it will pretty routinely throw a panic report in syslog/kern.log, and the rsync/tree operation will later hang solidly such that even ^C doesn't break back to the shell, and any further file operations will likely hang and lock up their shells. Often requires power cycle rather than just shutdown -r to reboot.
syslog/kern.log:
Feb 2 16:52:43 donny kernel: [ 517.355790] VERIFY3(sa.sa_magic == SA_MAGIC) failed (*value* == 3100762) <- *value* varies e.g. 1612270437, 1612270437, 1612283815
Feb 2 16:52:43 donny kernel: [ 517.362904] PANIC at zfs_vfsops.
Feb 2 16:52:43 donny kernel: [ 517.368512] Showing stack for process *pid* <- *pid* varies
Feb 2 16:52:43 donny kernel: [ 517.368522] CPU: 1 PID: *pid* Comm: dp_sync_taskq Tainted: P C OE 5.4.0-1028-raspi #31-Ubuntu <- *pid* as above
Feb 2 16:52:43 donny kernel: [ 517.368525] Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Feb 2 16:52:43 donny kernel: [ 517.368527] Call trace:
Feb 2 16:52:43 donny kernel: [ 517.368540] dump_backtrace+
Feb 2 16:52:43 donny kernel: [ 517.368544] show_stack+
Feb 2 16:52:43 donny kernel: [ 517.368549] dump_stack+
Feb 2 16:52:43 donny kernel: [ 517.368583] spl_dumpstack+
Feb 2 16:52:43 donny kernel: [ 517.368606] spl_panic+0xe0/0xf8 [spl]
Feb 2 16:52:43 donny kernel: [ 517.368837] zfs_space_
Feb 2 16:52:43 donny kernel: [ 517.368968] dmu_objset_
Feb 2 16:52:43 donny kernel: [ 517.369094] dnode_sync+
Feb 2 16:52:43 donny kernel: [ 517.369216] dmu_objset_
Feb 2 16:52:43 donny kernel: [ 517.369333] sync_dnodes_
Feb 2 16:52:43 donny kernel: [ 517.369353] taskq_thread+
Feb 2 16:52:43 donny kernel: [ 517.369363] kthread+0x150/0x170
Feb 2 16:52:43 donny kernel: [ 517.369368] ret_from_
rsync/tree command carries on running (for a while), then later log file shows:
Feb 2 16:56:11 donny kernel: [ 725.993423] INFO: task dp_sync_taskq:*pid* blocked for more than 120 seconds.
Feb 2 16:56:11 donny kernel: [ 726.000633] Tainted: P C OE 5.4.0-1028-raspi #31-Ubuntu
Feb 2 16:56:11 donny kernel: [ 726.007610] "echo 0 > /proc/sys/
Feb 2 16:56:11 donny kernel: [ 726.015586] dp_sync_taskq D 0 *pid* 2 0x00000028
Feb 2 16:56:11 donny kernel: [ 726.015595] Call trace:
Feb 2 16:56:11 donny kernel: [ 726.015608] __switch_
Feb 2 16:56:11 donny kernel: [ 726.015615] __schedule+
Feb 2 16:56:11 donny kernel: [ 726.015621] schedule+0x4c/0xe8
Feb 2 16:56:11 donny kernel: [ 726.015650] spl_panic+0xf4/0xf8 [spl]
Feb 2 16:56:11 donny kernel: [ 726.015829] zfs_space_
Feb 2 16:56:11 donny kernel: [ 726.015939] dmu_objset_
Feb 2 16:56:11 donny kernel: [ 726.016045] dnode_sync+
Feb 2 16:56:11 donny kernel: [ 726.016151] dmu_objset_
Feb 2 16:56:11 donny kernel: [ 726.016256] sync_dnodes_
Feb 2 16:56:11 donny kernel: [ 726.016276] taskq_thread+
Feb 2 16:56:11 donny kernel: [ 726.016283] kthread+0x150/0x170
Feb 2 16:56:11 donny kernel: [ 726.016288] ret_from_
Feb 2 16:56:11 donny kernel: [ 726.016298] INFO: task txg_sync:*num* blocked for more than 120 seconds. <- *num* varies
Feb 2 16:56:11 donny kernel: [ 726.023051] Tainted: P C OE 5.4.0-1028-raspi #31-Ubuntu
Feb 2 16:56:11 donny kernel: [ 726.029957] "echo 0 > /proc/sys/
Feb 2 16:56:11 donny kernel: [ 726.037919] txg_sync D 0 *num* 2 0x00000028
Feb 2 16:56:11 donny kernel: [ 726.037929] Call trace:
Feb 2 16:56:11 donny kernel: [ 726.037939] __switch_
Feb 2 16:56:11 donny kernel: [ 726.037945] __schedule+
Feb 2 16:56:11 donny kernel: [ 726.037951] schedule+0x4c/0xe8
Feb 2 16:56:11 donny kernel: [ 726.037973] taskq_wait+
Feb 2 16:56:11 donny kernel: [ 726.038142] dmu_objset_
Feb 2 16:56:11 donny kernel: [ 726.038263] dsl_dataset_
Feb 2 16:56:11 donny kernel: [ 726.038391] dsl_pool_
Feb 2 16:56:11 donny kernel: [ 726.038509] spa_sync_
Feb 2 16:56:11 donny kernel: [ 726.038617] spa_sync+
Feb 2 16:56:11 donny kernel: [ 726.038736] txg_sync_
Feb 2 16:56:11 donny kernel: [ 726.038756] thread_
Feb 2 16:56:11 donny kernel: [ 726.038768] kthread+0x150/0x170
Feb 2 16:56:11 donny kernel: [ 726.038774] ret_from_
I don't know if this is related to bug #1906476 https:/
System Information:
ubuntu 20.04.2 (5.4.0-1028-raspi) server on RPi4/8G with 8TB external Seagate formatted zfs
# lsb_release -rd
Description: Ubuntu 20.04.2 LTS
Release: 20.04
# apt-cache policy zfsutils-linux
zfsutils-linux:
Installed: 0.8.3-1ubuntu12.6
Candidate: 0.8.3-1ubuntu12.6
Version table:
*** 0.8.3-1ubuntu12.6 500
500 http://
100 /var/lib/
0.
500 http://
# zpool version
zfs-0.8.
zfs-kmod-
Changed in zfs: | |
status: | Unknown → New |
Going back through logfiles the first occurrence is c:585:zfs_ space_delta_ cb()
Jan 29 12:27:52 donny kernel: [72122.085286] PANIC at zfs_vfsops.
So it seems to have started after I upgraded from Linux version 5.4.0-1026-raspi to 1028
Jan 27 04:27:18 donny kernel: [ 0.000000] Linux version 5.4.0-1026-raspi (buildd@ bos02-arm64- 062) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1 ~20.04) ) #29-Ubuntu SMP PREEMPT Mon Dec 14 17:01:16 UTC 2020 (Ubuntu 5.4.0-1026.29-raspi 5.4.78)
Jan 28 16:25:31 donny kernel: [ 0.000000] Linux version 5.4.0-1028-raspi (buildd@ bos02-arm64- 034) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1 ~20.04) ) #31-Ubuntu SMP PREEMPT Wed Jan 20 11:30:45 UTC 2021 (Ubuntu 5.4.0-1028.31-raspi 5.4.78)