zpool import PANIC at zio.c:266
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
zfs-linux (Ubuntu) |
Won't Fix
|
High
|
Colin Ian King |
Bug Description
1)
#lsb_release -rd
Description: Ubuntu 18.04.4 LTS
Release: 18.04
2)
#apt-cache policy zfsutils-linux
zfsutils-linux:
Installerad: 0.7.5-1ubuntu16.9
Kandidat: 0.7.5-1ubuntu16.9
# dpkg -l | grep -i "zfs"
rc libzfs1 0.6.2-1~precise amd64 Native ZFS filesystem library for Linux
rc libzfs2 0.6.5.4-1~precise amd64 Native OpenZFS filesystem library for Linux
ii libzfs2linux 0.7.5-1ubuntu16.9 amd64 OpenZFS filesystem library for Linux
rc libzpool1 0.6.2-1~precise amd64 Native ZFS pool library for Linux
rc libzpool2 0.6.5.4-1~precise amd64 Native OpenZFS pool library for Linux
ii libzpool2linux 0.7.5-1ubuntu16.9 amd64 OpenZFS pool library for Linux
rc ubuntu-zfs 8~precise amd64 Native ZFS filesystem metapackage for Ubuntu.
rc zfs-dkms 0.7.5-1ubuntu16.9 all OpenZFS filesystem kernel modules for Linux
ii zfs-doc 0.7.5-1ubuntu16.9 all Native OpenZFS filesystem documentation and examples.
ii zfs-initramfs 0.7.5-1ubuntu16.9 all OpenZFS root filesystem capabilities for Linux - initramfs
ii zfs-zed 0.7.5-1ubuntu16.9 amd64 OpenZFS Event Daemon
rc zfsutils 0.6.5.4-1~precise amd64 Native OpenZFS management utilities for Linux
ii zfsutils-linux 0.7.5-1ubuntu16.9 amd64 command-line tools to manage OpenZFS filesystems
#zpool status
no pools available
# zpool import
pool: datastore
id: 3190464655986727485
state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
some features will not be available without an explicit 'zpool upgrade'.
config:
datastore ONLINE
raidz2-0 ONLINE
sdb ONLINE
sdc ONLINE
sdd ONLINE
sde ONLINE
sdf ONLINE
sdg ONLINE
# zpool import -o readonly=on datastore
# zpool status
pool: datastore
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub in progress since Sun May 10 00:24:01 2020
15,1T scanned out of 15,1T at 1B/s, (scan is slow, no estimated time)
1,44M repaired, 100,01% done
config:
NAME STATE READ WRITE CKSUM
datastore ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
sdg ONLINE 0 0 0
errors: No known data errors
If I now tries to import the datastore in normal way, it Panic and hangs.
# zpool export datastore
# zpool import -F datastore
VERIFY3(c < (1ULL << 24) >> 9) failed (36028797018963967 < 32768)
PANIC at zio.c:266:
Some output from the logs:
May 22 09:22:14 Crichton kernel: [ 44.582834] VERIFY3(c < (1ULL << 24) >> 9) failed (36028797018963967 < 32768)
May 22 09:22:14 Crichton kernel: [ 44.582870] PANIC at zio.c:266:
May 22 09:22:14 Crichton kernel: [ 44.582890] Showing stack for process 2180
May 22 09:22:14 Crichton kernel: [ 44.582893] CPU: 4 PID: 2180 Comm: txg_sync Tainted: P IO 4.15.0-101-generic #102-Ubuntu
May 22 09:22:14 Crichton kernel: [ 44.582894] Hardware name: Gigabyte Technology Co., Ltd. Z87X-UD4H/
May 22 09:22:14 Crichton kernel: [ 44.582895] Call Trace:
May 22 09:22:14 Crichton kernel: [ 44.582902] dump_stack+
May 22 09:22:14 Crichton kernel: [ 44.582911] spl_dumpstack+
May 22 09:22:14 Crichton kernel: [ 44.582915] spl_panic+
May 22 09:22:14 Crichton kernel: [ 44.582919] ? __switch_
May 22 09:22:14 Crichton kernel: [ 44.582921] ? _cond_resched+
May 22 09:22:14 Crichton kernel: [ 44.582924] ? kmem_cache_
May 22 09:22:14 Crichton kernel: [ 44.582927] ? spl_kmem_
May 22 09:22:14 Crichton kernel: [ 44.582931] ? spl_kmem_
May 22 09:22:14 Crichton kernel: [ 44.582934] ? spl_kmem_
May 22 09:22:14 Crichton kernel: [ 44.582978] zio_buf_
May 22 09:22:14 Crichton kernel: [ 44.583008] abd_alloc_
May 22 09:22:14 Crichton kernel: [ 44.583032] abd_alloc+
May 22 09:22:14 Crichton kernel: [ 44.583036] ? spl_kmem_
May 22 09:22:14 Crichton kernel: [ 44.583059] arc_hdr_
May 22 09:22:14 Crichton kernel: [ 44.583083] arc_hdr_
May 22 09:22:14 Crichton kernel: [ 44.583106] arc_read+
May 22 09:22:14 Crichton kernel: [ 44.583129] ? arc_buf_
May 22 09:22:14 Crichton kernel: [ 44.583158] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583184] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583210] ? dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583236] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583260] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583285] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583309] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583333] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583357] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583381] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583404] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583427] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583451] dsl_scan_
May 22 09:22:14 Crichton kernel: [ 44.583481] ? zio_wait+
May 22 09:22:14 Crichton kernel: [ 44.583509] spa_sync+
May 22 09:22:14 Crichton kernel: [ 44.583540] txg_sync_
May 22 09:22:14 Crichton kernel: [ 44.583542] ? __switch_
May 22 09:22:14 Crichton kernel: [ 44.583570] ? txg_quiesce_
May 22 09:22:14 Crichton kernel: [ 44.583574] thread_
May 22 09:22:14 Crichton kernel: [ 44.583577] kthread+0x121/0x140
May 22 09:22:14 Crichton kernel: [ 44.583580] ? __thread_
May 22 09:22:14 Crichton kernel: [ 44.583581] ? kthread_
May 22 09:22:14 Crichton kernel: [ 44.583583] ret_from_
---
I also tried to use zdb to see if it could find any errors. This gets interrupted after a while.
# set zfs:zfs_recover=1
# set aok=1
# zdb -e -bcsvL datastore
Traversing all blocks to verify checksums ...
2.29T completed ( 144MB/s) estimated time remaining: 25hr 46min 43sec c < (1ULL << 24) >> 9 (0x7fffffffffffff < 0x8000)
ASSERT at ../../module/
3)
After a reboot, ZFS pool never stopped trying to import the pool. It was blocking the boot-sequence and I had to rename the /etc/zfs/
There was an upgrade made before reboot, but that was not anything with zfs.
The problem is that zpool datastore is not possible to import in normal way. Only in readonly=on is I able to import it.
Also the Scub seems to be active on the readonly, but it will never stops since it not possible to write to the pool.
I cannot stop scrub either since I can only import it at readonly.
If I import it at readonly it works and I can access the files. But if import normally a panic occur, and the process hangs.
4)
Expected to be able to import the zpool as normal.
description: | updated |
Can you try to temporarily disable the spa verification to see if this allows the import to at least complete to see if this helps.
echo 0 | sudo tee /sys/module/ zfs/parameters/ spa_load_ verify_ metadata zfs/parameters/ spa_load_ verify_ data
echo 0 | sudo tee /sys/module/
do the import, and then re-enable:
echo 1 | sudo tee /sys/module/ zfs/parameters/ spa_load_ verify_ metadata zfs/parameters/ spa_load_ verify_ data
echo 1 | sudo tee /sys/module/