zpool import PANIC at zio.c:266

[Andreas Mattisson]

1)
#lsb_release -rd
Description: Ubuntu 18.04.4 LTS
Release: 18.04

2)
#apt-cache policy zfsutils-linux
zfsutils-linux:
  Installerad: 0.7.5-1ubuntu16.9
  Kandidat: 0.7.5-1ubuntu16.9

# dpkg -l | grep -i "zfs"
rc libzfs1 0.6.2-1~precise amd64 Native ZFS filesystem library for Linux
rc libzfs2 0.6.5.4-1~precise amd64 Native OpenZFS filesystem library for Linux
ii libzfs2linux 0.7.5-1ubuntu16.9 amd64 OpenZFS filesystem library for Linux
rc libzpool1 0.6.2-1~precise amd64 Native ZFS pool library for Linux
rc libzpool2 0.6.5.4-1~precise amd64 Native OpenZFS pool library for Linux
ii libzpool2linux 0.7.5-1ubuntu16.9 amd64 OpenZFS pool library for Linux
rc ubuntu-zfs 8~precise amd64 Native ZFS filesystem metapackage for Ubuntu.
rc zfs-dkms 0.7.5-1ubuntu16.9 all OpenZFS filesystem kernel modules for Linux
ii zfs-doc 0.7.5-1ubuntu16.9 all Native OpenZFS filesystem documentation and examples.
ii zfs-initramfs 0.7.5-1ubuntu16.9 all OpenZFS root filesystem capabilities for Linux - initramfs
ii zfs-zed 0.7.5-1ubuntu16.9 amd64 OpenZFS Event Daemon
rc zfsutils 0.6.5.4-1~precise amd64 Native OpenZFS management utilities for Linux
ii zfsutils-linux 0.7.5-1ubuntu16.9 amd64 command-line tools to manage OpenZFS filesystems

#zpool status
no pools available

# zpool import
   pool: datastore
     id: 3190464655986727485
  state: ONLINE
 status: Some supported features are not enabled on the pool.
 action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
 config:

        datastore ONLINE
          raidz2-0 ONLINE
            sdb ONLINE
            sdc ONLINE
            sdd ONLINE
            sde ONLINE
            sdf ONLINE
            sdg ONLINE

# zpool import -o readonly=on datastore
# zpool status
  pool: datastore
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub in progress since Sun May 10 00:24:01 2020
        15,1T scanned out of 15,1T at 1B/s, (scan is slow, no estimated time)
        1,44M repaired, 100,01% done
config:

        NAME STATE READ WRITE CKSUM
        datastore ONLINE 0 0 0
          raidz2-0 ONLINE 0 0 0
            sdb ONLINE 0 0 0
            sdc ONLINE 0 0 0
            sdd ONLINE 0 0 0
            sde ONLINE 0 0 0
            sdf ONLINE 0 0 0
            sdg ONLINE 0 0 0

errors: No known data errors

If I now tries to import the datastore in normal way, it Panic and hangs.

# zpool export datastore
# zpool import -F datastore

VERIFY3(c < (1ULL << 24) >> 9) failed (36028797018963967 < 32768)
PANIC at zio.c:266:zio_buf_alloc()

Some output from the logs:

May 22 09:22:14 Crichton kernel: [ 44.582834] VERIFY3(c < (1ULL << 24) >> 9) failed (36028797018963967 < 32768)
May 22 09:22:14 Crichton kernel: [ 44.582870] PANIC at zio.c:266:zio_buf_alloc()
May 22 09:22:14 Crichton kernel: [ 44.582890] Showing stack for process 2180
May 22 09:22:14 Crichton kernel: [ 44.582893] CPU: 4 PID: 2180 Comm: txg_sync Tainted: P IO 4.15.0-101-generic #102-Ubuntu
May 22 09:22:14 Crichton kernel: [ 44.582894] Hardware name: Gigabyte Technology Co., Ltd. Z87X-UD4H/Z87X-UD4H-CF, BIOS F7 08/03/2013
May 22 09:22:14 Crichton kernel: [ 44.582895] Call Trace:
May 22 09:22:14 Crichton kernel: [ 44.582902] dump_stack+0x6d/0x8e
May 22 09:22:14 Crichton kernel: [ 44.582911] spl_dumpstack+0x42/0x50 [spl]
May 22 09:22:14 Crichton kernel: [ 44.582915] spl_panic+0xc8/0x110 [spl]
May 22 09:22:14 Crichton kernel: [ 44.582919] ? __switch_to_asm+0x41/0x70
May 22 09:22:14 Crichton kernel: [ 44.582921] ? _cond_resched+0x19/0x40
May 22 09:22:14 Crichton kernel: [ 44.582924] ? kmem_cache_alloc+0xab/0x1c0
May 22 09:22:14 Crichton kernel: [ 44.582927] ? spl_kmem_cache_alloc+0x73/0x7a0 [spl]
May 22 09:22:14 Crichton kernel: [ 44.582931] ? spl_kmem_cache_alloc+0x73/0x7a0 [spl]
May 22 09:22:14 Crichton kernel: [ 44.582934] ? spl_kmem_cache_alloc+0x73/0x7a0 [spl]
May 22 09:22:14 Crichton kernel: [ 44.582978] zio_buf_alloc+0x43/0x60 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583008] abd_alloc_linear+0x67/0xe0 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583032] abd_alloc+0x391/0x470 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583036] ? spl_kmem_cache_alloc+0x113/0x7a0 [spl]
May 22 09:22:14 Crichton kernel: [ 44.583059] arc_hdr_alloc_pabd+0xe7/0xf0 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583083] arc_hdr_alloc+0xb5/0x130 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583106] arc_read+0x219/0xa80 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583129] ? arc_buf_destroy+0x140/0x140 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583158] dsl_scan_prefetch.isra.8+0xb7/0xd0 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583184] dsl_scan_visitbp+0x3c6/0xd60 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583210] ? dsl_scan_prefetch.isra.8+0xb7/0xd0 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583236] dsl_scan_visitbp+0x7c5/0xd60 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583260] dsl_scan_visitbp+0x487/0xd60 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583285] dsl_scan_visitbp+0x487/0xd60 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583309] dsl_scan_visitbp+0x487/0xd60 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583333] dsl_scan_visitbp+0x487/0xd60 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583357] dsl_scan_visitbp+0x487/0xd60 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583381] dsl_scan_visitbp+0x487/0xd60 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583404] dsl_scan_visitbp+0x97b/0xd60 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583427] dsl_scan_visitds+0x108/0x540 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583451] dsl_scan_sync+0x2ef/0xc30 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583481] ? zio_wait+0x141/0x1b0 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583509] spa_sync+0x49e/0xd80 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583540] txg_sync_thread+0x2cd/0x4a0 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583542] ? __switch_to_asm+0x35/0x70
May 22 09:22:14 Crichton kernel: [ 44.583570] ? txg_quiesce_thread+0x3d0/0x3d0 [zfs]
May 22 09:22:14 Crichton kernel: [ 44.583574] thread_generic_wrapper+0x74/0x90 [spl]
May 22 09:22:14 Crichton kernel: [ 44.583577] kthread+0x121/0x140
May 22 09:22:14 Crichton kernel: [ 44.583580] ? __thread_exit+0x20/0x20 [spl]
May 22 09:22:14 Crichton kernel: [ 44.583581] ? kthread_create_worker_on_cpu+0x70/0x70
May 22 09:22:14 Crichton kernel: [ 44.583583] ret_from_fork+0x35/0x40

---

I also tried to use zdb to see if it could find any errors. This gets interrupted after a while.

# set zfs:zfs_recover=1
# set aok=1
# zdb -e -bcsvL datastore

Traversing all blocks to verify checksums ...

2.29T completed ( 144MB/s) estimated time remaining: 25hr 46min 43sec c < (1ULL << 24) >> 9 (0x7fffffffffffff < 0x8000)
ASSERT at ../../module/zfs/zio.c:266:zio_buf_alloc()Avbruten (SIGABRT)

3)
After a reboot, ZFS pool never stopped trying to import the pool. It was blocking the boot-sequence and I had to rename the /etc/zfs/zpool.cache to another name, in order to boot the system.
There was an upgrade made before reboot, but that was not anything with zfs.

The problem is that zpool datastore is not possible to import in normal way. Only in readonly=on is I able to import it.
Also the Scub seems to be active on the readonly, but it will never stops since it not possible to write to the pool.
I cannot stop scrub either since I can only import it at readonly.

If I import it at readonly it works and I can access the files. But if import normally a panic occur, and the process hangs.

4)
Expected to be able to import the zpool as normal.

[Colin Ian King (colin-king)]

Can you try to temporarily disable the spa verification to see if this allows the import to at least complete to see if this helps.

echo 0 | sudo tee /sys/module/zfs/parameters/spa_load_verify_metadata
echo 0 | sudo tee /sys/module/zfs/parameters/spa_load_verify_data

do the import, and then re-enable:

echo 1 | sudo tee /sys/module/zfs/parameters/spa_load_verify_metadata
echo 1 | sudo tee /sys/module/zfs/parameters/spa_load_verify_data

[Andreas Mattisson (andreasmatt)]

# cat /sys/module/zfs/parameters/spa_load_verify_metadata
1
# echo 0 | sudo tee /sys/module/zfs/parameters/spa_load_verify_metadata
0
# echo 0 | sudo tee /sys/module/zfs/parameters/spa_load_verify_data
0
# zpool import
   pool: datastore
     id: 3190464655986727485
  state: ONLINE
 status: Some supported features are not enabled on the pool.
 action: The pool can be imported using its name or numeric identifier, though
        some features will not be available without an explicit 'zpool upgrade'.
 config:

        datastore ONLINE
          raidz2-0 ONLINE
            sdb ONLINE
            sdc ONLINE
            sdd ONLINE
            sde ONLINE
            sdf ONLINE
            sdg ONLINE
# zpool import datastore
or
# zpool import -f datastore

Cause this output:
VERIFY3(c < SPA_MAXBLOCKSIZE >> SPA_MINBLOCKSHIFT) failed (...)
PANIC at zio.c:293:zio_buf_alloc()

[Andreas Mattisson (andreasmatt)]

Thanks for you fast reply, forgot to add a description.
I tried you suggested solution. I first had to remove /etc/zfs/zpool.cache in order to be able to do this.

Then I tried your solution 2 times, once with no flags, and then with -f flag.
The panic still occurs when doing this.

Should I try zdb again?

Any other suggestions ?

[Colin Ian King (colin-king)]

Apologies for the delay in responding.

It may be worth trying the -A, -AA, -AAA options in zdb

[Colin Ian King (colin-king)]

The comment #4 has not been responded to for 6 weeks, so this bug will be closed as Won't Fix as it inactive. If this is still and issue, please re-open this bug and we can continue to try to work on this issue.