ZFS create ignores ashift

[Thekigen]

Hi,

I've ran into an issue trying to create a ZFS pool with Ubuntu 22.04. In my case it seems that the ZFS command ignores the ashift and because the drives are reporting an insane physical sector size of 256KB, and ZFS goes for the maximum ashift of 64KB sector size despite me specifying the 4KB sector size.

It appears the upstream OpenZFS project has already fixed these issues. They just need to be backported to Ubuntu.
https://github.com/openzfs/zfs/commit/37f6845c6f86b1d04593e55d94318326006f4b5d

My command:
zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -o autoexpand=on \
    -o autoreplace=on \
    -O acltype=posixacl -O xattr=sa -O dnodesize=auto \
    -O compression=zstd-3 \
    -O normalization=formD \
    -O recordsize=32k \
    -O relatime=on \
    pg \
    mirror nvme0n1 nvme1n1 mirror nvme2n1 nvme3n1 mirror nvme4n1 nvme5n1 mirror nvme6n1 nvme7n1 mirror nvme8n1 nvme9n1 \
    mirror nvme10n1 nvme11n1 mirror nvme12n1 nvme13n1 mirror nvme14n1 nvme15n1 mirror nvme16n1 nvme17n1 mirror nvme18n1 nvme19n1 \
    mirror nvme20n1 nvme21n1 \
    spare nvme22n1 nvme23n1

The resulting zpool status output:
zpool status
  pool: pg
 state: ONLINE
status: One or more devices are configured to use a non-native block size.
        Expect reduced performance.
action: Replace affected devices with devices that support the
        configured block size, or migrate data to a properly configured
        pool.
config:

        NAME STATE READ WRITE CKSUM
        pg ONLINE 0 0 0
          mirror-0 ONLINE 0 0 0
            nvme0n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme1n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
          mirror-1 ONLINE 0 0 0
            nvme2n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme3n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
          mirror-2 ONLINE 0 0 0
            nvme4n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme5n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
          mirror-3 ONLINE 0 0 0
            nvme6n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme7n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
          mirror-4 ONLINE 0 0 0
            nvme8n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme9n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
          mirror-5 ONLINE 0 0 0
            nvme10n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme11n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
          mirror-6 ONLINE 0 0 0
            nvme12n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme13n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
          mirror-7 ONLINE 0 0 0
            nvme14n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme15n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
          mirror-8 ONLINE 0 0 0
            nvme16n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme17n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
          mirror-9 ONLINE 0 0 0
            nvme18n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme19n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
          mirror-10 ONLINE 0 0 0
            nvme20n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
            nvme21n1 ONLINE 0 0 0 block size: 65536B configured, 262144B native
        spares
          nvme22n1 AVAIL
          nvme23n1 AVAIL

errors: No known data errors

lsb_release -rd
Description: Ubuntu 22.04.2 LTS
Release: 22.04

apt-cache policy zfsutils-linux
zfsutils-linux:
  Installed: 2.1.5-1ubuntu6~22.04.1
  Candidate: 2.1.5-1ubuntu6~22.04.1
  Version table:
 *** 2.1.5-1ubuntu6~22.04.1 500
        500 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.1.4-0ubuntu0.1 500
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     2.1.2-1ubuntu3 500
        500 http://us.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: zfsutils-linux 2.1.5-1ubuntu6~22.04.1
ProcVersionSignature: Ubuntu 5.15.0-67.74-generic 5.15.85
Uname: Linux 5.15.0-67-generic x86_64
NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
ApportVersion: 2.20.11-0ubuntu82.3
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Sat Mar 4 12:10:21 2023
ProcEnviron:
 SHELL=/bin/bash
 LANG=en_US.UTF-8
 TERM=xterm
 XDG_RUNTIME_DIR=<set>
 PATH=(custom, no user)
SourcePackage: zfs-linux
UpgradeStatus: Upgraded to jammy on 2023-03-02 (1 days ago)

[Thekigen (thekigen)]

Also, to show that ashift was set to 16 instead of 12.
zdb | grep ashift
            ashift: 16
            ashift: 16
            ashift: 16
            ashift: 16
            ashift: 16
            ashift: 16
            ashift: 16
            ashift: 16
            ashift: 16
            ashift: 16
            ashift: 16

[Thekigen (thekigen)]

Related reports in OpenZFS:
https://github.com/openzfs/zfs/issues/14409

Zpool reports ashift 12, but zdb reports they're all set to 16.
zpool get ashift
NAME PROPERTY VALUE SOURCE
pg ashift 12 local

cat /sys/class/block/nvme0n1/queue/logical_block_size
4096

cat /sys/class/block/nvme0n1/queue/physical_block_size
262144

[Thekigen (thekigen)]

Found a work around.

echo 12 >> /sys/module/zfs/parameters/zfs_vdev_max_auto_ashift