blk_update_request: operation not supported error (WRITE_ZEROES) on Sandisk Skyhawk nvme SSD

[Xing Li]

Similar to https://bugs.launchpad.net/oem-priority/+bug/1872383

OS: Ubuntu 20.04
Kernel Affected: 5.4.0-48-generic and mainline/5.8.12-050812-generic
Device: Sandisk Skyhawk NVME SSD
FileSystem: Ext4

Eerror (Caused by newly formatted and mounted Ext4 partition. Can confirm the errors are thrown by the ext4lazyInit kernel thread)
-----
[ 6305.633884] print_req_error: 20 callbacks suppressed
[ 6305.633887] blk_update_request: operation not supported error, dev nvme0n1, sector 340812032 op 0x9:(WRITE_ZEROES) flags 0x0 phys_seg 0 prio class 0
[ 6305.639529] blk_update_request: operation not supported error, dev nvme0n1, sector 340812289 op 0x9:(WRITE_ZEROES) flags 0x0 phys_seg 0 prio class 0

Sandisk Skyhawk nvme firmware version/info

Node SN Model Namespace Usage Format FW Rev
----
/dev/nvme0n1 A04506C0 SDLC2CLR-016T-3NA1 1 1.60 TB / 1.60 TB 512 B + 0 B N00A
/dev/nvme1n1 A04507FB SDLC2CLR-016T-3NA1 1 1.60 TB / 1.60 TB 512 B + 0 B N00A

Both identical drives have same issue so not related to a single defective drive.

lspci output
------------

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Skyhawk Series NVME SSD [15b7:2001] (rev 01)
03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d0] (rev 01)
03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
20:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
20:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
20:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
20:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
20:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
21:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04)
22:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41)
23:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
24:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
25:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
2b:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Skyhawk Series NVME SSD [15b7:2001] (rev 01)
2c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
2d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
2d:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
2d:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
2d:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
30:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
31:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)

nvme id-ctrl -H /dev/nvme0
-------
NVME Identify Controller:
vid : 0x15b7
ssvid : 0x11ab
sn : A04506C0
mn : SDLC2CLR-016T-3NA1
fr : N00A
rab : 16
ieee : 731100
cmic : 0
  [3:3] : 0 ANA not supported
  [2:2] : 0 PCI
  [1:1] : 0 Single Controller
  [0:0] : 0 Single Port

mdts : 5
cntlid : 0
ver : 0x10200
rtd3r : 0
rtd3e : 0
oaes : 0
[14:14] : 0 Endurance Group Event Aggregate Log Page Change Notice Not Supported
[13:13] : 0 LBA Status Information Notices Not Supported
[12:12] : 0 Predictable Latency Event Aggregate Log Change Notices Not Supported
[11:11] : 0 Asymmetric Namespace Access Change Notices Not Supported
  [9:9] : 0 Firmware Activation Notices Not Supported
  [8:8] : 0 Namespace Attribute Changed Event Not Supported

ctratt : 0
  [9:9] : 0 UUID List Not Supported
  [7:7] : 0 Namespace Granularity Not Supported
  [5:5] : 0 Predictable Latency Mode Not Supported
  [4:4] : 0 Endurance Groups Not Supported
  [3:3] : 0 Read Recovery Levels Not Supported
  [2:2] : 0 NVM Sets Not Supported
  [1:1] : 0 Non-Operational Power State Permissive Not Supported
  [0:0] : 0 128-bit Host Identifier Not Supported

rrls : 0
crdt1 : 0
crdt2 : 0
crdt3 : 0
oacs : 0x6
  [9:9] : 0 Get LBA Status Capability Not Supported
  [8:8] : 0 Doorbell Buffer Config Not Supported
  [7:7] : 0 Virtualization Management Not Supported
  [6:6] : 0 NVMe-MI Send and Receive Not Supported
  [5:5] : 0 Directives Not Supported
  [4:4] : 0 Device Self-test Not Supported
  [3:3] : 0 NS Management and Attachment Not Supported
  [2:2] : 0x1 FW Commit and Download Supported
  [1:1] : 0x1 Format NVM Supported
  [0:0] : 0 Security Send and Receive Not Supported

acl : 3
aerl : 3
frmw : 0x14
  [4:4] : 0x1 Firmware Activate Without Reset Supported
  [3:1] : 0x2 Number of Firmware Slots
  [0:0] : 0 Firmware Slot 1 Read/Write

lpa : 0
  [3:3] : 0 Telemetry host/controller initiated log page Not Supported
  [2:2] : 0 Extended data for Get Log Page Not Supported
  [1:1] : 0 Command Effects Log Page Not Supported
  [0:0] : 0 SMART/Health Log Page per NS Not Supported

elpe : 63
npss : 2
avscc : 0x1
  [0:0] : 0x1 Admin Vendor Specific Commands uses NVMe Format

apsta : 0
  [0:0] : 0 Autonomous Power State Transitions Not Supported

wctemp : 343
cctemp : 348
mtfa : 0
hmpre : 0
hmmin : 0
tnvmcap : 0
unvmcap : 0
rpmbs : 0
 [31:24]: 0 Access Size
 [23:16]: 0 Total Size
  [5:3] : 0 Authentication Method
  [2:0] : 0 Number of RPMB Units

edstt : 0
dsto : 0
fwug : 0
kas : 0
hctma : 0
  [0:0] : 0 Host Controlled Thermal Management Not Supported

mntmt : 0
mxtmt : 0
sanicap : 0
  [31:30] : 0 Additional media modification after sanitize operation completes successfully is not defined
  [29:29] : 0 No-Deallocate After Sanitize bit in Sanitize command Supported
    [2:2] : 0 Overwrite Sanitize Operation Not Supported
    [1:1] : 0 Block Erase Sanitize Operation Not Supported
    [0:0] : 0 Crypto Erase Sanitize Operation Not Supported

hmminds : 0
hmmaxd : 0
nsetidmax : 0
anatt : 0
anacap : 0
  [7:7] : 0 Non-zero group ID Not Supported
  [6:6] : 0 Group ID does not change
  [4:4] : 0 ANA Change state Not Supported
  [3:3] : 0 ANA Persistent Loss state Not Supported
  [2:2] : 0 ANA Inaccessible state Not Supported
  [1:1] : 0 ANA Non-optimized state Not Supported
  [0:0] : 0 ANA Optimized state Not Supported

anagrpmax : 0
nanagrpid : 0
sqes : 0x66
  [7:4] : 0x6 Max SQ Entry Size (64)
  [3:0] : 0x6 Min SQ Entry Size (64)

cqes : 0x44
  [7:4] : 0x4 Max CQ Entry Size (16)
  [3:0] : 0x4 Min CQ Entry Size (16)

maxcmd : 0
nn : 1
oncs : 0x1f
  [7:7] : 0 Verify Not Supported
  [6:6] : 0 Timestamp Not Supported
  [5:5] : 0 Reservations Not Supported
  [4:4] : 0x1 Save and Select Supported
  [3:3] : 0x1 Write Zeroes Supported
  [2:2] : 0x1 Data Set Management Supported
  [1:1] : 0x1 Write Uncorrectable Supported
  [0:0] : 0x1 Compare Supported

fuses : 0
  [0:0] : 0 Fused Compare and Write Not Supported

fna : 0x5
  [2:2] : 0x1 Crypto Erase Supported as part of Secure Erase
  [1:1] : 0 Crypto Erase Applies to Single Namespace(s)
  [0:0] : 0x1 Format Applies to All Namespace(s)

vwc : 0x1
  [0:0] : 0x1 Volatile Write Cache Present

awun : 0
awupf : 0
nvscc : 1
  [0:0] : 0x1 NVM Vendor Specific Commands uses NVMe Format

nwpc : 0
  [2:2] : 0 Permanent Write Protect Not Supported
  [1:1] : 0 Write Protect Until Power Supply Not Supported
  [0:0] : 0 No Write Protect and Write Protect Namespace Not Supported

acwu : 0
sgls : 0x10001
 [21:21]: 0 Transport SGL Data Block Descriptor Not Supported
 [20:20]: 0 Address Offsets Not Supported
 [19:19]: 0 Metadata Pointer Containing SGL Descriptor is Not Supported
 [18:18]: 0 SGL Length Larger than Buffer Not Supported
 [17:17]: 0 Byte-Aligned Contig. MD Buffer Not Supported
 [16:16]: 0x1 SGL Bit-Bucket Supported
  [2:2] : 0 Keyed SGL Data Block descriptor Not Supported
  [1:0] : 0x1 Scatter-Gather Lists Supported. No Dword alignment required.

mnan : 0
subnqn :
ioccsz : 0
iorcsz : 0
icdoff : 0
ctrattr : 0
  [0:0] : 0 Dynamic Controller Model

msdbd : 0
ps 0 : mp:12.00W operational enlat:15000 exlat:15000 rrt:0 rrl:0
          rwt:0 rwl:0 idle_power:- active_power:-
ps 1 : mp:11.00W operational enlat:15000 exlat:15000 rrt:1 rrl:1
          rwt:1 rwl:1 idle_power:- active_power:-
ps 2 : mp:9.00W operational enlat:15000 exlat:15000 rrt:2 rrl:2
          rwt:2 rwl:2 idle_power:- active_power:-

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1899503

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Xing Li (diegomontoya)]

No logs missing. Relevant log entries, lspci, and Nvme info is embedded in issue text.

[Kai-Heng Feng (kaihengfeng)]

Please test the kernel here:
https://people.canonical.com/~khfeng/lp1899503/

[Xing Li (diegomontoya)]

Kai, confirmed your patched kernel fixed issue.

To double confirm, I first rebooted with mount option "noinit_itable" removed and the WRITE_ZEROES errors immediately returned upon reboot.

I then switched over to your kernel, rebooted, and no errors. ext4lazyinit is running in the background and the errors logs are clean.

[Kai-Heng Feng (kaihengfeng)]

https://lkml.org/lkml/2020/10/13/135

[Kai-Heng Feng (kaihengfeng)]

Does this issue only happen on certain firmware versions?

[Xing Li (diegomontoya)]

The following is a reply from the ssd provider. I could could independently find info online regarding new firmwares. Note that Sandisk was folded into Western Digital.

"These are newest WD branded drives and WD already confirmed they are the newest firmware / no upgrades are available."

[Xing Li (diegomontoya)]

The drive in question is also known as (aka rebranded as) Western Digital Ultrastar DC SN620 series after Sandisk acquisition.