blk_update_request: I/O error

Bug #2102025 reported by Steve
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
New
Undecided
Unassigned

Bug Description

Hi,

We have multiple server os crash with the attached screenshot error. The ubuntu versions that we use x.x.x.122 and x.x.x.124.

cat /proc/version_signature
Ubuntu 5.15.0-124.134-generic 5.15.163

The os gots read only and the only way to bring it back is to reset the server.
Hardware wise we don't really see any issue, we even checked with the vendor, however they also didn't find anything.

We would like to eliminate this issue ASAP because the running system is critical.

Thank you for your help.
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Mar 18 11:12 seq
 crw-rw---- 1 root audio 116, 33 Mar 18 11:12 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: [Errno 2] No such file or directory: 'fuser'
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HPE ProLiant DL385 Gen10 Plus v2
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 LANGUAGE=en_US:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.15.0-124-generic root=/dev/mapper/vg_sys-lv_root ro crashkernel=auto audit=1 audit_backlog_limit=8192 nopti noibrs noretp spectre_v2=off scsi_mod.use_blk_mq=1 dm_mod.use_blk_mq=y processor.max_cstate=0 iommu=pt amd_iommu=on audit=1
ProcVersionSignature: Ubuntu 5.15.0-124.134-generic 5.15.163
RelatedPackageVersions:
 linux-restricted-modules-5.15.0-124-generic N/A
 linux-backports-modules-5.15.0-124-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3.36
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy
Uname: Linux 5.15.0-124-generic x86_64
UpgradeStatus: Upgraded to jammy on 2024-12-24 (84 days ago)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 02/21/2025
dmi.bios.release: 3.62
dmi.bios.vendor: HPE
dmi.bios.version: A42
dmi.board.name: ProLiant DL385 Gen10 Plus v2
dmi.board.vendor: HPE
dmi.chassis.type: 23
dmi.chassis.vendor: HPE
dmi.ec.firmware.release: 3.11
dmi.modalias: dmi:bvnHPE:bvrA42:bd02/21/2025:br3.62:efr3.11:svnHPE:pnProLiantDL385Gen10Plusv2:pvr:rvnHPE:rnProLiantDL385Gen10Plusv2:rvr:cvnHPE:ct23:cvr:skuP38411-B21:
dmi.product.family: ProLiant
dmi.product.name: ProLiant DL385 Gen10 Plus v2
dmi.product.sku: P38411-B21
dmi.sys.vendor: HPE
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Mar 6 10:30 seq
 crw-rw---- 1 root audio 116, 33 Mar 6 10:30 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.6
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: [Errno 2] No such file or directory: 'fuser'
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: HPE ProLiant DL385 Gen10 Plus v2
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 LANGUAGE=en_US:
 TERM=xterm
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 mgag200drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.15.0-124-generic root=/dev/mapper/vg_sys-lv_root ro crashkernel=auto audit=1 audit_backlog_limit=8192 nopti noibrs noretp spectre_v2=off scsi_mod.use_blk_mq=1 dm_mod.use_blk_mq=y processor.max_cstate=0 iommu=pt amd_iommu=on audit=1
ProcVersionSignature: Ubuntu 5.15.0-124.134-generic 5.15.163
RelatedPackageVersions:
 linux-restricted-modules-5.15.0-124-generic N/A
 linux-backports-modules-5.15.0-124-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3.36
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy
Uname: Linux 5.15.0-124-generic x86_64
UpgradeStatus: Upgraded to jammy on 2024-12-26 (82 days ago)
UserGroups: N/A
_MarkForUpload: True
dmi.bios.date: 10/04/2024
dmi.bios.release: 3.40
dmi.bios.vendor: HPE
dmi.bios.version: A42
dmi.board.name: ProLiant DL385 Gen10 Plus v2
dmi.board.vendor: HPE
dmi.chassis.type: 23
dmi.chassis.vendor: HPE
dmi.ec.firmware.release: 3.9
dmi.modalias: dmi:bvnHPE:bvrA42:bd10/04/2024:br3.40:efr3.9:svnHPE:pnProLiantDL385Gen10Plusv2:pvr:rvnHPE:rnProLiantDL385Gen10Plusv2:rvr:cvnHPE:ct23:cvr:skuP38411-B21:
dmi.product.family: ProLiant
dmi.product.name: ProLiant DL385 Gen10 Plus v2
dmi.product.sku: P38411-B21
dmi.sys.vendor: HPE

Revision history for this message
Steve (badb0y) wrote :
Steve (badb0y)
description: updated
Revision history for this message
Chris Chiu (mschiu77) wrote :

Hi Steve, thank you for submitting this and for helping to improve Ubuntu.

Can you install and use the `apport-collect` to help collect information on this ticket. Thanks
`

Revision history for this message
Steve (badb0y) wrote : CRDA.txt

apport information

tags: added: apport-collected jammy
description: updated
Revision history for this message
Steve (badb0y) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Steve (badb0y) wrote : Lspci.txt

apport information

Revision history for this message
Steve (badb0y) wrote : Lspci-vt.txt

apport information

Revision history for this message
Steve (badb0y) wrote : Lsusb.txt

apport information

Revision history for this message
Steve (badb0y) wrote : Lsusb-t.txt

apport information

Revision history for this message
Steve (badb0y) wrote : Lsusb-v.txt

apport information

Revision history for this message
Steve (badb0y) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Steve (badb0y) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Steve (badb0y) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Steve (badb0y) wrote : ProcModules.txt

apport information

Revision history for this message
Steve (badb0y) wrote : UdevDb.txt

apport information

Revision history for this message
Steve (badb0y) wrote : WifiSyslog.txt
Revision history for this message
Steve (badb0y) wrote : acpidump.txt

apport information

description: updated
Revision history for this message
Steve (badb0y) wrote : CRDA.txt

apport information

Revision history for this message
Steve (badb0y) wrote : CurrentDmesg.txt

apport information

Revision history for this message
Steve (badb0y) wrote : Lspci.txt

apport information

Revision history for this message
Steve (badb0y) wrote : Lspci-vt.txt

apport information

Revision history for this message
Steve (badb0y) wrote : Lsusb.txt

apport information

Revision history for this message
Steve (badb0y) wrote : Lsusb-t.txt

apport information

Revision history for this message
Steve (badb0y) wrote : Lsusb-v.txt

apport information

Revision history for this message
Steve (badb0y) wrote : ProcCpuinfo.txt

apport information

Revision history for this message
Steve (badb0y) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Steve (badb0y) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Steve (badb0y) wrote : ProcModules.txt

apport information

Revision history for this message
Steve (badb0y) wrote : UdevDb.txt

apport information

Revision history for this message
Steve (badb0y) wrote : WifiSyslog.txt
Revision history for this message
Steve (badb0y) wrote : acpidump.txt

apport information

Revision history for this message
Steve (badb0y) wrote :

Added from 2 different crashed servers.

Revision history for this message
Chris Chiu (mschiu77) wrote :

Found lots of disk related error.
[ 1656.051694] ssaducli[25093]: segfault at 1638000 ip 00000000008e3668 sp 00007fff515d79a8 error 4 in ssaducli[400000+a27000]
[ 1656.051705] Code: c3 90 48 8b 07 41 89 d2 44 0f b7 40 02 41 83 e8 04 66 41 83 f8 03 76 45 48 8d 48 04 eb 09 66 66 66 90 40 84 ff 75 32 40 38 31 <44> 0f b7 49 02 0f 94 c2 44 38 51 01 0f 94 c0 20 c2 89 d7 75 0e 41
[ 1713.170336] sd 0:0:2:0: [sdc] tag#661 Sense Key : Recovered Error [current] [descriptor]
[ 1713.170342] sd 0:0:2:0: [sdc] tag#661 Add. Sense: Defect list not found

Dose it ever work on old kernel version?

Revision history for this message
Steve (badb0y) wrote (last edit ):

It is actually defect list not found, and it is not related to the OS disk, is it?
Could this be related you think? Not sure what is this error though.

Revision history for this message
Chris Chiu (mschiu77) wrote :

what's your os disk? I can see dm-44 been mounted read-only. I need to know it's because of disk error or kernel issue. It would be better for us to identify the problem if there was a working kernel.

[ 72.436547] systemd[1]: Starting Remount Root and Kernel File Systems...
[ 72.447164] EXT4-fs (dm-44): re-mounted. Opts: errors=remount-ro. Quota mode: none.

[ 73.577539] EXT4-fs (sdo2): mounting ext2 file system using the ext4 subsystem
[ 73.579367] EXT4-fs (sdo2): mounted filesystem without journal. Opts: (null). Quota mode: none.
[ 73.653782] FAT-fs (sdo1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

Revision history for this message
Steve (badb0y) wrote (last edit ):

The operating system disk is /dev/sdo.

Yeah, that's the issue, it became read only which results crash of the server.

4 servers are currently affected. 3 of them have different operating system disk.

We have never faced this issue in ubuntu 20.04, this started to occur since we moved to 22.04.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.