[sas-1126]scsi: hisi_sas: Replace in_softirq() check in hisi_sas_task_exec()

Bug #1854000 reported by Fred Kimmy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kunpeng920
Fix Released
Undecided
Unassigned
Ubuntu-18.04
Won't Fix
Undecided
Unassigned
Ubuntu-18.04-hwe
Fix Released
Undecided
Unassigned
Ubuntu-19.04
Won't Fix
Undecided
Unassigned
Ubuntu-19.10
Won't Fix
Undecided
Unassigned
Ubuntu-20.04
Fix Released
Undecided
Unassigned
Upstream-kernel
Fix Released
Undecided
Unassigned
linux (Ubuntu)
Fix Released
Undecided
dann frazier

Bug Description

"[Steps to Reproduce]
1. run fio
2“echo 1 > /sys/kernel/debug/hisi_sas/0000\:74\:02.0/trigger_dump”

[Actual Results]
[ 448.405504] CPU: 27 PID: 13560 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1
[ 448.405506] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019
[ 448.405507] Call trace:
[ 448.405512] dump_backtrace+0x0/0x150
[ 448.405513] show_stack+0x24/0x30
[ 448.405516] dump_stack+0xa0/0xc4
[ 448.405518] __schedule_bug+0x68/0x88
[ 448.405520] __schedule+0x4b8/0x548
[ 448.405521] schedule+0x40/0xd0
[ 448.405523] schedule_timeout+0x200/0x378
[ 448.405524] __down+0x78/0xc8
[ 448.405526] down+0x54/0x70
[ 448.405533] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main]
f=11): [M(11)][1[ 448.405535] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main]
[ 448.405541] sas_queuecommand+0x168/0x1b0 [libsas]
[ 448.405544] scsi_queue_rq+0x2ac/0x980
[ 448.405547] blk_mq_dispatch_rq_list+0xb0/0x550
[ 448.405548] blk_mq_do_dispatch_sched+0x6c/0x110
[ 448.405550] blk_mq_sched_dispatch_requests+0x114/0x1d8
[ 448.405551] __blk_mq_run_hw_queue+0xb8/0x130
[ 448.405552] __blk_mq_delay_run_hw_queue+0x1c0/0x220
[ 448.405553] blk_mq_run_hw_queue+0xb0/0x128
[ 448.405554] blk_mq_sched_insert_requests+0xdc/0x208
[ 448.405555] blk_mq_flush_plug_list+0x1b4/0x3a0
[ 448.405557] blk_flush_plug_list+0xdc/0x110
[ 448.405558] blk_finish_plug+0x3c/0x50
[ 448.405560] blkdev_write_iter+0xc0/0x130
[ 448.405562] aio_write+0xec/0x1a0
[ 448.405563] io_submit_one+0x4f4/0x8d8
[ 448.405564] __arm64_sys_io_submit+0xdc/0x280
[ 448.405566] el0_svc_common.constprop.0+0xe0/0x1e0
[ 448.405567] el0_svc_handler+0x34/0x90
[ 448.405569] el0_svc+0x10/0x14
[ 448.405571] CPU: 26 PID: 13559 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1
[ 448.405572] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019
[ 448.405572] Call trace:
[ 448.405574] dump_backtrace+0x0/0x150
[ 448.405575] show_stack+0x24/0x30
[ 448.405577] dump_stack+0xa0/0xc4
[ 448.405578] __schedule_bug+0x68/0x88
[ 448.405580] __schedule+0x4b8/0x548
[ 448.405581] schedule+0x40/0xd0
[ 448.405582] schedule_timeout+0x200/0x378
[ 448.405583] __down+0x78/0xc8
[ 448.405584] down+0x54/0x70
[ 448.405587] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main]
[ 448.405590] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main]
[ 448.405594] sas_queuecommand+0x168/0x1b0 [libsas]
[ 448.405595] scsi_queue_rq+0x2ac/0x980
[ 448.405596] blk_mq_dispatch_rq_list+0xb0/0x550
[ 448.405598] blk_mq_do_dispatch_sched+0x6c/0x110
[ 448.405599] blk_mq_sched_dispatch_requests+0x114/0x1d8
[ 448.405600] __blk_mq_run_hw_queue+0xb8/0x130
[ 448.405601] __blk_mq_delay_run_hw_queue+0x1c0/0x220
[ 448.405602] blk_mq_run_hw_queue+0xb0/0x128
[ 448.405603] blk_mq_sched_insert_requests+0xdc/0x208
[ 448.405605] blk_mq_flush_plug_list+0x1b4/0x3a0
[ 448.405606] blk_flush_plug_list+0xdc/0x110
[ 448.405607] blk_finish_plug+0x3c/0x50
[ 448.405608] blkdev_write_iter+0xc0/0x130
[ 448.405610] aio_write+0xec/0x1a0
[ 448.405611] io_submit_one+0x4f4/0x8d8
[ 448.405612] __arm64_sys_io_submit+0xdc/0x280
[ 448.405614] el0_svc_common.constprop.0+0xe0/0x1e0
[ 448.405615] el0_svc_handler+0x34/0x90
[ 448.405616] el0_svc+0x10/0x14
[ 448.405620] CPU: 86 PID: 13534 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1
1.7%][r=1163MiB/[ 448.405621] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019
[ 448.405622] Call trace:
[ 448.405625] dump_backtrace+0x0/0x150
[ 448.405626] show_stack+0x24/0x30
[ 448.405631] dump_stack+0xa0/0xc4
[ 448.405635] __schedule_bug+0x68/0x88
[ 448.405636] __schedule+0x4b8/0x548
[ 448.405637] schedule+0x40/0xd0
[ 448.405639] schedule_timeout+0x200/0x378
[ 448.405640] __down+0x78/0xc8
[ 448.405642] down+0x54/0x70
[ 448.405646] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main]
[ 448.405649] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main]
[ 448.405654] sas_ata_qc_issue+0x194/0x220 [libsas]
[ 448.405656] ata_qc_issue+0x118/0x2b8
[ 448.405657] ata_scsi_translate+0xe8/0x1c8
[ 448.405658] ata_sas_queuecmd+0xa0/0x218
[ 448.405662] sas_queuecommand+0xbc/0x1b0 [libsas]
[ 448.405663] scsi_queue_rq+0x2ac/0x980
[ 448.405665] blk_mq_dispatch_rq_list+0xb0/0x550
[ 448.405667] blk_mq_do_dispatch_sched+0x6c/0x110
[ 448.405668] blk_mq_sched_dispatch_requests+0x114/0x1d8
[ 448.405669] __blk_mq_run_hw_queue+0xb8/0x130
[ 448.405670] __blk_mq_delay_run_hw_queue+0x1c0/0x220
[ 448.405672] blk_mq_run_hw_queue+0xb0/0x128
[ 448.405673] blk_mq_sched_insert_requests+0xdc/0x208
[ 448.405674] blk_mq_flush_plug_list+0x1b4/0x3a0
[ 448.405677] blk_flush_plug_list+0xdc/0x110
[ 448.405679] blk_finish_plug+0x3c/0x50
[ 448.405683] blkdev_direct_IO+0x404/0x550
[ 448.405686] generic_file_read_iter+0x9c/0x848
[ 448.405688] blkdev_read_iter+0x50/0x78
[ 448.405691] aio_read+0xc8/0x170
[ 448.405692] io_submit_one+0x1fc/0x8d8
[ 448.405694] __arm64_sys_io_submit+0xdc/0x280
[ 448.405697] el0_svc_common.constprop.0+0xe0/0x1e0
[ 448.405698] el0_svc_handler+0x34/0x90
[ 448.405699] el0_svc+0x10/0x14
[ 448.410674] Modules linked in: ib_ipoib ib_umad rpcrdma ib_iser libiscsi scsi_transport_iscsi vfio_iommu_type1 vfio_pci vfio_virqfd vfio hns_roce_hw_v2 crct10dif_ce hns3 hclge hnae3 hisi_hpre ses hisi_zip hisi_qm uacce ip_tables x_tables hisi_sas_v3_hw hisi_sas_main libsas scsi_transport_sas
[ 448.416488] ib_ipoib ib_umad rpcrdma ib_iser libiscsi scsi_transport_iscsi vfio_iommu_type1 vfio_pci vfio_virqfd vfio hns_roce_hw_v2 crct10dif_ce hns3 hclge hnae3 hisi_hpre ses hisi_zip hisi_qm uacce ip_tables x_tables hisi_sas_v3_hw hisi_sas_main libsas scsi_transport_sas
[ 448.422046] BUG: scheduling while atomic: fio/13566/0x00000002
[ 448.422048] Modules linked in: ib_ipoib ib_umad rpcrdma ib_iser libiscsi scsi_transport_iscsi vfio_iommu_type1 vfio_pci vfio_virqfd vfio hns_roce_hw_v2 crct10dif_ce hns3 hclge hnae3 hisi_hpre ses hisi_zip hisi_qm uacce ip_tables x_tables hisi_sas_v3_hw hisi_sas_main libsas scsi_transport_sas
s,w=1164MiB/s][r[ 448.422062] CPU: 61 PID: 13566 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1
[ 448.422063] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019
[ 448.422064] Call trace:
[ 448.422067] dump_backtrace+0x0/0x150
[ 448.422069] show_stack+0x24/0x30
[ 448.422071] dump_stack+0xa0/0xc4
[ 448.422073] __schedule_bug+0x68/0x88
[ 448.422075] __schedule+0x4b8/0x548
[ 448.422076] schedule+0x40/0xd0
[ 448.422077] schedule_timeout+0x200/0x378
[ 448.422079] __down+0x78/0xc8
[ 448.422081] down+0x54/0x70
[ 448.422085] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main]
[ 448.422088] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main]
[ 448.422093] sas_ata_qc_issue+0x194/0x220 [libsas]
[ 448.422099] ata_qc_issue+0x118/0x2b8
[ 448.422101] ata_scsi_translate+0xe8/0x1c8
[ 448.422103] ata_sas_queuecmd+0xa0/0x218
[ 448.422106] sas_queuecommand+0xbc/0x1b0 [libsas]
[ 448.422108] scsi_queue_rq+0x2ac/0x980
[ 448.422110] blk_mq_dispatch_rq_list+0xb0/0x550
[ 448.422111] blk_mq_do_dispatch_sched+0x6c/0x110
[ 448.422113] blk_mq_sched_dispatch_requests+0x114/0x1d8
[ 448.422114] __blk_mq_run_hw_queue+0xb8/0x130
[ 448.422115] __blk_mq_delay_run_hw_queue+0x1c0/0x220
[ 448.422116] blk_mq_run_hw_queue+0xb0/0x128
[ 448.422118] blk_mq_sched_insert_requests+0xdc/0x208
[ 448.422119] blk_mq_flush_plug_list+0x1b4/0x3a0
[ 448.422121] blk_flush_plug_list+0xdc/0x110
[ 448.422122] blk_finish_plug+0x3c/0x50
[ 448.422124] blkdev_direct_IO+0x404/0x550
[ 448.422126] generic_file_read_iter+0x9c/0x848
[ 448.422127] blkdev_read_iter+0x50/0x78
[ 448.422129] aio_read+0xc8/0x170
[ 448.422130] io_submit_one+0x1fc/0x8d8
[ 448.422132] __arm64_sys_io_submit+0xdc/0x280
[ 448.422134] el0_svc_common.constprop.0+0xe0/0x1e0
[ 448.422135] el0_svc_handler+0x34/0x90
[ 448.422136] el0_svc+0x10/0x14
[ 448.422305] CPU: 5 PID: 13564 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1
[ 448.422306] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019
[ 448.422307] Call trace:
[ 448.422311] dump_backtrace+0x0/0x150
[ 448.422315] show_stack+0x24/0x30
[ 450.109478] dump_stack+0xa0/0xc4
=298k,w=298k IOP[ 450.112779] __schedule_bug+0x68/0x88
[ 450.117802] __schedule+0x4b8/0x548
[ 450.121276] schedule+0x40/0xd0
[ 450.124405] schedule_timeout+0x200/0x378
[ 450.128399] __down+0x78/0xc8
S][eta 03m:02s]
[ 450.131356] down+0x54/0x70
[ 450.135520] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main]
Jobs: 11 (f=11):[ 450.141760] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main]
[ 450.148861] sas_queuecommand+0x168/0x1b0 [libsas]
 [M(11)][12.1%][[ 450.153631] scsi_queue_rq+0x2ac/0x980
[ 450.158743] blk_mq_dispatch_rq_list+0xb0/0x550
r=100KiB/s,w=52K[ 450.163254] blk_mq_do_dispatch_sched+0x6c/0x110
[ 450.169229] blk_mq_sched_dispatch_requests+0x114/0x1d8
iB/s][r=25,w=13 [ 450.174431] __blk_mq_run_hw_queue+0xb8/0x130
[ 450.180148] __blk_mq_delay_run_hw_queue+0x1c0/0x220
[ 450.185091] blk_mq_run_hw_queue+0xb0/0x128
IOPS][eta 03m:01[ 450.189259] blk_mq_sched_insert_requests+0xdc/0x208
[ 450.195580] blk_mq_flush_plug_list+0x1b4/0x3a0
[ 450.200093] blk_flush_plug_list+0xdc/0x110
s]
[ 450.204258] blk_finish_plug+0x3c/0x50
[ 450.208852] blkdev_direct_IO+0x404/0x550
[ 450.212842] generic_file_read_iter+0x9c/0x848
[ 450.217265] blkdev_read_iter+0x50/0x78
[ 450.221083] aio_read+0xc8/0x170
[ 450.224295] io_submit_one+0x1fc/0x8d8
[ 450.228026] __arm64_sys_io_submit+0xdc/0x280
[ 450.232364] el0_svc_common.constprop.0+0xe0/0x1e0
[ 450.237132] el0_svc_handler+0x34/0x90
[ 450.240862] el0_svc+0x10/0x14
[ 450.243905] CPU: 23 PID: 13563 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1
[ 450.243957] Unable to handle kernel paging request at virtual address 0000ffff784fb6f8
[ 450.252734] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019
[ 450.252735] Call trace:
[ 450.252738] dump_backtrace+0x0/0x150
[ 450.252739] show_stack+0x24/0x30
[ 450.252742] dump_stack+0xa0/0xc4
[ 450.260619] Mem abort info:
[ 450.269446] __schedule_bug+0x68/0x88
[ 450.269447] __schedule+0x4b8/0x548
[ 450.269448] schedule+0x40/0xd0
[ 450.269450] schedule_timeout+0x200/0x378
[ 450.269451] __down+0x78/0xc8
[ 450.269453] down+0x54/0x70
[ 450.271888] ESR = 0x92000007
[ 450.271890] Exception class = DABT (lower EL), IL = 32 bits
[ 450.275537] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main]
[ 450.275539] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main]

[Expected Results]
Call trace occurs

[Reproducibility]
occasionally

[Additional information]
Hardware: D06 CS
Firmware: NA+I59
Kernel: NA

[Resolution]
To solve the issue, check preemptible() to avoid disabling preempt multiple
when flag HISI_SAS_REJECT_CMD_BIT is set."

scsi: hisi_sas: Replace in_softirq() check in hisi_sas_task_exec()

description: updated
dann frazier (dannf)
Changed in kunpeng920:
status: New → Triaged
Revision history for this message
dann frazier (dannf) wrote :

debugfs interfaces are not considered stable, so this does not qualify for an SRU. Will investigate backporting into the 20.04 kernel (which should be v5.4) once it lands in Linus' tree (it is currently in linux-next).

dann frazier (dannf)
Changed in linux (Ubuntu):
status: New → In Progress
assignee: nobody → dann frazier (dannf)
Ike Panhc (ikepanhc)
Changed in kunpeng920:
status: Triaged → In Progress
Revision history for this message
Ike Panhc (ikepanhc) wrote :
Changed in linux (Ubuntu):
status: In Progress → Fix Committed
Changed in kunpeng920:
status: In Progress → Fix Committed
Revision history for this message
Andrew Cloke (andrew-cloke) wrote :

Patch has now progressed to focal/master.

Changed in linux (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Ike Panhc (ikepanhc) wrote :

In 5.3 HWE kernel already.

Changed in kunpeng920:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.