"[Steps to Reproduce] 1. Dump manually when running IO; [Actual Results] [ 448.405504] CPU: 27 PID: 13560 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1 [ 448.405506] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019 [ 448.405507] Call trace: [ 448.405512] dump_backtrace+0x0/0x150 [ 448.405513] show_stack+0x24/0x30 [ 448.405516] dump_stack+0xa0/0xc4 [ 448.405518] __schedule_bug+0x68/0x88 [ 448.405520] __schedule+0x4b8/0x548 [ 448.405521] schedule+0x40/0xd0 [ 448.405523] schedule_timeout+0x200/0x378 [ 448.405524] __down+0x78/0xc8 [ 448.405526] down+0x54/0x70 [ 448.405533] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main] f=11): [M(11)][1[ 448.405535] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main] [ 448.405541] sas_queuecommand+0x168/0x1b0 [libsas] [ 448.405544] scsi_queue_rq+0x2ac/0x980 [ 448.405547] blk_mq_dispatch_rq_list+0xb0/0x550 [ 448.405548] blk_mq_do_dispatch_sched+0x6c/0x110 [ 448.405550] blk_mq_sched_dispatch_requests+0x114/0x1d8 [ 448.405551] __blk_mq_run_hw_queue+0xb8/0x130 [ 448.405552] __blk_mq_delay_run_hw_queue+0x1c0/0x220 [ 448.405553] blk_mq_run_hw_queue+0xb0/0x128 [ 448.405554] blk_mq_sched_insert_requests+0xdc/0x208 [ 448.405555] blk_mq_flush_plug_list+0x1b4/0x3a0 [ 448.405557] blk_flush_plug_list+0xdc/0x110 [ 448.405558] blk_finish_plug+0x3c/0x50 [ 448.405560] blkdev_write_iter+0xc0/0x130 [ 448.405562] aio_write+0xec/0x1a0 [ 448.405563] io_submit_one+0x4f4/0x8d8 [ 448.405564] __arm64_sys_io_submit+0xdc/0x280 [ 448.405566] el0_svc_common.constprop.0+0xe0/0x1e0 [ 448.405567] el0_svc_handler+0x34/0x90 [ 448.405569] el0_svc+0x10/0x14 [ 448.405571] CPU: 26 PID: 13559 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1 [ 448.405572] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019 [ 448.405572] Call trace: [ 448.405574] dump_backtrace+0x0/0x150 [ 448.405575] show_stack+0x24/0x30 [ 448.405577] dump_stack+0xa0/0xc4 [ 448.405578] __schedule_bug+0x68/0x88 [ 448.405580] __schedule+0x4b8/0x548 [ 448.405581] schedule+0x40/0xd0 [ 448.405582] schedule_timeout+0x200/0x378 [ 448.405583] __down+0x78/0xc8 [ 448.405584] down+0x54/0x70 [ 448.405587] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main] [ 448.405590] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main] [ 448.405594] sas_queuecommand+0x168/0x1b0 [libsas] [ 448.405595] scsi_queue_rq+0x2ac/0x980 [ 448.405596] blk_mq_dispatch_rq_list+0xb0/0x550 [ 448.405598] blk_mq_do_dispatch_sched+0x6c/0x110 [ 448.405599] blk_mq_sched_dispatch_requests+0x114/0x1d8 [ 448.405600] __blk_mq_run_hw_queue+0xb8/0x130 [ 448.405601] __blk_mq_delay_run_hw_queue+0x1c0/0x220 [ 448.405602] blk_mq_run_hw_queue+0xb0/0x128 [ 448.405603] blk_mq_sched_insert_requests+0xdc/0x208 [ 448.405605] blk_mq_flush_plug_list+0x1b4/0x3a0 [ 448.405606] blk_flush_plug_list+0xdc/0x110 [ 448.405607] blk_finish_plug+0x3c/0x50 [ 448.405608] blkdev_write_iter+0xc0/0x130 [ 448.405610] aio_write+0xec/0x1a0 [ 448.405611] io_submit_one+0x4f4/0x8d8 [ 448.405612] __arm64_sys_io_submit+0xdc/0x280 [ 448.405614] el0_svc_common.constprop.0+0xe0/0x1e0 [ 448.405615] el0_svc_handler+0x34/0x90 [ 448.405616] el0_svc+0x10/0x14 [ 448.405620] CPU: 86 PID: 13534 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1 1.7%][r=1163MiB/[ 448.405621] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019 [ 448.405622] Call trace: [ 448.405625] dump_backtrace+0x0/0x150 [ 448.405626] show_stack+0x24/0x30 [ 448.405631] dump_stack+0xa0/0xc4 [ 448.405635] __schedule_bug+0x68/0x88 [ 448.405636] __schedule+0x4b8/0x548 [ 448.405637] schedule+0x40/0xd0 [ 448.405639] schedule_timeout+0x200/0x378 [ 448.405640] __down+0x78/0xc8 [ 448.405642] down+0x54/0x70 [ 448.405646] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main] [ 448.405649] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main] [ 448.405654] sas_ata_qc_issue+0x194/0x220 [libsas] [ 448.405656] ata_qc_issue+0x118/0x2b8 [ 448.405657] ata_scsi_translate+0xe8/0x1c8 [ 448.405658] ata_sas_queuecmd+0xa0/0x218 [ 448.405662] sas_queuecommand+0xbc/0x1b0 [libsas] [ 448.405663] scsi_queue_rq+0x2ac/0x980 [ 448.405665] blk_mq_dispatch_rq_list+0xb0/0x550 [ 448.405667] blk_mq_do_dispatch_sched+0x6c/0x110 [ 448.405668] blk_mq_sched_dispatch_requests+0x114/0x1d8 [ 448.405669] __blk_mq_run_hw_queue+0xb8/0x130 [ 448.405670] __blk_mq_delay_run_hw_queue+0x1c0/0x220 [ 448.405672] blk_mq_run_hw_queue+0xb0/0x128 [ 448.405673] blk_mq_sched_insert_requests+0xdc/0x208 [ 448.405674] blk_mq_flush_plug_list+0x1b4/0x3a0 [ 448.405677] blk_flush_plug_list+0xdc/0x110 [ 448.405679] blk_finish_plug+0x3c/0x50 [ 448.405683] blkdev_direct_IO+0x404/0x550 [ 448.405686] generic_file_read_iter+0x9c/0x848 [ 448.405688] blkdev_read_iter+0x50/0x78 [ 448.405691] aio_read+0xc8/0x170 [ 448.405692] io_submit_one+0x1fc/0x8d8 [ 448.405694] __arm64_sys_io_submit+0xdc/0x280 [ 448.405697] el0_svc_common.constprop.0+0xe0/0x1e0 [ 448.405698] el0_svc_handler+0x34/0x90 [ 448.405699] el0_svc+0x10/0x14 [ 448.410674] Modules linked in: ib_ipoib ib_umad rpcrdma ib_iser libiscsi scsi_transport_iscsi vfio_iommu_type1 vfio_pci vfio_virqfd vfio hns_roce_hw_v2 crct10dif_ce hns3 hclge hnae3 hisi_hpre ses hisi_zip hisi_qm uacce ip_tables x_tables hisi_sas_v3_hw hisi_sas_main libsas scsi_transport_sas [ 448.416488] ib_ipoib ib_umad rpcrdma ib_iser libiscsi scsi_transport_iscsi vfio_iommu_type1 vfio_pci vfio_virqfd vfio hns_roce_hw_v2 crct10dif_ce hns3 hclge hnae3 hisi_hpre ses hisi_zip hisi_qm uacce ip_tables x_tables hisi_sas_v3_hw hisi_sas_main libsas scsi_transport_sas [ 448.422046] BUG: scheduling while atomic: fio/13566/0x00000002 [ 448.422048] Modules linked in: ib_ipoib ib_umad rpcrdma ib_iser libiscsi scsi_transport_iscsi vfio_iommu_type1 vfio_pci vfio_virqfd vfio hns_roce_hw_v2 crct10dif_ce hns3 hclge hnae3 hisi_hpre ses hisi_zip hisi_qm uacce ip_tables x_tables hisi_sas_v3_hw hisi_sas_main libsas scsi_transport_sas s,w=1164MiB/s][r[ 448.422062] CPU: 61 PID: 13566 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1 [ 448.422063] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019 [ 448.422064] Call trace: [ 448.422067] dump_backtrace+0x0/0x150 [ 448.422069] show_stack+0x24/0x30 [ 448.422071] dump_stack+0xa0/0xc4 [ 448.422073] __schedule_bug+0x68/0x88 [ 448.422075] __schedule+0x4b8/0x548 [ 448.422076] schedule+0x40/0xd0 [ 448.422077] schedule_timeout+0x200/0x378 [ 448.422079] __down+0x78/0xc8 [ 448.422081] down+0x54/0x70 [ 448.422085] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main] [ 448.422088] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main] [ 448.422093] sas_ata_qc_issue+0x194/0x220 [libsas] [ 448.422099] ata_qc_issue+0x118/0x2b8 [ 448.422101] ata_scsi_translate+0xe8/0x1c8 [ 448.422103] ata_sas_queuecmd+0xa0/0x218 [ 448.422106] sas_queuecommand+0xbc/0x1b0 [libsas] [ 448.422108] scsi_queue_rq+0x2ac/0x980 [ 448.422110] blk_mq_dispatch_rq_list+0xb0/0x550 [ 448.422111] blk_mq_do_dispatch_sched+0x6c/0x110 [ 448.422113] blk_mq_sched_dispatch_requests+0x114/0x1d8 [ 448.422114] __blk_mq_run_hw_queue+0xb8/0x130 [ 448.422115] __blk_mq_delay_run_hw_queue+0x1c0/0x220 [ 448.422116] blk_mq_run_hw_queue+0xb0/0x128 [ 448.422118] blk_mq_sched_insert_requests+0xdc/0x208 [ 448.422119] blk_mq_flush_plug_list+0x1b4/0x3a0 [ 448.422121] blk_flush_plug_list+0xdc/0x110 [ 448.422122] blk_finish_plug+0x3c/0x50 [ 448.422124] blkdev_direct_IO+0x404/0x550 [ 448.422126] generic_file_read_iter+0x9c/0x848 [ 448.422127] blkdev_read_iter+0x50/0x78 [ 448.422129] aio_read+0xc8/0x170 [ 448.422130] io_submit_one+0x1fc/0x8d8 [ 448.422132] __arm64_sys_io_submit+0xdc/0x280 [ 448.422134] el0_svc_common.constprop.0+0xe0/0x1e0 [ 448.422135] el0_svc_handler+0x34/0x90 [ 448.422136] el0_svc+0x10/0x14 [ 448.422305] CPU: 5 PID: 13564 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1 [ 448.422306] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019 [ 448.422307] Call trace: [ 448.422311] dump_backtrace+0x0/0x150 [ 448.422315] show_stack+0x24/0x30 [ 450.109478] dump_stack+0xa0/0xc4 =298k,w=298k IOP[ 450.112779] __schedule_bug+0x68/0x88 [ 450.117802] __schedule+0x4b8/0x548 [ 450.121276] schedule+0x40/0xd0 [ 450.124405] schedule_timeout+0x200/0x378 [ 450.128399] __down+0x78/0xc8 S][eta 03m:02s] [ 450.131356] down+0x54/0x70 [ 450.135520] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main] Jobs: 11 (f=11):[ 450.141760] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main] [ 450.148861] sas_queuecommand+0x168/0x1b0 [libsas] [M(11)][12.1%][[ 450.153631] scsi_queue_rq+0x2ac/0x980 [ 450.158743] blk_mq_dispatch_rq_list+0xb0/0x550 r=100KiB/s,w=52K[ 450.163254] blk_mq_do_dispatch_sched+0x6c/0x110 [ 450.169229] blk_mq_sched_dispatch_requests+0x114/0x1d8 iB/s][r=25,w=13 [ 450.174431] __blk_mq_run_hw_queue+0xb8/0x130 [ 450.180148] __blk_mq_delay_run_hw_queue+0x1c0/0x220 [ 450.185091] blk_mq_run_hw_queue+0xb0/0x128 IOPS][eta 03m:01[ 450.189259] blk_mq_sched_insert_requests+0xdc/0x208 [ 450.195580] blk_mq_flush_plug_list+0x1b4/0x3a0 [ 450.200093] blk_flush_plug_list+0xdc/0x110 s] [ 450.204258] blk_finish_plug+0x3c/0x50 [ 450.208852] blkdev_direct_IO+0x404/0x550 [ 450.212842] generic_file_read_iter+0x9c/0x848 [ 450.217265] blkdev_read_iter+0x50/0x78 [ 450.221083] aio_read+0xc8/0x170 [ 450.224295] io_submit_one+0x1fc/0x8d8 [ 450.228026] __arm64_sys_io_submit+0xdc/0x280 [ 450.232364] el0_svc_common.constprop.0+0xe0/0x1e0 [ 450.237132] el0_svc_handler+0x34/0x90 [ 450.240862] el0_svc+0x10/0x14 [ 450.243905] CPU: 23 PID: 13563 Comm: fio Tainted: G W 5.3.0-rc4-gae89c9a3-dirty #1 [ 450.243957] Unable to handle kernel paging request at virtual address 0000ffff784fb6f8 [ 450.252734] Hardware name: Huawei TaiShan 2280 V2/BC82AMDC, BIOS 2280-V2 CS V3.B080.01 09/19/2019 [ 450.252735] Call trace: [ 450.252738] dump_backtrace+0x0/0x150 [ 450.252739] show_stack+0x24/0x30 [ 450.252742] dump_stack+0xa0/0xc4 [ 450.260619] Mem abort info: [ 450.269446] __schedule_bug+0x68/0x88 [ 450.269447] __schedule+0x4b8/0x548 [ 450.269448] schedule+0x40/0xd0 [ 450.269450] schedule_timeout+0x200/0x378 [ 450.269451] __down+0x78/0xc8 [ 450.269453] down+0x54/0x70 [ 450.271888] ESR = 0x92000007 [ 450.271890] Exception class = DABT (lower EL), IL = 32 bits [ 450.275537] hisi_sas_task_exec.isra.10+0x598/0x8d8 [hisi_sas_main] [ 450.275539] hisi_sas_queue_command+0x28/0x38 [hisi_sas_main] [Expected Results] Call trace occurs [Reproducibility] occasionally [Additional information] Hardware: D06 CS Firmware: NA+I59 Kernel: NA [Resolution] To solve the issue, check preemptible() to avoid disabling preempt multiple when flag HISI_SAS_REJECT_CMD_BIT is set." scsi: hisi_sas: Replace in_softirq() check in hisi_sas_task_exec()