Kernel panic on `5.4.0-1033-gke` (Kernel panic - not syncing: Aiee, killing interrupt handler!) possibly iscsi related
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux-gke (Ubuntu) |
New
|
Undecided
|
Khaled El Mously | ||
Focal |
Fix Released
|
Undecided
|
Khaled El Mously |
Bug Description
[Impact]
Kernel panic during high iscsi activity
This stacktrace
[ 223.386958] BUG: scheduling while atomic: iscsiadm/
[ 223.393390] Modules linked in: tcp_diag inet_diag xt_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs sch_htb ebt_ip ebtable_filter ebtables veth xt_mark br_netfilter iptable_mangle xt_MASQUERADE xt_comment xt_addrtype iptable_nat binfmt_misc iptable_filter bpfilter xt_conntrack nf_nat bridge stp llc xfrm_user xfrm_algo aufs overlay nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper input_leds serio_raw sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_
[ 223.393429] CPU: 6 PID: 18136 Comm: iscsiadm Kdump: loaded Not tainted 5.4.0-1033-gke #35~18.04.1-Ubuntu
[ 223.393430] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 223.393430] Call Trace:
[ 223.393439] dump_stack+
[ 223.393464] __schedule_
[ 223.393467] __schedule+
[ 223.393469] schedule+0x33/0xa0
[ 223.393472] __lock_
[ 223.393475] ? wait_woken+
[ 223.393477] lock_sock_
[ 223.393479] inet_getname+
[ 223.393482] kernel_
[ 223.393485] iscsi_sw_
[ 223.393494] show_conn_
[ 223.393496] dev_attr_
[ 223.393499] sysfs_kf_
[ 223.393502] kernfs_
[ 223.393504] seq_read+0xda/0x420
[ 223.393506] kernfs_
[ 223.393510] __vfs_read+
[ 223.393512] vfs_read+0x8e/0x130
[ 223.393513] ksys_read+0xa7/0xe0
[ 223.393515] __x64_sys_
[ 223.393518] do_syscall_
[ 223.393521] entry_SYSCALL_
[ 223.393523] RIP: 0033:0x7f45793ce910
[ 223.393525] Code: b6 fe ff ff 48 8d 3d 0f be 08 00 48 83 ec 08 e8 06 db 01 00 66 0f 1f 44 00 00 83 3d f9 2d 2c 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 de 9b 01 00 48 89 04 24
[ 223.393526] RSP: 002b:00007ffd9f
[ 223.393527] RAX: ffffffffffffffda RBX: 00007ffd9fa13820 RCX: 00007f45793ce910
[ 223.393528] RDX: 0000000000000100 RSI: 00007ffd9fa13720 RDI: 0000000000000003
[ 223.393528] RBP: 00007ffd9fa13720 R08: 0000000000000000 R09: 0000000000000000
[ 223.393529] R10: 0000000000000064 R11: 0000000000000246 R12: 0000000000000003
[ 223.393530] R13: 00007ffd9fa13c60 R14: 0000555b0d613708 R15: 0000555b0d613300
[ 223.393581] sd 1:0:0:0: [sdb] Write Protect is off
[ 223.393583] sd 1:0:0:0: [sdb] Mode Sense: 43 00 10 08
[ 223.393660] iscsiadm[18136]: segfault at 7ffd9fa12e58 ip 0000555b0ccd95af sp 00007ffd9fa12e60 error 6 in iscsiadm[
[ 223.393666] Code: ba 00 02 00 00 48 81 ec 10 04 00 00 48 89 e7 48 8d 9c 24 00 02 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 08 04 00 00 31 c0 <e8> 3c ed 00 00 ba 00 02 00 00 4c 89 ee 48 89 e7 e8 6c ed 00 00 ba
[ 223.394992] sd 1:0:0:0: alua: transition timeout set to 60 seconds
[ 223.394997] sd 1:0:0:0: alua: port group 02 state N non-preferred supports TOlUSNA
[ 223.395018] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 223.395435] Kernel panic - not syncing: Aiee, killing interrupt handler!
[ 223.396802] sd 1:0:0:0: [sdb] Optimal transfer size 262144 bytes
[ 223.402387] CPU: 6 PID: 18136 Comm: iscsiadm Kdump: loaded Tainted: G W 5.4.0-1033-gke #35~18.04.1-Ubuntu
[ 223.402388] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 223.402389] Call Trace:
[ 223.402395] dump_stack+
[ 223.402398] panic+0xfe/0x2e4
[ 223.402400] do_exit+0x899/0xb90
[ 223.402402] do_group_
[ 223.402406] get_signal+
[ 223.402409] do_signal+
[ 223.402414] ? __bad_area_
[ 223.457597] exit_to_
[ 223.462090] prepare_
[ 223.466782] retint_user+0x8/0x8
[ 223.470131] RIP: 0033:0x555b0ccd95af
[ 223.473827] Code: ba 00 02 00 00 48 81 ec 10 04 00 00 48 89 e7 48 8d 9c 24 00 02 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 08 04 00 00 31 c0 <e8> 3c ed 00 00 ba 00 02 00 00 4c 89 ee 48 89 e7 e8 6c ed 00 00 ba
[ 223.492991] RSP: 002b:00007ffd9f
[ 223.498333] RAX: 0000000000000000 RBX: 00007ffd9fa13060 RCX: 00007f45793ce335
[ 223.505579] RDX: 0000000000000200 RSI: 0000555b0cf146a0 RDI: 00007ffd9fa12e60
[ 223.513025] RBP: 00007ffd9fa13ecc R08: 0000000000000000 R09: 0000000080808000
[ 223.520268] R10: 0000000000000075 R11: 0000000000000246 R12: 00007ffd9fa13540
[ 223.527521] R13: 00007ffd9fa13540 R14: 0000000000000200 R15: 0000555b0d613300
Which happens during high iscsi activity
This issue is also identified in linux-5.8, reported here ( https:/
[Fix]
The fix is https:/
[Test]
Affected customer has reported that they can no longer reproduce the problem with this fix applied. They were readily reproducing the crash without it.
[Regression potential]
I'm not aware of any. The patch seems reasonable. It is accepted in mainline and backported to the stable kernels too. It is present in groovy 5.8 as of https:/
no longer affects: | kernel-sru-workflow |
description: | updated |
Changed in linux-gcp (Ubuntu Focal): | |
assignee: | nobody → Khaled El Mously (kmously) |
no longer affects: | linux-gcp (Ubuntu) |
no longer affects: | linux-gcp (Ubuntu Focal) |
Changed in linux-gke (Ubuntu): | |
assignee: | nobody → Khaled El Mously (kmously) |
Changed in linux-gke (Ubuntu Focal): | |
assignee: | nobody → Khaled El Mously (kmously) |
description: | updated |
Changed in linux-gke (Ubuntu Focal): | |
status: | New → Fix Committed |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification- needed- focal' to 'verification- done-focal' . If the problem still exists, change the tag 'verification- needed- focal' to 'verification- failed- focal'.
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/ /wiki.ubuntu. com/Testing/ EnableProposed for documentation how to enable and use -proposed. Thank you!