| 2021-03-30 05:58:31 |
Khaled El Mously |
bug |
|
|
added bug |
| 2021-03-30 05:59:11 |
Khaled El Mously |
bug task added |
|
linux-gcp (Ubuntu) |
|
| 2021-03-30 05:59:18 |
Khaled El Mously |
bug task deleted |
kernel-sru-workflow |
|
|
| 2021-03-30 05:59:32 |
Khaled El Mously |
nominated for series |
|
Ubuntu Focal |
|
| 2021-03-30 05:59:32 |
Khaled El Mously |
bug task added |
|
linux-gcp (Ubuntu Focal) |
|
| 2021-03-30 06:07:43 |
Khaled El Mously |
description |
This stacktrace
[ 223.386958] BUG: scheduling while atomic: iscsiadm/18136/0x00000200
[ 223.393390] Modules linked in: tcp_diag inet_diag xt_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs sch_htb ebt_ip ebtable_filter ebtables veth xt_mark br_netfilter iptable_mangle xt_MASQUERADE xt_comment xt_addrtype iptable_nat binfmt_misc iptable_filter bpfilter xt_conntrack nf_nat bridge stp llc xfrm_user xfrm_algo aufs overlay nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper input_leds serio_raw sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi virtio_rng ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear psmouse virtio_net net_failover failover
[ 223.393429] CPU: 6 PID: 18136 Comm: iscsiadm Kdump: loaded Not tainted 5.4.0-1033-gke #35~18.04.1-Ubuntu
[ 223.393430] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 223.393430] Call Trace:
[ 223.393439] dump_stack+0x6d/0x95
[ 223.393464] __schedule_bug+0x55/0x70
[ 223.393467] __schedule+0x61b/0x710
[ 223.393469] schedule+0x33/0xa0
[ 223.393472] __lock_sock+0x7d/0xc0
[ 223.393475] ? wait_woken+0x80/0x80
[ 223.393477] lock_sock_nested+0x64/0x70
[ 223.393479] inet_getname+0xaa/0xe0
[ 223.393482] kernel_getpeername+0x1b/0x20
[ 223.393485] iscsi_sw_tcp_conn_get_param+0xa6/0x110 [iscsi_tcp]
[ 223.393494] show_conn_ep_param_ISCSI_PARAM_CONN_ADDRESS+0x7e/0xa0 [scsi_transport_iscsi]
[ 223.393496] dev_attr_show+0x1d/0x50
[ 223.393499] sysfs_kf_seq_show+0xa1/0x110
[ 223.393502] kernfs_seq_show+0x27/0x30
[ 223.393504] seq_read+0xda/0x420
[ 223.393506] kernfs_fop_read+0x141/0x1a0
[ 223.393510] __vfs_read+0x1b/0x40
[ 223.393512] vfs_read+0x8e/0x130
[ 223.393513] ksys_read+0xa7/0xe0
[ 223.393515] __x64_sys_read+0x1a/0x20
[ 223.393518] do_syscall_64+0x57/0x190
[ 223.393521] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 223.393523] RIP: 0033:0x7f45793ce910
[ 223.393525] Code: b6 fe ff ff 48 8d 3d 0f be 08 00 48 83 ec 08 e8 06 db 01 00 66 0f 1f 44 00 00 83 3d f9 2d 2c 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 de 9b 01 00 48 89 04 24
[ 223.393526] RSP: 002b:00007ffd9fa13688 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 223.393527] RAX: ffffffffffffffda RBX: 00007ffd9fa13820 RCX: 00007f45793ce910
[ 223.393528] RDX: 0000000000000100 RSI: 00007ffd9fa13720 RDI: 0000000000000003
[ 223.393528] RBP: 00007ffd9fa13720 R08: 0000000000000000 R09: 0000000000000000
[ 223.393529] R10: 0000000000000064 R11: 0000000000000246 R12: 0000000000000003
[ 223.393530] R13: 00007ffd9fa13c60 R14: 0000555b0d613708 R15: 0000555b0d613300
[ 223.393581] sd 1:0:0:0: [sdb] Write Protect is off
[ 223.393583] sd 1:0:0:0: [sdb] Mode Sense: 43 00 10 08
[ 223.393660] iscsiadm[18136]: segfault at 7ffd9fa12e58 ip 0000555b0ccd95af sp 00007ffd9fa12e60 error 6 in iscsiadm[555b0ccb4000+58000]
[ 223.393666] Code: ba 00 02 00 00 48 81 ec 10 04 00 00 48 89 e7 48 8d 9c 24 00 02 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 08 04 00 00 31 c0 <e8> 3c ed 00 00 ba 00 02 00 00 4c 89 ee 48 89 e7 e8 6c ed 00 00 ba
[ 223.394992] sd 1:0:0:0: alua: transition timeout set to 60 seconds
[ 223.394997] sd 1:0:0:0: alua: port group 02 state N non-preferred supports TOlUSNA
[ 223.395018] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 223.395435] Kernel panic - not syncing: Aiee, killing interrupt handler!
[ 223.396802] sd 1:0:0:0: [sdb] Optimal transfer size 262144 bytes
[ 223.402387] CPU: 6 PID: 18136 Comm: iscsiadm Kdump: loaded Tainted: G W 5.4.0-1033-gke #35~18.04.1-Ubuntu
[ 223.402388] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 223.402389] Call Trace:
[ 223.402395] dump_stack+0x6d/0x95
[ 223.402398] panic+0xfe/0x2e4
[ 223.402400] do_exit+0x899/0xb90
[ 223.402402] do_group_exit+0x43/0xa0
[ 223.402406] get_signal+0x14f/0x860
[ 223.402409] do_signal+0x34/0x6d0
[ 223.402414] ? __bad_area_nosemaphore+0x149/0x1f0
[ 223.457597] exit_to_usermode_loop+0x8e/0x100
[ 223.462090] prepare_exit_to_usermode+0x91/0xa0
[ 223.466782] retint_user+0x8/0x8
[ 223.470131] RIP: 0033:0x555b0ccd95af
[ 223.473827] Code: ba 00 02 00 00 48 81 ec 10 04 00 00 48 89 e7 48 8d 9c 24 00 02 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 08 04 00 00 31 c0 <e8> 3c ed 00 00 ba 00 02 00 00 4c 89 ee 48 89 e7 e8 6c ed 00 00 ba
[ 223.492991] RSP: 002b:00007ffd9fa12e60 EFLAGS: 00010246
[ 223.498333] RAX: 0000000000000000 RBX: 00007ffd9fa13060 RCX: 00007f45793ce335
[ 223.505579] RDX: 0000000000000200 RSI: 0000555b0cf146a0 RDI: 00007ffd9fa12e60
[ 223.513025] RBP: 00007ffd9fa13ecc R08: 0000000000000000 R09: 0000000080808000
[ 223.520268] R10: 0000000000000075 R11: 0000000000000246 R12: 00007ffd9fa13540
[ 223.527521] R13: 00007ffd9fa13540 R14: 0000000000000200 R15: 0000555b0d613300
Which happens during high iscsi activity |
This stacktrace
[ 223.386958] BUG: scheduling while atomic: iscsiadm/18136/0x00000200
[ 223.393390] Modules linked in: tcp_diag inet_diag xt_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs sch_htb ebt_ip ebtable_filter ebtables veth xt_mark br_netfilter iptable_mangle xt_MASQUERADE xt_comment xt_addrtype iptable_nat binfmt_misc iptable_filter bpfilter xt_conntrack nf_nat bridge stp llc xfrm_user xfrm_algo aufs overlay nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper input_leds serio_raw sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi virtio_rng ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear psmouse virtio_net net_failover failover
[ 223.393429] CPU: 6 PID: 18136 Comm: iscsiadm Kdump: loaded Not tainted 5.4.0-1033-gke #35~18.04.1-Ubuntu
[ 223.393430] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 223.393430] Call Trace:
[ 223.393439] dump_stack+0x6d/0x95
[ 223.393464] __schedule_bug+0x55/0x70
[ 223.393467] __schedule+0x61b/0x710
[ 223.393469] schedule+0x33/0xa0
[ 223.393472] __lock_sock+0x7d/0xc0
[ 223.393475] ? wait_woken+0x80/0x80
[ 223.393477] lock_sock_nested+0x64/0x70
[ 223.393479] inet_getname+0xaa/0xe0
[ 223.393482] kernel_getpeername+0x1b/0x20
[ 223.393485] iscsi_sw_tcp_conn_get_param+0xa6/0x110 [iscsi_tcp]
[ 223.393494] show_conn_ep_param_ISCSI_PARAM_CONN_ADDRESS+0x7e/0xa0 [scsi_transport_iscsi]
[ 223.393496] dev_attr_show+0x1d/0x50
[ 223.393499] sysfs_kf_seq_show+0xa1/0x110
[ 223.393502] kernfs_seq_show+0x27/0x30
[ 223.393504] seq_read+0xda/0x420
[ 223.393506] kernfs_fop_read+0x141/0x1a0
[ 223.393510] __vfs_read+0x1b/0x40
[ 223.393512] vfs_read+0x8e/0x130
[ 223.393513] ksys_read+0xa7/0xe0
[ 223.393515] __x64_sys_read+0x1a/0x20
[ 223.393518] do_syscall_64+0x57/0x190
[ 223.393521] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 223.393523] RIP: 0033:0x7f45793ce910
[ 223.393525] Code: b6 fe ff ff 48 8d 3d 0f be 08 00 48 83 ec 08 e8 06 db 01 00 66 0f 1f 44 00 00 83 3d f9 2d 2c 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 de 9b 01 00 48 89 04 24
[ 223.393526] RSP: 002b:00007ffd9fa13688 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 223.393527] RAX: ffffffffffffffda RBX: 00007ffd9fa13820 RCX: 00007f45793ce910
[ 223.393528] RDX: 0000000000000100 RSI: 00007ffd9fa13720 RDI: 0000000000000003
[ 223.393528] RBP: 00007ffd9fa13720 R08: 0000000000000000 R09: 0000000000000000
[ 223.393529] R10: 0000000000000064 R11: 0000000000000246 R12: 0000000000000003
[ 223.393530] R13: 00007ffd9fa13c60 R14: 0000555b0d613708 R15: 0000555b0d613300
[ 223.393581] sd 1:0:0:0: [sdb] Write Protect is off
[ 223.393583] sd 1:0:0:0: [sdb] Mode Sense: 43 00 10 08
[ 223.393660] iscsiadm[18136]: segfault at 7ffd9fa12e58 ip 0000555b0ccd95af sp 00007ffd9fa12e60 error 6 in iscsiadm[555b0ccb4000+58000]
[ 223.393666] Code: ba 00 02 00 00 48 81 ec 10 04 00 00 48 89 e7 48 8d 9c 24 00 02 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 08 04 00 00 31 c0 <e8> 3c ed 00 00 ba 00 02 00 00 4c 89 ee 48 89 e7 e8 6c ed 00 00 ba
[ 223.394992] sd 1:0:0:0: alua: transition timeout set to 60 seconds
[ 223.394997] sd 1:0:0:0: alua: port group 02 state N non-preferred supports TOlUSNA
[ 223.395018] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 223.395435] Kernel panic - not syncing: Aiee, killing interrupt handler!
[ 223.396802] sd 1:0:0:0: [sdb] Optimal transfer size 262144 bytes
[ 223.402387] CPU: 6 PID: 18136 Comm: iscsiadm Kdump: loaded Tainted: G W 5.4.0-1033-gke #35~18.04.1-Ubuntu
[ 223.402388] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 223.402389] Call Trace:
[ 223.402395] dump_stack+0x6d/0x95
[ 223.402398] panic+0xfe/0x2e4
[ 223.402400] do_exit+0x899/0xb90
[ 223.402402] do_group_exit+0x43/0xa0
[ 223.402406] get_signal+0x14f/0x860
[ 223.402409] do_signal+0x34/0x6d0
[ 223.402414] ? __bad_area_nosemaphore+0x149/0x1f0
[ 223.457597] exit_to_usermode_loop+0x8e/0x100
[ 223.462090] prepare_exit_to_usermode+0x91/0xa0
[ 223.466782] retint_user+0x8/0x8
[ 223.470131] RIP: 0033:0x555b0ccd95af
[ 223.473827] Code: ba 00 02 00 00 48 81 ec 10 04 00 00 48 89 e7 48 8d 9c 24 00 02 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 08 04 00 00 31 c0 <e8> 3c ed 00 00 ba 00 02 00 00 4c 89 ee 48 89 e7 e8 6c ed 00 00 ba
[ 223.492991] RSP: 002b:00007ffd9fa12e60 EFLAGS: 00010246
[ 223.498333] RAX: 0000000000000000 RBX: 00007ffd9fa13060 RCX: 00007f45793ce335
[ 223.505579] RDX: 0000000000000200 RSI: 0000555b0cf146a0 RDI: 00007ffd9fa12e60
[ 223.513025] RBP: 00007ffd9fa13ecc R08: 0000000000000000 R09: 0000000080808000
[ 223.520268] R10: 0000000000000075 R11: 0000000000000246 R12: 00007ffd9fa13540
[ 223.527521] R13: 00007ffd9fa13540 R14: 0000000000000200 R15: 0000555b0d613300
Which happens during high iscsi activity
This issue is also identified in linux-5.8, reported here ( https://lkml.org/lkml/2020/7/28/1085 ) for example. It affects the gcp-5.4 kernel specifically because gcp-5.4 has backported '1b66d253610c7 ("bpf: Add get{peer, sock}name attach types for sock_addr")' which introduces the issue.
The fix is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bcf3a2953d36bbfb9bd44ccb3db0897d935cc485 from 5.9 |
|
| 2021-03-30 06:07:55 |
Khaled El Mously |
linux-gcp (Ubuntu Focal): assignee |
|
Khaled El Mously (kmously) |
|
| 2021-04-01 07:52:21 |
Khaled El Mously |
bug task added |
|
linux-gke (Ubuntu) |
|
| 2021-04-01 07:52:31 |
Khaled El Mously |
bug task deleted |
linux-gcp (Ubuntu) |
|
|
| 2021-04-01 07:52:39 |
Khaled El Mously |
bug task deleted |
linux-gcp (Ubuntu Focal) |
|
|
| 2021-04-01 07:52:44 |
Khaled El Mously |
linux-gke (Ubuntu): assignee |
|
Khaled El Mously (kmously) |
|
| 2021-04-01 07:52:48 |
Khaled El Mously |
linux-gke (Ubuntu Focal): assignee |
|
Khaled El Mously (kmously) |
|
| 2021-04-01 08:00:50 |
Khaled El Mously |
description |
This stacktrace
[ 223.386958] BUG: scheduling while atomic: iscsiadm/18136/0x00000200
[ 223.393390] Modules linked in: tcp_diag inet_diag xt_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs sch_htb ebt_ip ebtable_filter ebtables veth xt_mark br_netfilter iptable_mangle xt_MASQUERADE xt_comment xt_addrtype iptable_nat binfmt_misc iptable_filter bpfilter xt_conntrack nf_nat bridge stp llc xfrm_user xfrm_algo aufs overlay nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper input_leds serio_raw sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi virtio_rng ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear psmouse virtio_net net_failover failover
[ 223.393429] CPU: 6 PID: 18136 Comm: iscsiadm Kdump: loaded Not tainted 5.4.0-1033-gke #35~18.04.1-Ubuntu
[ 223.393430] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 223.393430] Call Trace:
[ 223.393439] dump_stack+0x6d/0x95
[ 223.393464] __schedule_bug+0x55/0x70
[ 223.393467] __schedule+0x61b/0x710
[ 223.393469] schedule+0x33/0xa0
[ 223.393472] __lock_sock+0x7d/0xc0
[ 223.393475] ? wait_woken+0x80/0x80
[ 223.393477] lock_sock_nested+0x64/0x70
[ 223.393479] inet_getname+0xaa/0xe0
[ 223.393482] kernel_getpeername+0x1b/0x20
[ 223.393485] iscsi_sw_tcp_conn_get_param+0xa6/0x110 [iscsi_tcp]
[ 223.393494] show_conn_ep_param_ISCSI_PARAM_CONN_ADDRESS+0x7e/0xa0 [scsi_transport_iscsi]
[ 223.393496] dev_attr_show+0x1d/0x50
[ 223.393499] sysfs_kf_seq_show+0xa1/0x110
[ 223.393502] kernfs_seq_show+0x27/0x30
[ 223.393504] seq_read+0xda/0x420
[ 223.393506] kernfs_fop_read+0x141/0x1a0
[ 223.393510] __vfs_read+0x1b/0x40
[ 223.393512] vfs_read+0x8e/0x130
[ 223.393513] ksys_read+0xa7/0xe0
[ 223.393515] __x64_sys_read+0x1a/0x20
[ 223.393518] do_syscall_64+0x57/0x190
[ 223.393521] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 223.393523] RIP: 0033:0x7f45793ce910
[ 223.393525] Code: b6 fe ff ff 48 8d 3d 0f be 08 00 48 83 ec 08 e8 06 db 01 00 66 0f 1f 44 00 00 83 3d f9 2d 2c 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 de 9b 01 00 48 89 04 24
[ 223.393526] RSP: 002b:00007ffd9fa13688 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 223.393527] RAX: ffffffffffffffda RBX: 00007ffd9fa13820 RCX: 00007f45793ce910
[ 223.393528] RDX: 0000000000000100 RSI: 00007ffd9fa13720 RDI: 0000000000000003
[ 223.393528] RBP: 00007ffd9fa13720 R08: 0000000000000000 R09: 0000000000000000
[ 223.393529] R10: 0000000000000064 R11: 0000000000000246 R12: 0000000000000003
[ 223.393530] R13: 00007ffd9fa13c60 R14: 0000555b0d613708 R15: 0000555b0d613300
[ 223.393581] sd 1:0:0:0: [sdb] Write Protect is off
[ 223.393583] sd 1:0:0:0: [sdb] Mode Sense: 43 00 10 08
[ 223.393660] iscsiadm[18136]: segfault at 7ffd9fa12e58 ip 0000555b0ccd95af sp 00007ffd9fa12e60 error 6 in iscsiadm[555b0ccb4000+58000]
[ 223.393666] Code: ba 00 02 00 00 48 81 ec 10 04 00 00 48 89 e7 48 8d 9c 24 00 02 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 08 04 00 00 31 c0 <e8> 3c ed 00 00 ba 00 02 00 00 4c 89 ee 48 89 e7 e8 6c ed 00 00 ba
[ 223.394992] sd 1:0:0:0: alua: transition timeout set to 60 seconds
[ 223.394997] sd 1:0:0:0: alua: port group 02 state N non-preferred supports TOlUSNA
[ 223.395018] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 223.395435] Kernel panic - not syncing: Aiee, killing interrupt handler!
[ 223.396802] sd 1:0:0:0: [sdb] Optimal transfer size 262144 bytes
[ 223.402387] CPU: 6 PID: 18136 Comm: iscsiadm Kdump: loaded Tainted: G W 5.4.0-1033-gke #35~18.04.1-Ubuntu
[ 223.402388] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 223.402389] Call Trace:
[ 223.402395] dump_stack+0x6d/0x95
[ 223.402398] panic+0xfe/0x2e4
[ 223.402400] do_exit+0x899/0xb90
[ 223.402402] do_group_exit+0x43/0xa0
[ 223.402406] get_signal+0x14f/0x860
[ 223.402409] do_signal+0x34/0x6d0
[ 223.402414] ? __bad_area_nosemaphore+0x149/0x1f0
[ 223.457597] exit_to_usermode_loop+0x8e/0x100
[ 223.462090] prepare_exit_to_usermode+0x91/0xa0
[ 223.466782] retint_user+0x8/0x8
[ 223.470131] RIP: 0033:0x555b0ccd95af
[ 223.473827] Code: ba 00 02 00 00 48 81 ec 10 04 00 00 48 89 e7 48 8d 9c 24 00 02 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 08 04 00 00 31 c0 <e8> 3c ed 00 00 ba 00 02 00 00 4c 89 ee 48 89 e7 e8 6c ed 00 00 ba
[ 223.492991] RSP: 002b:00007ffd9fa12e60 EFLAGS: 00010246
[ 223.498333] RAX: 0000000000000000 RBX: 00007ffd9fa13060 RCX: 00007f45793ce335
[ 223.505579] RDX: 0000000000000200 RSI: 0000555b0cf146a0 RDI: 00007ffd9fa12e60
[ 223.513025] RBP: 00007ffd9fa13ecc R08: 0000000000000000 R09: 0000000080808000
[ 223.520268] R10: 0000000000000075 R11: 0000000000000246 R12: 00007ffd9fa13540
[ 223.527521] R13: 00007ffd9fa13540 R14: 0000000000000200 R15: 0000555b0d613300
Which happens during high iscsi activity
This issue is also identified in linux-5.8, reported here ( https://lkml.org/lkml/2020/7/28/1085 ) for example. It affects the gcp-5.4 kernel specifically because gcp-5.4 has backported '1b66d253610c7 ("bpf: Add get{peer, sock}name attach types for sock_addr")' which introduces the issue.
The fix is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bcf3a2953d36bbfb9bd44ccb3db0897d935cc485 from 5.9 |
[Impact]
Kernel panic during high iscsi activity
This stacktrace
[ 223.386958] BUG: scheduling while atomic: iscsiadm/18136/0x00000200
[ 223.393390] Modules linked in: tcp_diag inet_diag xt_nat ipt_REJECT nf_reject_ipv4 xt_tcpudp ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs sch_htb ebt_ip ebtable_filter ebtables veth xt_mark br_netfilter iptable_mangle xt_MASQUERADE xt_comment xt_addrtype iptable_nat binfmt_misc iptable_filter bpfilter xt_conntrack nf_nat bridge stp llc xfrm_user xfrm_algo aufs overlay nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper input_leds serio_raw sch_fq_codel sunrpc ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi virtio_rng ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq raid1 raid0 multipath linear psmouse virtio_net net_failover failover
[ 223.393429] CPU: 6 PID: 18136 Comm: iscsiadm Kdump: loaded Not tainted 5.4.0-1033-gke #35~18.04.1-Ubuntu
[ 223.393430] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 223.393430] Call Trace:
[ 223.393439] dump_stack+0x6d/0x95
[ 223.393464] __schedule_bug+0x55/0x70
[ 223.393467] __schedule+0x61b/0x710
[ 223.393469] schedule+0x33/0xa0
[ 223.393472] __lock_sock+0x7d/0xc0
[ 223.393475] ? wait_woken+0x80/0x80
[ 223.393477] lock_sock_nested+0x64/0x70
[ 223.393479] inet_getname+0xaa/0xe0
[ 223.393482] kernel_getpeername+0x1b/0x20
[ 223.393485] iscsi_sw_tcp_conn_get_param+0xa6/0x110 [iscsi_tcp]
[ 223.393494] show_conn_ep_param_ISCSI_PARAM_CONN_ADDRESS+0x7e/0xa0 [scsi_transport_iscsi]
[ 223.393496] dev_attr_show+0x1d/0x50
[ 223.393499] sysfs_kf_seq_show+0xa1/0x110
[ 223.393502] kernfs_seq_show+0x27/0x30
[ 223.393504] seq_read+0xda/0x420
[ 223.393506] kernfs_fop_read+0x141/0x1a0
[ 223.393510] __vfs_read+0x1b/0x40
[ 223.393512] vfs_read+0x8e/0x130
[ 223.393513] ksys_read+0xa7/0xe0
[ 223.393515] __x64_sys_read+0x1a/0x20
[ 223.393518] do_syscall_64+0x57/0x190
[ 223.393521] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 223.393523] RIP: 0033:0x7f45793ce910
[ 223.393525] Code: b6 fe ff ff 48 8d 3d 0f be 08 00 48 83 ec 08 e8 06 db 01 00 66 0f 1f 44 00 00 83 3d f9 2d 2c 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 de 9b 01 00 48 89 04 24
[ 223.393526] RSP: 002b:00007ffd9fa13688 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[ 223.393527] RAX: ffffffffffffffda RBX: 00007ffd9fa13820 RCX: 00007f45793ce910
[ 223.393528] RDX: 0000000000000100 RSI: 00007ffd9fa13720 RDI: 0000000000000003
[ 223.393528] RBP: 00007ffd9fa13720 R08: 0000000000000000 R09: 0000000000000000
[ 223.393529] R10: 0000000000000064 R11: 0000000000000246 R12: 0000000000000003
[ 223.393530] R13: 00007ffd9fa13c60 R14: 0000555b0d613708 R15: 0000555b0d613300
[ 223.393581] sd 1:0:0:0: [sdb] Write Protect is off
[ 223.393583] sd 1:0:0:0: [sdb] Mode Sense: 43 00 10 08
[ 223.393660] iscsiadm[18136]: segfault at 7ffd9fa12e58 ip 0000555b0ccd95af sp 00007ffd9fa12e60 error 6 in iscsiadm[555b0ccb4000+58000]
[ 223.393666] Code: ba 00 02 00 00 48 81 ec 10 04 00 00 48 89 e7 48 8d 9c 24 00 02 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 08 04 00 00 31 c0 <e8> 3c ed 00 00 ba 00 02 00 00 4c 89 ee 48 89 e7 e8 6c ed 00 00 ba
[ 223.394992] sd 1:0:0:0: alua: transition timeout set to 60 seconds
[ 223.394997] sd 1:0:0:0: alua: port group 02 state N non-preferred supports TOlUSNA
[ 223.395018] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 223.395435] Kernel panic - not syncing: Aiee, killing interrupt handler!
[ 223.396802] sd 1:0:0:0: [sdb] Optimal transfer size 262144 bytes
[ 223.402387] CPU: 6 PID: 18136 Comm: iscsiadm Kdump: loaded Tainted: G W 5.4.0-1033-gke #35~18.04.1-Ubuntu
[ 223.402388] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
[ 223.402389] Call Trace:
[ 223.402395] dump_stack+0x6d/0x95
[ 223.402398] panic+0xfe/0x2e4
[ 223.402400] do_exit+0x899/0xb90
[ 223.402402] do_group_exit+0x43/0xa0
[ 223.402406] get_signal+0x14f/0x860
[ 223.402409] do_signal+0x34/0x6d0
[ 223.402414] ? __bad_area_nosemaphore+0x149/0x1f0
[ 223.457597] exit_to_usermode_loop+0x8e/0x100
[ 223.462090] prepare_exit_to_usermode+0x91/0xa0
[ 223.466782] retint_user+0x8/0x8
[ 223.470131] RIP: 0033:0x555b0ccd95af
[ 223.473827] Code: ba 00 02 00 00 48 81 ec 10 04 00 00 48 89 e7 48 8d 9c 24 00 02 00 00 64 48 8b 04 25 28 00 00 00 48 89 84 24 08 04 00 00 31 c0 <e8> 3c ed 00 00 ba 00 02 00 00 4c 89 ee 48 89 e7 e8 6c ed 00 00 ba
[ 223.492991] RSP: 002b:00007ffd9fa12e60 EFLAGS: 00010246
[ 223.498333] RAX: 0000000000000000 RBX: 00007ffd9fa13060 RCX: 00007f45793ce335
[ 223.505579] RDX: 0000000000000200 RSI: 0000555b0cf146a0 RDI: 00007ffd9fa12e60
[ 223.513025] RBP: 00007ffd9fa13ecc R08: 0000000000000000 R09: 0000000080808000
[ 223.520268] R10: 0000000000000075 R11: 0000000000000246 R12: 00007ffd9fa13540
[ 223.527521] R13: 00007ffd9fa13540 R14: 0000000000000200 R15: 0000555b0d613300
Which happens during high iscsi activity
This issue is also identified in linux-5.8, reported here ( https://lkml.org/lkml/2020/7/28/1085 ) for example. It affects the gcp-5.4 kernel specifically because gcp-5.4 has backported '1b66d253610c7 ("bpf: Add get{peer, sock}name attach types for sock_addr")' which introduces the issue.
[Fix]
The fix is https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bcf3a2953d36bbfb9bd44ccb3db0897d935cc485 from 5.9
[Test]
Affected customer has reported that they can no longer reproduce the problem with this fix applied. They were readily reproducing the crash without it.
[Regression potential]
I'm not aware of any. The patch seems reasonable. It is accepted in mainline and backported to the stable kernels too. It is present in groovy 5.8 as of https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1898853 |
|
| 2021-04-06 23:08:13 |
Khaled El Mously |
linux-gke (Ubuntu Focal): status |
New |
Fix Committed |
|
| 2021-04-09 10:37:45 |
Ubuntu Kernel Bot |
tags |
|
verification-needed-focal |
|
| 2021-04-13 08:16:30 |
Launchpad Janitor |
linux-gke (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
