5.13 RT kernel hits scheduling while atomic with stress-ng enosys stressor

[Colin Ian King]

stress-ng --enosys 0 -t 60 causes:

[ 513.783315] BUG: scheduling while atomic: stress-ng/42082/0x00000002
[ 513.783319] Modules linked in: cuse snd_seq snd_seq_device dccp_ipv4 dccp atm chacha_generic chacha_x86_64 libchacha xxhash_generic wp512 streebog_generic sm3_generic sha3_generic rmd160 poly1305_generic poly1305_x86_64 nhpoly1305_avx2 nhpoly1305_sse2 nhpoly1305 libpoly1305 michael_mic md4 cmac ccm algif_rng twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common sm4_generic serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic fcrypt des3_ede_x86_64 des_generic libdes cast6_avx_x86_64 cast6_generic cast5_avx_x86_64 cast5_generic cast_common camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 blowfish_generic blowfish_x86_64 blowfish_common algif_skcipher algif_hash aegis128 aegis128_aesni algif_aead af_alg nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr intel_rapl_common kvm_intel joydev kvm input_leds rapl serio_raw snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg
[ 513.783355] snd_intel_sdw_acpi snd_hda_codec snd_hda_core snd_hwdep mac_hid snd_pcm snd_timer snd soundcore qemu_fw_cfg sch_fq_codel msr virtio_rng ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear hid_generic qxl usbhid drm_ttm_helper ttm crct10dif_pclmul hid drm_kms_helper crc32_pclmul syscopyarea sysfillrect ghash_clmulni_intel sysimgblt fb_sys_fops aesni_intel cec crypto_simd cryptd psmouse virtio_net ahci rc_core virtio_blk i2c_i801 net_failover libahci drm lpc_ich i2c_smbus failover
[ 513.783387] CPU: 7 PID: 42082 Comm: stress-ng Tainted: G W 5.13.0-1003-realtime #3-Ubuntu
[ 513.783389] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[ 513.783390] Call Trace:
[ 513.783393] show_stack+0x52/0x58
[ 513.783397] dump_stack+0x7d/0x9c
[ 513.783400] __schedule_bug.cold+0x4a/0x5b
[ 513.783403] __schedule+0x53a/0x6b0
[ 513.783405] ? rt_spin_unlock+0x18/0x50
[ 513.783407] ? task_blocks_on_rt_mutex.constprop.0.isra.0+0x168/0x430
[ 513.783409] schedule_rtlock+0x1f/0x40
[ 513.783411] rtlock_slowlock_locked+0xf7/0x240
[ 513.783412] ? memcg_slab_post_alloc_hook+0x53/0x280
[ 513.783416] rt_spin_lock+0x46/0x70
[ 513.783417] alloc_pid+0x1b4/0x3e0
[ 513.783421] copy_process+0x9ef/0x15e0
[ 513.783423] ? kernel_wait4+0xcf/0x150
[ 513.783425] kernel_clone+0x9d/0x350
[ 513.783427] ? __do_sys_wait4+0x84/0x90
[ 513.783429] ? rt_spin_unlock+0x18/0x50
[ 513.783430] __do_sys_clone+0x5d/0x80
[ 513.783432] __x64_sys_clone+0x25/0x30
[ 513.783434] do_syscall_64+0x61/0xb0
[ 513.783446] ? handle_mm_fault+0xdf/0x2c0
[ 513.783449] ? do_user_addr_fault+0x1eb/0x670
[ 513.783452] ? exit_to_user_mode_prepare+0x37/0xb0
[ 513.783454] ? irqentry_exit_to_user_mode+0x9/0x20
[ 513.783455] ? irqentry_exit+0x33/0x40
[ 513.783457] ? exc_page_fault+0x92/0x1c0
[ 513.783458] ? asm_exc_page_fault+0x8/0x30
[ 513.783460] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 513.783462] RIP: 0033:0x7fce804fcb59
[ 513.783464] Code: ed 0f 85 1a 01 00 00 64 48 8b 04 25 10 00 00 00 45 31 c0 31 d2 31 f6 bf 11 00 20 01 4c 8d 90 d0 02 00 00 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 93 00 00 00 41 89 c5 85 c0 0f 85 a0 00 00
[ 513.783465] RSP: 002b:00007ffd36496060 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
[ 513.783471] RAX: ffffffffffffffda RBX: 00007ffd36496140 RCX: 00007fce804fcb59
[ 513.783472] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
[ 513.783473] RBP: 0000000000000000 R08: 0000000000000000 R09: 000055eebbfb7140
[ 513.783473] R10: 00007fce80019e50 R11: 0000000000000246 R12: 0000000000000000
[ 513.783474] R13: 000000000000a656 R14: 00007ffd364965a0 R15: 00007fce7ffff340

[Colin Ian King (colin-king)]

and stress-ng --dup 0 produces:

[ 506.394685] BUG: scheduling while atomic: stress-ng/35875/0x00000002
[ 506.394819] CPU: 0 PID: 35855 Comm: stress-ng Tainted: G W 5.13.0-1003-realtime #3-Ubuntu
[ 506.394821] i2c_i801 net_failover libahci
[ 506.394822] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[ 506.394822] drm lpc_ich i2c_smbus
[ 506.394823] Call Trace:
[ 506.394824] failover
[ 506.394826] show_stack+0x52/0x58
[ 506.394830] dump_stack+0x7d/0x9c
[ 506.394833] __schedule_bug.cold+0x4a/0x5b
[ 506.394836] __schedule+0x53a/0x6b0
[ 506.394839] ? rt_spin_unlock+0x18/0x50
[ 506.394841] ? task_blocks_on_rt_mutex.constprop.0.isra.0+0x168/0x430
[ 506.394843] schedule_rtlock+0x1f/0x40
[ 506.394848] rtlock_slowlock_locked+0xf7/0x240
[ 506.394849] ? memcg_slab_post_alloc_hook+0x53/0x280
[ 506.394855] rt_spin_lock+0x46/0x70
[ 506.394857] alloc_pid+0x1b4/0x3e0
[ 506.395239] copy_process+0x9ef/0x15e0
[ 506.395256] kernel_clone+0x9d/0x350
[ 506.395266] ? switch_fpu_return+0x4d/0x100
[ 506.395279] __do_sys_clone+0x5d/0x80
[ 506.395295] __x64_sys_clone+0x25/0x30
[ 506.395297] do_syscall_64+0x61/0xb0
[ 506.395310] ? do_syscall_64+0x6e/0xb0
[ 506.395326] entry_SYSCALL_64_after_hwframe+0x44/0xae
  506.395340] RIP: 0033:0x7f17ff98ab59
[ 506.395354] Code: ed 0f 85 1a 01 00 00 64 48 8b 04 25 10 00 00 00 45 31 c0 31 d2 31 f6 bf 11 00 20 01 4c 8d 90 d0 02 00 00 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 93 00 00 00 41 89 c5 85 c0 0f 85 a0 00 00
[ 506.395364] RSP: 002b:00007fffedbceab0 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
[ 506.395366] RAX: ffffffffffffffda RBX: 00007fffedbced40 RCX: 00007f17ff98ab59
[ 506.395367] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
[ 506.395368] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f17fe23c000
[ 506.395387] R10: 00007f17ff4a7e50 R11: 0000000000000246 R12: 0000000000000000
[ 506.395395] R13: 00007f17fe23c000 R14: 0000000000000001 R15: 0000000000100001

[Colin Ian King (colin-king)]

stress-ng --daemon -t 60 produces:

[ 68.860977] BUG: scheduling while atomic: stress-ng/91343/0x00000002
[ 68.860981] Modules linked in: nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr joydev intel_rapl_common kvm_intel kvm rapl input_leds serio_raw snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi mac_hid snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_timer snd qemu_fw_cfg soundcore sch_fq_codel msr virtio_rng ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd qxl drm_ttm_helper ttm psmouse drm_kms_helper i2c_i801 syscopyarea ahci sysfillrect i2c_smbus libahci lpc_ich sysimgblt fb_sys_fops cec rc_core virtio_blk virtio_net drm net_failover failover hid_generic usbhid hid
[ 68.861023] CPU: 6 PID: 91343 Comm: stress-ng Not tainted 5.13.0-1003-realtime #3-Ubuntu
[ 68.861025] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[ 68.861027] Call Trace:
[ 68.861030] show_stack+0x52/0x58
[ 68.861033] dump_stack+0x7d/0x9c
[ 68.861036] __schedule_bug.cold+0x4a/0x5b
[ 68.861040] __schedule+0x53a/0x6b0
[ 68.861043] ? task_blocks_on_rt_mutex.constprop.0.isra.0+0x390/0x430
[ 68.861046] schedule_rtlock+0x1f/0x40
[ 68.861048] rtlock_slowlock_locked+0xf7/0x240
[ 68.861050] ? memcg_slab_post_alloc_hook+0x53/0x280
[ 68.861053] rt_spin_lock+0x46/0x70
[ 68.861055] alloc_pid+0x1b4/0x3e0
[ 68.861058] copy_process+0x9ef/0x15e0
[ 68.861062] kernel_clone+0x9d/0x350
[ 68.861064] ? ksys_write+0xba/0xf0
[ 68.861067] __do_sys_clone+0x5d/0x80
[ 68.861070] __x64_sys_clone+0x25/0x30
[ 68.861072] do_syscall_64+0x61/0xb0
[ 68.861074] ? syscall_exit_to_user_mode+0x27/0x50
[ 68.861076] ? __x64_sys_capget+0x16/0x20
[ 68.861078] ? do_syscall_64+0x6e/0xb0
[ 68.861080] ? asm_exc_page_fault+0x8/0x30
[ 68.861083] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 68.861085] RIP: 0033:0x7f9ba1822b59
[ 68.861087] Code: ed 0f 85 1a 01 00 00 64 48 8b 04 25 10 00 00 00 45 31 c0 31 d2 31 f6 bf 11 00 20 01 4c 8d 90 d0 02 00 00 b8 38 00 00 00 0f 05 <48> 3d 00 f0 ff ff 0f 87 93 00 00 00 41 89 c5 85 c0 0f 85 a0 00 00
[ 68.861089] RSP: 002b:00007ffc6f1aa8d0 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
[ 68.861091] RAX: ffffffffffffffda RBX: 00007ffc6f1aab10 RCX: 00007f9ba1822b59
[ 68.861092] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
[ 68.861093] RBP: 0000000000000000 R08: 0000000000000000 R09: 00007f9ba133fb80
[ 68.861094] R10: 00007f9ba133fe50 R11: 0000000000000246 R12: 0000000000000000
[ 68.861095] R13: 0000000000000001 R14: 0000000000000002 R15: 0000000000000064
[ 68.861118] show_signal_msg: 20 callbacks suppressed
[ 68.861122] stress-ng[91343]: segfault at 7ffc6f1aa8c8 ip 00007f9ba1822c20 sp 00007ffc6f1aa8d0 error 7 in libc-2.33.so[7f9ba176b000+16b000]
[ 68.861128] Code: 00 00 48 8b 15 51 22 10 00 f7 d8 41 bd ff ff ff ff 64 89 02 66 0f 1f 44 00 00 85 ed 0f 85 b0 00 00 00 44 8...

[Colin Ian King (colin-king)]

stress-ng --af-alg 0 -t 60 (a few rounds) produces:

[ 99.445636] BUG: scheduling while atomic: kthreadd/2/0x00000002
[ 99.445647] Modules linked in: chacha_generic chacha_x86_64 libchacha xxhash_generic wp512 streebog_generic sm3_generic sha3_generic rmd160 poly1305_generic poly1305_x86_64 nhpoly1305_avx2 nhpoly1305_sse2 nhpoly1305 libpoly1305 michael_mic md4 cmac ccm algif_rng twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common sm4_generic serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic fcrypt des3_ede_x86_64 des_generic libdes cast6_avx_x86_64 cast6_generic cast5_avx_x86_64 cast5_generic cast_common camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 blowfish_generic blowfish_x86_64 blowfish_common algif_skcipher algif_hash aegis128 aegis128_aesni algif_aead af_alg nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr joydev intel_rapl_common kvm_intel kvm rapl input_leds serio_raw snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg mac_hid snd_intel_sdw_acpi snd_hda_codec
[ 99.445686] snd_hda_core snd_hwdep snd_pcm snd_timer snd qemu_fw_cfg soundcore sch_fq_codel msr virtio_rng ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd qxl drm_ttm_helper ttm psmouse i2c_i801 drm_kms_helper ahci lpc_ich i2c_smbus libahci syscopyarea sysfillrect sysimgblt virtio_net fb_sys_fops net_failover virtio_blk failover cec rc_core drm hid_generic usbhid hid
[ 99.445721] CPU: 6 PID: 2 Comm: kthreadd Not tainted 5.13.0-1003-realtime #3-Ubuntu
[ 99.445723] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 0.0.0 02/06/2015
[ 99.445726] Call Trace:
[ 99.445728] show_stack+0x52/0x58
[ 99.445745] dump_stack+0x7d/0x9c
[ 99.445750] __schedule_bug.cold+0x4a/0x5b
[ 99.445755] __schedule+0x53a/0x6b0
[ 99.445761] ? task_blocks_on_rt_mutex.constprop.0.isra.0+0x390/0x430
[ 99.445765] schedule_rtlock+0x1f/0x40
[ 99.445766] rtlock_slowlock_locked+0xf7/0x240
[ 99.445768] rt_spin_lock+0x46/0x70
[ 99.445769] alloc_pid+0x1b4/0x3e0
[ 99.445779] copy_process+0x9ef/0x15e0
[ 99.445783] ? asm_sysvec_reschedule_ipi+0x12/0x20
[ 99.445786] kernel_clone+0x9d/0x350
[ 99.445788] kernel_thread+0x55/0x70
[ 99.445790] ? __kthread_parkme+0xa0/0xa0
[ 99.445794] kthreadd+0x29b/0x2f0
[ 99.445795] ? kthread_is_per_cpu+0x30/0x30
[ 99.445797] ret_from_fork+0x22/0x30

[Colin Ian King (colin-king)]

stock 5.13 + same config + RT patches work fine, so this is something todo with the sauce patches methinks.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1938918

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.