| 2022-07-16 09:07:44 |
Henry Goffin |
bug |
|
|
added bug |
| 2022-07-18 22:52:02 |
Henry Goffin |
affects |
linux-meta-aws-5.15 (Ubuntu) |
linux-aws-5.15 (Ubuntu) |
|
| 2022-09-15 13:52:32 |
Stefan Bader |
affects |
linux-aws-5.15 (Ubuntu) |
linux (Ubuntu) |
|
| 2022-09-15 13:52:32 |
Stefan Bader |
linux (Ubuntu): status |
New |
Confirmed |
|
| 2022-09-15 13:52:42 |
Stefan Bader |
nominated for series |
|
Ubuntu Jammy |
|
| 2022-09-15 13:52:42 |
Stefan Bader |
bug task added |
|
linux (Ubuntu Jammy) |
|
| 2022-09-15 13:53:12 |
Stefan Bader |
linux (Ubuntu Jammy): importance |
Undecided |
High |
|
| 2022-09-15 13:53:12 |
Stefan Bader |
linux (Ubuntu Jammy): status |
New |
Confirmed |
|
| 2022-09-15 13:53:20 |
Stefan Bader |
linux (Ubuntu): status |
Confirmed |
Invalid |
|
| 2022-09-15 14:57:35 |
Stefan Bader |
description |
The kernel 5.15 amdgpu module crashes on load with a “BUG: kernel NULL pointer dereference” on Amazon EC2 G4ad hardware (custom AMD Radeon V520 Pro datacenter GPU) on focal (HWE) and jammy with kernel 5.15.0-1011 and possibly earlier, up through latest (revision 1015.19). This crash bug did not exist in any of the focal HWE 5.13 kernels.
This is probably an upstream kernel bug, but I am also filing it here because existing focal users on EC2 will suddenly stop having access to their AMD GPUs after a reboot once the new 5.15 HWE kernel is installed.
The full backtrace from dmesg is below. The offending function call which crashes in the 5.15 kernel corresponds to this source (sorry, not the right source tree, but the same driver) https://github.com/torvalds/linux/blob/8bb7eca972ad531c9b149c0a51ab43a417385813/drivers/gpu/drm/amd/amdgpu/amdgpu_fb.c#L345
A workaround that I have discovered is adding “options amdgpu virtual_display=;” to a new modprobe.d configuration file - something which shouldn’t be required, but is at least harmless.
Here is the relevant BUG message and backtrace from dmesg:
[ 318.111721] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 318.115443] #PF: supervisor instruction fetch in kernel mode
[ 318.118443] #PF: error_code(0x0010) - not-present page
[ 318.121177] PGD 0 P4D 0
[ 318.122688] Oops: 0010 [#1] SMP NOPTI
[ 318.124592] CPU: 6 PID: 13667 Comm: modprobe Tainted: G W 5.15.0-1015-aws #19~20.04.1-Ubuntu
[ 318.129711] Hardware name: Amazon EC2 g4ad.2xlarge/, BIOS 1.0 10/16/2017
[ 318.133167] RIP: 0010:0x0
[ 318.134704] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
[ 318.138291] RSP: 0018:ffff9841828d78e0 EFLAGS: 00010246
[ 318.140938] RAX: 0000000000000000 RBX: ffff8a4f16ae8000 RCX: 0000000000000001
[ 318.144604] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8a4f16ae8000
[ 318.148319] RBP: ffff9841828d7908 R08: ffff8a4f02460278 R09: ffff8a4f06422c40
[ 318.152151] R10: c01c42494f8affff R11: ffff8a4f01dcb5b8 R12: ffff8a4f024602e8
[ 318.155929] R13: ffffffffc107e4a0 R14: 0000000000000000 R15: ffff8a4f02460010
[ 318.159685] FS: 00007f8afdc1c740(0000) GS:ffff8a55f0980000(0000) knlGS:0000000000000000
[ 318.163897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 318.167038] CR2: ffffffffffffffd6 CR3: 000000011817e000 CR4: 00000000003506e0
[ 318.170758] Call Trace:
[ 318.173964] <TASK>
[ 318.176822] __drm_helper_disable_unused_functions+0xe7/0x100 [drm_kms_helper]
[ 318.184230] drm_helper_disable_unused_functions+0x44/0x50 [drm_kms_helper]
[ 318.189761] amdgpu_fbdev_init+0x104/0x110 [amdgpu]
[ 318.194264] amdgpu_device_init.cold+0x7cc/0xc48 [amdgpu]
[ 318.199061] ? pci_read_config_byte+0x27/0x40
[ 318.203206] amdgpu_driver_load_kms+0x1e/0x270 [amdgpu]
[ 318.207901] amdgpu_pci_probe+0x1ea/0x290 [amdgpu]
[ 318.212445] local_pci_probe+0x4b/0x90
[ 318.216386] pci_device_probe+0x182/0x1f0
[ 318.220407] really_probe.part.0+0xcb/0x370
[ 318.224460] really_probe+0x40/0x80
[ 318.228232] __driver_probe_device+0x115/0x190
[ 318.232412] driver_probe_device+0x23/0xa0
[ 318.236436] __driver_attach+0xbd/0x160
[ 318.240348] ? __device_attach_driver+0x110/0x110
[ 318.244637] bus_for_each_dev+0x7e/0xc0
[ 318.248570] driver_attach+0x1e/0x20
[ 318.252474] bus_add_driver+0x161/0x200
[ 318.256412] driver_register+0x74/0xd0
[ 318.260332] __pci_register_driver+0x68/0x70
[ 318.264496] amdgpu_init+0x7c/0x1000 [amdgpu]
[ 318.268841] ? 0xffffffffc146e000
[ 318.272521] do_one_initcall+0x48/0x1d0
[ 318.276446] ? __cond_resched+0x19/0x30
[ 318.280378] ? kmem_cache_alloc_trace+0x15a/0x420
[ 318.284736] do_init_module+0x52/0x230
[ 318.288644] load_module+0x1372/0x1600
[ 318.292529] __do_sys_finit_module+0xbf/0x120
[ 318.296706] ? __do_sys_finit_module+0xbf/0x120
[ 318.300947] __x64_sys_finit_module+0x1a/0x20
[ 318.305265] do_syscall_64+0x5c/0xc0
[ 318.308984] ? do_syscall_64+0x69/0xc0
[ 318.312841] ? do_syscall_64+0x69/0xc0
[ 318.316693] ? do_syscall_64+0x69/0xc0
[ 318.320545] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 318.324975] RIP: 0033:0x7f8afdd6273d
[ 318.328700] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 23 37 0d 00 f7 d8 64 89 01 48
[ 318.343601] RSP: 002b:00007ffe48e5c8c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 318.351197] RAX: ffffffffffffffda RBX: 0000560c79e39260 RCX: 00007f8afdd6273d
[ 318.356678] RDX: 0000000000000000 RSI: 0000560c78e96358 RDI: 000000000000000f
[ 318.362340] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000
[ 318.367743] R10: 000000000000000f R11: 0000000000000246 R12: 0000560c78e96358
[ 318.373241] R13: 0000000000000000 R14: 0000560c79e39390 R15: 0000560c79e39260
[ 318.378699] </TASK>
[ 318.381783] Modules linked in: amdgpu(+) iommu_v2 gpu_sched drm_ttm_helper ttm drm_kms_helper cec rc_core i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt btrfs blake2b_generic xor zstd_compress raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ppdev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel psmouse crypto_simd parport_pc input_leds parport cryptd ena serio_raw sch_fq_codel ipmi_devintf ipmi_msghandler msr drm ip_tables x_tables autofs4
[ 318.418449] CR2: 0000000000000000
[ 318.422130] ---[ end trace d6b9efffe55f5322 ]---
[ 318.426391] RIP: 0010:0x0
[ 318.429681] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
[ 318.434994] RSP: 0018:ffff9841828d78e0 EFLAGS: 00010246
[ 318.439489] RAX: 0000000000000000 RBX: ffff8a4f16ae8000 RCX: 0000000000000001
[ 318.444896] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8a4f16ae8000
[ 318.450518] RBP: ffff9841828d7908 R08: ffff8a4f02460278 R09: ffff8a4f06422c40
[ 318.455937] R10: c01c42494f8affff R11: ffff8a4f01dcb5b8 R12: ffff8a4f024602e8
[ 318.461581] R13: ffffffffc107e4a0 R14: 0000000000000000 R15: ffff8a4f02460010
[ 318.466999] FS: 00007f8afdc1c740(0000) GS:ffff8a55f0980000(0000) knlGS:0000000000000000
[ 318.474744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 318.479525] CR2: ffffffffffffffd6 CR3: 000000011817e000 CR4: 00000000003506e0 |
[SRU Justification]
Impact: The 5.15 kernel series introduced a regression compared to 5.13 related to virtual display code on AMD GPUs. This causes crashes for cloud/VM users back to Focal (using rolling cloud or HWE kernels).
Fix: Upstream stable 5.15.61 contains a fix for this which is only needed there since later kernel version have that code rewritten again. This change would get picked up normally in the next stable cycle after 2022.09.19. But given the impact we would apply the following patch from 5.15.61 ahead of time:
commit 27f8f5219fe4658537ba28fd01657e1062ac3960 linux-5.15.y
"drm/amdgpu: fix check in fbdev init"
Test case: Booting the affected kernel on a VM which passes through AMD GPU hardware.
Regression potential: Worst case some driver functions will not get disabled where they did before.
--- original description ---
The kernel 5.15 amdgpu module crashes on load with a “BUG: kernel NULL pointer dereference” on Amazon EC2 G4ad hardware (custom AMD Radeon V520 Pro datacenter GPU) on focal (HWE) and jammy with kernel 5.15.0-1011 and possibly earlier, up through latest (revision 1015.19). This crash bug did not exist in any of the focal HWE 5.13 kernels.
This is probably an upstream kernel bug, but I am also filing it here because existing focal users on EC2 will suddenly stop having access to their AMD GPUs after a reboot once the new 5.15 HWE kernel is installed.
The full backtrace from dmesg is below. The offending function call which crashes in the 5.15 kernel corresponds to this source (sorry, not the right source tree, but the same driver) https://github.com/torvalds/linux/blob/8bb7eca972ad531c9b149c0a51ab43a417385813/drivers/gpu/drm/amd/amdgpu/amdgpu_fb.c#L345
A workaround that I have discovered is adding “options amdgpu virtual_display=;” to a new modprobe.d configuration file - something which shouldn’t be required, but is at least harmless.
Here is the relevant BUG message and backtrace from dmesg:
[ 318.111721] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 318.115443] #PF: supervisor instruction fetch in kernel mode
[ 318.118443] #PF: error_code(0x0010) - not-present page
[ 318.121177] PGD 0 P4D 0
[ 318.122688] Oops: 0010 [#1] SMP NOPTI
[ 318.124592] CPU: 6 PID: 13667 Comm: modprobe Tainted: G W 5.15.0-1015-aws #19~20.04.1-Ubuntu
[ 318.129711] Hardware name: Amazon EC2 g4ad.2xlarge/, BIOS 1.0 10/16/2017
[ 318.133167] RIP: 0010:0x0
[ 318.134704] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
[ 318.138291] RSP: 0018:ffff9841828d78e0 EFLAGS: 00010246
[ 318.140938] RAX: 0000000000000000 RBX: ffff8a4f16ae8000 RCX: 0000000000000001
[ 318.144604] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8a4f16ae8000
[ 318.148319] RBP: ffff9841828d7908 R08: ffff8a4f02460278 R09: ffff8a4f06422c40
[ 318.152151] R10: c01c42494f8affff R11: ffff8a4f01dcb5b8 R12: ffff8a4f024602e8
[ 318.155929] R13: ffffffffc107e4a0 R14: 0000000000000000 R15: ffff8a4f02460010
[ 318.159685] FS: 00007f8afdc1c740(0000) GS:ffff8a55f0980000(0000) knlGS:0000000000000000
[ 318.163897] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 318.167038] CR2: ffffffffffffffd6 CR3: 000000011817e000 CR4: 00000000003506e0
[ 318.170758] Call Trace:
[ 318.173964] <TASK>
[ 318.176822] __drm_helper_disable_unused_functions+0xe7/0x100 [drm_kms_helper]
[ 318.184230] drm_helper_disable_unused_functions+0x44/0x50 [drm_kms_helper]
[ 318.189761] amdgpu_fbdev_init+0x104/0x110 [amdgpu]
[ 318.194264] amdgpu_device_init.cold+0x7cc/0xc48 [amdgpu]
[ 318.199061] ? pci_read_config_byte+0x27/0x40
[ 318.203206] amdgpu_driver_load_kms+0x1e/0x270 [amdgpu]
[ 318.207901] amdgpu_pci_probe+0x1ea/0x290 [amdgpu]
[ 318.212445] local_pci_probe+0x4b/0x90
[ 318.216386] pci_device_probe+0x182/0x1f0
[ 318.220407] really_probe.part.0+0xcb/0x370
[ 318.224460] really_probe+0x40/0x80
[ 318.228232] __driver_probe_device+0x115/0x190
[ 318.232412] driver_probe_device+0x23/0xa0
[ 318.236436] __driver_attach+0xbd/0x160
[ 318.240348] ? __device_attach_driver+0x110/0x110
[ 318.244637] bus_for_each_dev+0x7e/0xc0
[ 318.248570] driver_attach+0x1e/0x20
[ 318.252474] bus_add_driver+0x161/0x200
[ 318.256412] driver_register+0x74/0xd0
[ 318.260332] __pci_register_driver+0x68/0x70
[ 318.264496] amdgpu_init+0x7c/0x1000 [amdgpu]
[ 318.268841] ? 0xffffffffc146e000
[ 318.272521] do_one_initcall+0x48/0x1d0
[ 318.276446] ? __cond_resched+0x19/0x30
[ 318.280378] ? kmem_cache_alloc_trace+0x15a/0x420
[ 318.284736] do_init_module+0x52/0x230
[ 318.288644] load_module+0x1372/0x1600
[ 318.292529] __do_sys_finit_module+0xbf/0x120
[ 318.296706] ? __do_sys_finit_module+0xbf/0x120
[ 318.300947] __x64_sys_finit_module+0x1a/0x20
[ 318.305265] do_syscall_64+0x5c/0xc0
[ 318.308984] ? do_syscall_64+0x69/0xc0
[ 318.312841] ? do_syscall_64+0x69/0xc0
[ 318.316693] ? do_syscall_64+0x69/0xc0
[ 318.320545] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 318.324975] RIP: 0033:0x7f8afdd6273d
[ 318.328700] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 23 37 0d 00 f7 d8 64 89 01 48
[ 318.343601] RSP: 002b:00007ffe48e5c8c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 318.351197] RAX: ffffffffffffffda RBX: 0000560c79e39260 RCX: 00007f8afdd6273d
[ 318.356678] RDX: 0000000000000000 RSI: 0000560c78e96358 RDI: 000000000000000f
[ 318.362340] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000000000000000
[ 318.367743] R10: 000000000000000f R11: 0000000000000246 R12: 0000560c78e96358
[ 318.373241] R13: 0000000000000000 R14: 0000560c79e39390 R15: 0000560c79e39260
[ 318.378699] </TASK>
[ 318.381783] Modules linked in: amdgpu(+) iommu_v2 gpu_sched drm_ttm_helper ttm drm_kms_helper cec rc_core i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt btrfs blake2b_generic xor zstd_compress raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c nls_iso8859_1 dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua ppdev crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel psmouse crypto_simd parport_pc input_leds parport cryptd ena serio_raw sch_fq_codel ipmi_devintf ipmi_msghandler msr drm ip_tables x_tables autofs4
[ 318.418449] CR2: 0000000000000000
[ 318.422130] ---[ end trace d6b9efffe55f5322 ]---
[ 318.426391] RIP: 0010:0x0
[ 318.429681] Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
[ 318.434994] RSP: 0018:ffff9841828d78e0 EFLAGS: 00010246
[ 318.439489] RAX: 0000000000000000 RBX: ffff8a4f16ae8000 RCX: 0000000000000001
[ 318.444896] RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8a4f16ae8000
[ 318.450518] RBP: ffff9841828d7908 R08: ffff8a4f02460278 R09: ffff8a4f06422c40
[ 318.455937] R10: c01c42494f8affff R11: ffff8a4f01dcb5b8 R12: ffff8a4f024602e8
[ 318.461581] R13: ffffffffc107e4a0 R14: 0000000000000000 R15: ffff8a4f02460010
[ 318.466999] FS: 00007f8afdc1c740(0000) GS:ffff8a55f0980000(0000) knlGS:0000000000000000
[ 318.474744] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 318.479525] CR2: ffffffffffffffd6 CR3: 000000011817e000 CR4: 00000000003506e0 |
|
| 2022-09-15 14:59:21 |
Stefan Bader |
linux (Ubuntu Jammy): status |
Confirmed |
Fix Committed |
|
| 2022-10-06 16:20:26 |
Henry Goffin |
tags |
|
verification-done-jammy |
|
| 2022-10-10 08:44:51 |
Launchpad Janitor |
linux (Ubuntu Jammy): status |
Fix Committed |
Fix Released |
|
| 2022-10-10 08:44:51 |
Launchpad Janitor |
cve linked |
|
2022-3176 |
|
| 2022-12-07 15:49:48 |
Ubuntu Kernel Bot |
tags |
verification-done-jammy |
kernel-spammed-jammy-linux-bluefield verification-needed-jammy |
|
| 2022-12-12 12:47:00 |
Ubuntu Kernel Bot |
tags |
kernel-spammed-jammy-linux-bluefield verification-needed-jammy |
kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia verification-needed-jammy |
|
| 2024-03-01 06:16:57 |
Ubuntu Kernel Bot |
tags |
kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-nvidia verification-needed-jammy |
kernel-spammed-jammy-linux-bluefield kernel-spammed-jammy-linux-mtk-v2 kernel-spammed-jammy-linux-nvidia verification-needed-jammy verification-needed-jammy-linux-mtk |
|
