Activity log for bug #1946368

Date Who What changed Old value New value Message
2021-10-07 13:20:14 Dave Jones bug added bug
2021-10-07 13:20:20 Dave Jones nominated for series Ubuntu Impish
2021-10-07 13:20:20 Dave Jones bug task added linux-raspi (Ubuntu Impish)
2021-10-07 13:41:05 Launchpad Janitor linux-raspi (Ubuntu): status New Confirmed
2021-10-07 13:41:14 fossfreedom bug added subscriber fossfreedom
2021-10-09 09:26:34 Juerg Haefliger bug added subscriber Juerg Haefliger
2021-10-09 10:45:15 Sam Lane bug added subscriber Sam Lane
2021-10-09 14:33:50 Xinle Yao bug added subscriber Xinle Yao
2021-10-09 15:00:46 fprietog bug added subscriber fprietog
2021-10-11 21:33:15 Dave Jones linux-raspi (Ubuntu Impish): importance Undecided Critical
2021-10-14 13:00:17 Ubuntu QA Website tags iso-testing
2021-10-17 05:45:11 hujq bug added subscriber hujq
2021-10-17 15:00:19 Benjamin Lupton bug added subscriber Benjamin Lupton
2021-10-17 15:39:33 fossfreedom attachment added fossfreedom edid monitor https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/1946368/+attachment/5533679/+files/edid
2021-10-18 10:03:53 andrum99 bug added subscriber andrum99
2021-10-18 14:55:12 Juerg Haefliger description Under the current (5.13.0-1007.8) or proposed (5.13.0-1008.9) kernels for the Ubuntu Pi pre-installed desktop impish release, the HDMI output occasionally freezes. A known workaround at this time is to change the following line in /boot/firmware/config.txt: dtoverlay=vc4-kms-v3d To the following: dtoverlay=vc4-fkms-v3d In other words, to use the "fake" KMS overlay (fkms) instead of the "full" KMS overlay (kms). I've been unable to determine a reliable method of guaranteeing a freeze, but it appears to occur much more readily when video playback is occurring, and when other interactions (especially moving windows around, minimizing, restoring) occurs simultaneously. Display suspend also periodically causes the same hang, which made me suspect this might be related to #1944397 but it appears that had a separate cause (now resolved). The following dmesg outputs have been observed immediately after the display hang; this one from 1007.8: [ 513.762138] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 513.762288] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] co mmit wait timed out [ 513.762381] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_ done timed out [ 524.002211] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 524.002404] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:205:plane-25 ] commit wait timed out [ 534.242499] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 534.242657] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 534.250685] ------------[ cut here ]------------ [ 534.250701] refcount_t: underflow; use-after-free. [ 534.250735] WARNING: CPU: 1 PID: 120 at lib/refcount.c:87 refcount_dec_not_one+0xa0/0xbc [ 534.250758] Modules linked in: rfcomm cmac algif_hash algif_skcipher af_alg hci_uart btqca btrtl btbcm btintel bnep snd_soc_hdmi_codec vc4 btsdio snd_soc_core input_leds bluetooth snd_compress snd_bcm2835(C) snd_pcm_dmaengine ecdh_generic ecc snd_pcm brcmfmac snd_seq_midi snd_seq_midi_event bcm2835_codec(C) bcm 2835_isp(C) bcm2835_v4l2(C) brcmutil snd_rawmidi v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc cfg80211 videobuf2_memops videobuf2_v4l2 snd_seq videobuf2_common videodev snd_seq_devi ce mc snd_timer vc_sm_cma(C) raspberrypi_hwmon snd bcm2835_gpiomem rpivid_mem uio_pdrv_genirq uio sch_fq_ codel ip_tables x_tables autofs4 btrfs blake2b_generic xor xor_neon zstd_compress hid_generic usbhid raid 6_pq libcrc32c dm_mirror dm_region_hash dm_log spidev dwc2 v3d roles udc_core gpu_sched crct10dif_ce i2c_ brcmstb i2c_bcm2835 spi_bcm2835 drm_kms_helper syscopyarea xhci_pci xhci_pci_renesas sysfillrect sysimgbl t fb_sys_fops cec drm phy_generic ac97_bus aes_arm64 [ 534.251066] CPU: 1 PID: 120 Comm: kworker/1:2 Tainted: G WC 5.13.0-1007-raspi #8-Ubuntu [ 534.251076] Hardware name: Raspberry Pi 400 Rev 1.1 (DT) [ 534.251083] Workqueue: events drm_mode_rmfb_work_fn [drm] [ 534.251239] pstate: 60400005 (nZCv daif +PAN -UAO -TCO BTYPE=--) [ 534.251248] pc : refcount_dec_not_one+0xa0/0xbc [ 534.251257] lr : refcount_dec_not_one+0xa0/0xbc [ 534.251265] sp : ffff8000118cbb50 [ 534.251269] x29: ffff8000118cbb50 x28: ffff6cf7ee894400 x27: ffff6cf80502e000 [ 534.251285] x26: ffff6cf80502e000 x25: 0000000000000006 x24: ffff6cf7ee94c500 [ 534.251300] x23: ffffa94dff246018 x22: ffff6cf833068880 x21: ffff6cf805027c80 [ 534.251314] x20: ffff6cf88ead75ac x19: ffff6cf88ead7400 x18: 0000000000000000 [ 534.251328] x17: 0000000000000000 x16: ffffa94e0a243314 x15: 0000000000000000 [ 534.251342] x14: 0000000000000000 x13: 0000000000000030 x12: ffff800010035000 [ 534.251356] x11: ffffa94e0b30dfd0 x10: 00000000fffff000 x9 : ffffa94e09d09f54 [ 534.251370] x8 : 00000000ffffefff x7 : ffffa94e0b30dfd0 x6 : 0000000000000000 [ 534.251384] x5 : ffff6cf8b799f948 x4 : 0000000000000000 x3 : 0000000000000027 [ 534.251397] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff6cf800abec80 [ 534.251411] Call trace: [ 534.251416] refcount_dec_not_one+0xa0/0xbc [ 534.251424] vc4_bo_dec_usecnt+0x2c/0x120 [vc4] [ 534.251473] vc4_cleanup_fb+0x3c/0x4c [vc4] [ 534.251518] drm_atomic_helper_cleanup_planes+0x74/0xa0 [drm_kms_helper] [ 534.251604] vc4_atomic_commit_tail+0x24c/0x36c [vc4] [ 534.251648] commit_tail+0xac/0x190 [drm_kms_helper] [ 534.251723] drm_atomic_helper_commit+0x168/0x380 [drm_kms_helper] [ 534.251796] drm_atomic_commit+0x58/0x70 [drm] [ 534.251936] atomic_remove_fb+0x2a8/0x2f4 [drm] [ 534.252073] drm_framebuffer_remove+0x164/0x18c [drm] [ 534.252209] drm_mode_rmfb_work_fn+0x50/0x70 [drm] [ 534.252346] process_one_work+0x200/0x4d0 [ 534.252359] worker_thread+0x2c8/0x470 [ 534.252367] kthread+0x12c/0x140 [ 534.252374] ret_from_fork+0x10/0x3c [ 534.252386] ---[ end trace a97341262fc57e44 ]--- And a similar one from 1008.9 (note that most of the time, the stack trace *doesn't* appear hence I'm not sure if it's related to the display freeze itself, or something auxiliary): [ 221.914617] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 221.914617] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 221.914795] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 232.154711] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 232.154898] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit I can produce this same stack trace, but *without* a corresponding freeze by manually locking the desktop (Super+L) and waiting for the display fade. However, after the stack trace appears, one can press a key to bring the display back and login again happily. Here are several repeated traces from such activity under the 1008.9 proposed kernel: [ 1043.431061] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1043.431136] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1043.431384] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1053.671415] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1053.671705] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1063.911800] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1063.912147] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 1181.162739] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1181.418774] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1181.419072] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1191.658996] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1191.659289] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1201.899168] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1201.899438] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 1332.461450] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1332.461460] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1332.461717] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1342.701602] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1342.701890] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1352.941798] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1352.942067] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit Note that even when the display is frozen (and cannot be resurrected), only the display driver appears to crash; the system remains operational (at least for some time) happily responding to SSH login requests or, if video is playing, continuing audio output. The same occurs with the current linux-firmware-raspi2 release (which the above stack traces were taken from), or with the latest firmware (installed from an experimental package in my https://launchpad.net/~waveform/+archive/ubuntu/firmware PPA). [Impact] The UI of Impish desktop for raspi occasionally freezes which can be triggered by playing videos and moving windows around. Seems to happen more frequently when using higher resolution and/or higher refresh rates (higher than 1920x1080@60Hz). Kernel errors are along the lines of: [ 513.762138] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 513.762288] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] co mmit wait timed out [ 513.762381] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_ done timed out [Test Case] Install current Impish desktop for raspi on a Pi 4. Use a high-refresh/resolution monitor. Play videos, move windows around. [Fix] The Pi foundation identified a couple of upstream vc4 commits that seem to interfere with their downstream patches. Revert those to match the Pi foundations currently released 5.10 kernel. [Where Problems Could Occur] The changes are confined to the vc4 driver, so problems would only show up if that driver is in use. Which is only the case for the Ubuntu raspi desktop image, so sever images should not be impacted at all. [Original Description] Under the current (5.13.0-1007.8) or proposed (5.13.0-1008.9) kernels for the Ubuntu Pi pre-installed desktop impish release, the HDMI output occasionally freezes. A known workaround at this time is to change the following line in /boot/firmware/config.txt: dtoverlay=vc4-kms-v3d To the following: dtoverlay=vc4-fkms-v3d In other words, to use the "fake" KMS overlay (fkms) instead of the "full" KMS overlay (kms). I've been unable to determine a reliable method of guaranteeing a freeze, but it appears to occur much more readily when video playback is occurring, and when other interactions (especially moving windows around, minimizing, restoring) occurs simultaneously. Display suspend also periodically causes the same hang, which made me suspect this might be related to #1944397 but it appears that had a separate cause (now resolved). The following dmesg outputs have been observed immediately after the display hang; this one from 1007.8: [ 513.762138] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 513.762288] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] co mmit wait timed out [ 513.762381] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_ done timed out [ 524.002211] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 524.002404] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:205:plane-25 ] commit wait timed out [ 534.242499] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 534.242657] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 534.250685] ------------[ cut here ]------------ [ 534.250701] refcount_t: underflow; use-after-free. [ 534.250735] WARNING: CPU: 1 PID: 120 at lib/refcount.c:87 refcount_dec_not_one+0xa0/0xbc [ 534.250758] Modules linked in: rfcomm cmac algif_hash algif_skcipher af_alg hci_uart btqca btrtl btbcm  btintel bnep snd_soc_hdmi_codec vc4 btsdio snd_soc_core input_leds bluetooth snd_compress snd_bcm2835(C)  snd_pcm_dmaengine ecdh_generic ecc snd_pcm brcmfmac snd_seq_midi snd_seq_midi_event bcm2835_codec(C) bcm 2835_isp(C) bcm2835_v4l2(C) brcmutil snd_rawmidi v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc cfg80211 videobuf2_memops videobuf2_v4l2 snd_seq videobuf2_common videodev snd_seq_devi ce mc snd_timer vc_sm_cma(C) raspberrypi_hwmon snd bcm2835_gpiomem rpivid_mem uio_pdrv_genirq uio sch_fq_ codel ip_tables x_tables autofs4 btrfs blake2b_generic xor xor_neon zstd_compress hid_generic usbhid raid 6_pq libcrc32c dm_mirror dm_region_hash dm_log spidev dwc2 v3d roles udc_core gpu_sched crct10dif_ce i2c_ brcmstb i2c_bcm2835 spi_bcm2835 drm_kms_helper syscopyarea xhci_pci xhci_pci_renesas sysfillrect sysimgbl t fb_sys_fops cec drm phy_generic ac97_bus aes_arm64 [ 534.251066] CPU: 1 PID: 120 Comm: kworker/1:2 Tainted: G WC 5.13.0-1007-raspi #8-Ubuntu [ 534.251076] Hardware name: Raspberry Pi 400 Rev 1.1 (DT) [ 534.251083] Workqueue: events drm_mode_rmfb_work_fn [drm] [ 534.251239] pstate: 60400005 (nZCv daif +PAN -UAO -TCO BTYPE=--) [ 534.251248] pc : refcount_dec_not_one+0xa0/0xbc [ 534.251257] lr : refcount_dec_not_one+0xa0/0xbc [ 534.251265] sp : ffff8000118cbb50 [ 534.251269] x29: ffff8000118cbb50 x28: ffff6cf7ee894400 x27: ffff6cf80502e000 [ 534.251285] x26: ffff6cf80502e000 x25: 0000000000000006 x24: ffff6cf7ee94c500 [ 534.251300] x23: ffffa94dff246018 x22: ffff6cf833068880 x21: ffff6cf805027c80 [ 534.251314] x20: ffff6cf88ead75ac x19: ffff6cf88ead7400 x18: 0000000000000000 [ 534.251328] x17: 0000000000000000 x16: ffffa94e0a243314 x15: 0000000000000000 [ 534.251342] x14: 0000000000000000 x13: 0000000000000030 x12: ffff800010035000 [ 534.251356] x11: ffffa94e0b30dfd0 x10: 00000000fffff000 x9 : ffffa94e09d09f54 [ 534.251370] x8 : 00000000ffffefff x7 : ffffa94e0b30dfd0 x6 : 0000000000000000 [ 534.251384] x5 : ffff6cf8b799f948 x4 : 0000000000000000 x3 : 0000000000000027 [ 534.251397] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff6cf800abec80 [ 534.251411] Call trace: [ 534.251416] refcount_dec_not_one+0xa0/0xbc [ 534.251424] vc4_bo_dec_usecnt+0x2c/0x120 [vc4] [ 534.251473] vc4_cleanup_fb+0x3c/0x4c [vc4] [ 534.251518] drm_atomic_helper_cleanup_planes+0x74/0xa0 [drm_kms_helper] [ 534.251604] vc4_atomic_commit_tail+0x24c/0x36c [vc4] [ 534.251648] commit_tail+0xac/0x190 [drm_kms_helper] [ 534.251723] drm_atomic_helper_commit+0x168/0x380 [drm_kms_helper] [ 534.251796] drm_atomic_commit+0x58/0x70 [drm] [ 534.251936] atomic_remove_fb+0x2a8/0x2f4 [drm] [ 534.252073] drm_framebuffer_remove+0x164/0x18c [drm] [ 534.252209] drm_mode_rmfb_work_fn+0x50/0x70 [drm] [ 534.252346] process_one_work+0x200/0x4d0 [ 534.252359] worker_thread+0x2c8/0x470 [ 534.252367] kthread+0x12c/0x140 [ 534.252374] ret_from_fork+0x10/0x3c [ 534.252386] ---[ end trace a97341262fc57e44 ]--- And a similar one from 1008.9 (note that most of the time, the stack trace *doesn't* appear hence I'm not sure if it's related to the display freeze itself, or something auxiliary): [ 221.914617] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 221.914617] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 221.914795] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 232.154711] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 232.154898] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit I can produce this same stack trace, but *without* a corresponding freeze by manually locking the desktop (Super+L) and waiting for the display fade. However, after the stack trace appears, one can press a key to bring the display back and login again happily. Here are several repeated traces from such activity under the 1008.9 proposed kernel: [ 1043.431061] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1043.431136] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1043.431384] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1053.671415] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1053.671705] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1063.911800] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1063.912147] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 1181.162739] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1181.418774] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1181.419072] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1191.658996] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1191.659289] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1201.899168] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1201.899438] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 1332.461450] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1332.461460] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1332.461717] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1342.701602] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1342.701890] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1352.941798] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1352.942067] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit Note that even when the display is frozen (and cannot be resurrected), only the display driver appears to crash; the system remains operational (at least for some time) happily responding to SSH login requests or, if video is playing, continuing audio output. The same occurs with the current linux-firmware-raspi2 release (which the above stack traces were taken from), or with the latest firmware (installed from an experimental package in my https://launchpad.net/~waveform/+archive/ubuntu/firmware PPA).
2021-10-18 15:27:09 Juerg Haefliger description [Impact] The UI of Impish desktop for raspi occasionally freezes which can be triggered by playing videos and moving windows around. Seems to happen more frequently when using higher resolution and/or higher refresh rates (higher than 1920x1080@60Hz). Kernel errors are along the lines of: [ 513.762138] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 513.762288] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] co mmit wait timed out [ 513.762381] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_ done timed out [Test Case] Install current Impish desktop for raspi on a Pi 4. Use a high-refresh/resolution monitor. Play videos, move windows around. [Fix] The Pi foundation identified a couple of upstream vc4 commits that seem to interfere with their downstream patches. Revert those to match the Pi foundations currently released 5.10 kernel. [Where Problems Could Occur] The changes are confined to the vc4 driver, so problems would only show up if that driver is in use. Which is only the case for the Ubuntu raspi desktop image, so sever images should not be impacted at all. [Original Description] Under the current (5.13.0-1007.8) or proposed (5.13.0-1008.9) kernels for the Ubuntu Pi pre-installed desktop impish release, the HDMI output occasionally freezes. A known workaround at this time is to change the following line in /boot/firmware/config.txt: dtoverlay=vc4-kms-v3d To the following: dtoverlay=vc4-fkms-v3d In other words, to use the "fake" KMS overlay (fkms) instead of the "full" KMS overlay (kms). I've been unable to determine a reliable method of guaranteeing a freeze, but it appears to occur much more readily when video playback is occurring, and when other interactions (especially moving windows around, minimizing, restoring) occurs simultaneously. Display suspend also periodically causes the same hang, which made me suspect this might be related to #1944397 but it appears that had a separate cause (now resolved). The following dmesg outputs have been observed immediately after the display hang; this one from 1007.8: [ 513.762138] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 513.762288] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] co mmit wait timed out [ 513.762381] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_ done timed out [ 524.002211] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 524.002404] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:205:plane-25 ] commit wait timed out [ 534.242499] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 534.242657] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 534.250685] ------------[ cut here ]------------ [ 534.250701] refcount_t: underflow; use-after-free. [ 534.250735] WARNING: CPU: 1 PID: 120 at lib/refcount.c:87 refcount_dec_not_one+0xa0/0xbc [ 534.250758] Modules linked in: rfcomm cmac algif_hash algif_skcipher af_alg hci_uart btqca btrtl btbcm  btintel bnep snd_soc_hdmi_codec vc4 btsdio snd_soc_core input_leds bluetooth snd_compress snd_bcm2835(C)  snd_pcm_dmaengine ecdh_generic ecc snd_pcm brcmfmac snd_seq_midi snd_seq_midi_event bcm2835_codec(C) bcm 2835_isp(C) bcm2835_v4l2(C) brcmutil snd_rawmidi v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc cfg80211 videobuf2_memops videobuf2_v4l2 snd_seq videobuf2_common videodev snd_seq_devi ce mc snd_timer vc_sm_cma(C) raspberrypi_hwmon snd bcm2835_gpiomem rpivid_mem uio_pdrv_genirq uio sch_fq_ codel ip_tables x_tables autofs4 btrfs blake2b_generic xor xor_neon zstd_compress hid_generic usbhid raid 6_pq libcrc32c dm_mirror dm_region_hash dm_log spidev dwc2 v3d roles udc_core gpu_sched crct10dif_ce i2c_ brcmstb i2c_bcm2835 spi_bcm2835 drm_kms_helper syscopyarea xhci_pci xhci_pci_renesas sysfillrect sysimgbl t fb_sys_fops cec drm phy_generic ac97_bus aes_arm64 [ 534.251066] CPU: 1 PID: 120 Comm: kworker/1:2 Tainted: G WC 5.13.0-1007-raspi #8-Ubuntu [ 534.251076] Hardware name: Raspberry Pi 400 Rev 1.1 (DT) [ 534.251083] Workqueue: events drm_mode_rmfb_work_fn [drm] [ 534.251239] pstate: 60400005 (nZCv daif +PAN -UAO -TCO BTYPE=--) [ 534.251248] pc : refcount_dec_not_one+0xa0/0xbc [ 534.251257] lr : refcount_dec_not_one+0xa0/0xbc [ 534.251265] sp : ffff8000118cbb50 [ 534.251269] x29: ffff8000118cbb50 x28: ffff6cf7ee894400 x27: ffff6cf80502e000 [ 534.251285] x26: ffff6cf80502e000 x25: 0000000000000006 x24: ffff6cf7ee94c500 [ 534.251300] x23: ffffa94dff246018 x22: ffff6cf833068880 x21: ffff6cf805027c80 [ 534.251314] x20: ffff6cf88ead75ac x19: ffff6cf88ead7400 x18: 0000000000000000 [ 534.251328] x17: 0000000000000000 x16: ffffa94e0a243314 x15: 0000000000000000 [ 534.251342] x14: 0000000000000000 x13: 0000000000000030 x12: ffff800010035000 [ 534.251356] x11: ffffa94e0b30dfd0 x10: 00000000fffff000 x9 : ffffa94e09d09f54 [ 534.251370] x8 : 00000000ffffefff x7 : ffffa94e0b30dfd0 x6 : 0000000000000000 [ 534.251384] x5 : ffff6cf8b799f948 x4 : 0000000000000000 x3 : 0000000000000027 [ 534.251397] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff6cf800abec80 [ 534.251411] Call trace: [ 534.251416] refcount_dec_not_one+0xa0/0xbc [ 534.251424] vc4_bo_dec_usecnt+0x2c/0x120 [vc4] [ 534.251473] vc4_cleanup_fb+0x3c/0x4c [vc4] [ 534.251518] drm_atomic_helper_cleanup_planes+0x74/0xa0 [drm_kms_helper] [ 534.251604] vc4_atomic_commit_tail+0x24c/0x36c [vc4] [ 534.251648] commit_tail+0xac/0x190 [drm_kms_helper] [ 534.251723] drm_atomic_helper_commit+0x168/0x380 [drm_kms_helper] [ 534.251796] drm_atomic_commit+0x58/0x70 [drm] [ 534.251936] atomic_remove_fb+0x2a8/0x2f4 [drm] [ 534.252073] drm_framebuffer_remove+0x164/0x18c [drm] [ 534.252209] drm_mode_rmfb_work_fn+0x50/0x70 [drm] [ 534.252346] process_one_work+0x200/0x4d0 [ 534.252359] worker_thread+0x2c8/0x470 [ 534.252367] kthread+0x12c/0x140 [ 534.252374] ret_from_fork+0x10/0x3c [ 534.252386] ---[ end trace a97341262fc57e44 ]--- And a similar one from 1008.9 (note that most of the time, the stack trace *doesn't* appear hence I'm not sure if it's related to the display freeze itself, or something auxiliary): [ 221.914617] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 221.914617] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 221.914795] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 232.154711] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 232.154898] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit I can produce this same stack trace, but *without* a corresponding freeze by manually locking the desktop (Super+L) and waiting for the display fade. However, after the stack trace appears, one can press a key to bring the display back and login again happily. Here are several repeated traces from such activity under the 1008.9 proposed kernel: [ 1043.431061] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1043.431136] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1043.431384] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1053.671415] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1053.671705] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1063.911800] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1063.912147] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 1181.162739] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1181.418774] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1181.419072] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1191.658996] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1191.659289] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1201.899168] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1201.899438] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 1332.461450] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1332.461460] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1332.461717] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1342.701602] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1342.701890] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1352.941798] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1352.942067] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit Note that even when the display is frozen (and cannot be resurrected), only the display driver appears to crash; the system remains operational (at least for some time) happily responding to SSH login requests or, if video is playing, continuing audio output. The same occurs with the current linux-firmware-raspi2 release (which the above stack traces were taken from), or with the latest firmware (installed from an experimental package in my https://launchpad.net/~waveform/+archive/ubuntu/firmware PPA). [Impact] The UI of Impish desktop for raspi occasionally freezes which can be triggered by playing videos and moving windows around. Seems to happen more frequently when using higher resolutions and/or higher refresh rates (higher than 1920x1080@60Hz). Kernel errors are along the lines of: [ 513.762138] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 513.762288] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] co mmit wait timed out [ 513.762381] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_ done timed out [Test Case] Install current Impish desktop for raspi on a Pi 4. Use a high-refresh/resolution monitor. Play videos, move windows around. [Fix] The Pi foundation identified a couple of upstream vc4 commits that seem to interfere with their downstream patches. Revert those to match the Pi foundations currently released 5.10 kernel. [Where Problems Could Occur] The changes are confined to the vc4 driver, so problems would only show up if that driver is in use. Which is only the case for the Ubuntu raspi desktop image, so server images should not be impacted at all. [Original Description] Under the current (5.13.0-1007.8) or proposed (5.13.0-1008.9) kernels for the Ubuntu Pi pre-installed desktop impish release, the HDMI output occasionally freezes. A known workaround at this time is to change the following line in /boot/firmware/config.txt: dtoverlay=vc4-kms-v3d To the following: dtoverlay=vc4-fkms-v3d In other words, to use the "fake" KMS overlay (fkms) instead of the "full" KMS overlay (kms). I've been unable to determine a reliable method of guaranteeing a freeze, but it appears to occur much more readily when video playback is occurring, and when other interactions (especially moving windows around, minimizing, restoring) occurs simultaneously. Display suspend also periodically causes the same hang, which made me suspect this might be related to #1944397 but it appears that had a separate cause (now resolved). The following dmesg outputs have been observed immediately after the display hang; this one from 1007.8: [ 513.762138] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 513.762288] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] co mmit wait timed out [ 513.762381] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_ done timed out [ 524.002211] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 524.002404] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:205:plane-25 ] commit wait timed out [ 534.242499] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 534.242657] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 534.250685] ------------[ cut here ]------------ [ 534.250701] refcount_t: underflow; use-after-free. [ 534.250735] WARNING: CPU: 1 PID: 120 at lib/refcount.c:87 refcount_dec_not_one+0xa0/0xbc [ 534.250758] Modules linked in: rfcomm cmac algif_hash algif_skcipher af_alg hci_uart btqca btrtl btbcm  btintel bnep snd_soc_hdmi_codec vc4 btsdio snd_soc_core input_leds bluetooth snd_compress snd_bcm2835(C)  snd_pcm_dmaengine ecdh_generic ecc snd_pcm brcmfmac snd_seq_midi snd_seq_midi_event bcm2835_codec(C) bcm 2835_isp(C) bcm2835_v4l2(C) brcmutil snd_rawmidi v4l2_mem2mem bcm2835_mmal_vchiq(C) videobuf2_dma_contig videobuf2_vmalloc cfg80211 videobuf2_memops videobuf2_v4l2 snd_seq videobuf2_common videodev snd_seq_devi ce mc snd_timer vc_sm_cma(C) raspberrypi_hwmon snd bcm2835_gpiomem rpivid_mem uio_pdrv_genirq uio sch_fq_ codel ip_tables x_tables autofs4 btrfs blake2b_generic xor xor_neon zstd_compress hid_generic usbhid raid 6_pq libcrc32c dm_mirror dm_region_hash dm_log spidev dwc2 v3d roles udc_core gpu_sched crct10dif_ce i2c_ brcmstb i2c_bcm2835 spi_bcm2835 drm_kms_helper syscopyarea xhci_pci xhci_pci_renesas sysfillrect sysimgbl t fb_sys_fops cec drm phy_generic ac97_bus aes_arm64 [ 534.251066] CPU: 1 PID: 120 Comm: kworker/1:2 Tainted: G WC 5.13.0-1007-raspi #8-Ubuntu [ 534.251076] Hardware name: Raspberry Pi 400 Rev 1.1 (DT) [ 534.251083] Workqueue: events drm_mode_rmfb_work_fn [drm] [ 534.251239] pstate: 60400005 (nZCv daif +PAN -UAO -TCO BTYPE=--) [ 534.251248] pc : refcount_dec_not_one+0xa0/0xbc [ 534.251257] lr : refcount_dec_not_one+0xa0/0xbc [ 534.251265] sp : ffff8000118cbb50 [ 534.251269] x29: ffff8000118cbb50 x28: ffff6cf7ee894400 x27: ffff6cf80502e000 [ 534.251285] x26: ffff6cf80502e000 x25: 0000000000000006 x24: ffff6cf7ee94c500 [ 534.251300] x23: ffffa94dff246018 x22: ffff6cf833068880 x21: ffff6cf805027c80 [ 534.251314] x20: ffff6cf88ead75ac x19: ffff6cf88ead7400 x18: 0000000000000000 [ 534.251328] x17: 0000000000000000 x16: ffffa94e0a243314 x15: 0000000000000000 [ 534.251342] x14: 0000000000000000 x13: 0000000000000030 x12: ffff800010035000 [ 534.251356] x11: ffffa94e0b30dfd0 x10: 00000000fffff000 x9 : ffffa94e09d09f54 [ 534.251370] x8 : 00000000ffffefff x7 : ffffa94e0b30dfd0 x6 : 0000000000000000 [ 534.251384] x5 : ffff6cf8b799f948 x4 : 0000000000000000 x3 : 0000000000000027 [ 534.251397] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff6cf800abec80 [ 534.251411] Call trace: [ 534.251416] refcount_dec_not_one+0xa0/0xbc [ 534.251424] vc4_bo_dec_usecnt+0x2c/0x120 [vc4] [ 534.251473] vc4_cleanup_fb+0x3c/0x4c [vc4] [ 534.251518] drm_atomic_helper_cleanup_planes+0x74/0xa0 [drm_kms_helper] [ 534.251604] vc4_atomic_commit_tail+0x24c/0x36c [vc4] [ 534.251648] commit_tail+0xac/0x190 [drm_kms_helper] [ 534.251723] drm_atomic_helper_commit+0x168/0x380 [drm_kms_helper] [ 534.251796] drm_atomic_commit+0x58/0x70 [drm] [ 534.251936] atomic_remove_fb+0x2a8/0x2f4 [drm] [ 534.252073] drm_framebuffer_remove+0x164/0x18c [drm] [ 534.252209] drm_mode_rmfb_work_fn+0x50/0x70 [drm] [ 534.252346] process_one_work+0x200/0x4d0 [ 534.252359] worker_thread+0x2c8/0x470 [ 534.252367] kthread+0x12c/0x140 [ 534.252374] ret_from_fork+0x10/0x3c [ 534.252386] ---[ end trace a97341262fc57e44 ]--- And a similar one from 1008.9 (note that most of the time, the stack trace *doesn't* appear hence I'm not sure if it's related to the display freeze itself, or something auxiliary): [ 221.914617] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 221.914617] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 221.914795] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 232.154711] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 232.154898] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit I can produce this same stack trace, but *without* a corresponding freeze by manually locking the desktop (Super+L) and waiting for the display fade. However, after the stack trace appears, one can press a key to bring the display back and login again happily. Here are several repeated traces from such activity under the 1008.9 proposed kernel: [ 1043.431061] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1043.431136] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1043.431384] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1053.671415] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1053.671705] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1063.911800] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1063.912147] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 1181.162739] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1181.418774] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1181.419072] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1191.658996] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1191.659289] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1201.899168] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1201.899438] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit [ 1332.461450] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out [ 1332.461460] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1332.461717] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] commit wait timed out [ 1342.701602] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1342.701890] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:70:plane-3] commit wait timed out [ 1352.941798] [drm:drm_crtc_commit_wait [drm]] *ERROR* flip_done timed out [ 1352.942067] vc4-drm gpu: [drm] *ERROR* Timed out waiting for commit Note that even when the display is frozen (and cannot be resurrected), only the display driver appears to crash; the system remains operational (at least for some time) happily responding to SSH login requests or, if video is playing, continuing audio output. The same occurs with the current linux-firmware-raspi2 release (which the above stack traces were taken from), or with the latest firmware (installed from an experimental package in my https://launchpad.net/~waveform/+archive/ubuntu/firmware PPA).
2021-10-18 16:00:29 Juerg Haefliger linux-raspi (Ubuntu Impish): status Confirmed In Progress
2021-10-19 07:58:37 Juerg Haefliger linux-raspi (Ubuntu Impish): status In Progress Fix Committed
2021-10-20 23:04:04 Dennis Golden bug added subscriber Dennis Golden
2021-10-23 07:58:26 Joe Aztec bug added subscriber Joe Aztec
2021-10-23 23:29:14 Tim Halloran bug added subscriber Tim Halloran
2021-10-26 04:18:18 Costin Botescu bug added subscriber Costin Botescu
2021-10-26 10:00:23 Andreas Zeitler bug added subscriber Andreas Zeitler
2021-10-26 20:24:01 wendellgee bug added subscriber wendellgee
2021-10-27 18:43:53 Ganesh Chandar Devulapalli linux-raspi (Ubuntu Impish): status Fix Committed Fix Released
2021-10-27 18:56:38 fossfreedom linux-raspi (Ubuntu Impish): status Fix Released Fix Committed
2021-10-27 20:30:06 sam bug added subscriber sam
2021-10-29 11:06:13 Ubuntu Kernel Bot tags iso-testing iso-testing verification-needed-impish
2021-10-29 12:07:26 fprietog tags iso-testing verification-needed-impish iso-testing verification-done-impish
2021-11-01 22:02:56 Ak bug added subscriber Андрей
2021-11-02 21:05:01 Dennis Golden attachment added config.text for test https://bugs.launchpad.net/ubuntu/+source/linux-raspi/+bug/1946368/+attachment/5537761/+files/config.txt_5_13_1010_11
2021-11-06 23:33:35 Lloyd E James bug added subscriber Lloyd E James
2021-11-17 12:21:26 Launchpad Janitor linux-raspi (Ubuntu Impish): status Fix Committed Fix Released
2021-11-17 12:21:26 Launchpad Janitor cve linked 2021-3759
2022-02-02 18:39:14 Launchpad Janitor linux-raspi (Ubuntu): status Confirmed Fix Released
2022-02-02 18:39:14 Launchpad Janitor cve linked 2021-4155
2022-02-02 18:39:14 Launchpad Janitor cve linked 2022-0185