Comment 0 for bug 1908423

Since upgrading to 5.4.0-56-generic my RPI3 devices started freezing randomly, about once every few hours. Diagnostics so far have not turned up anything useful, still busy figuring out cross-compile bisect and a reasonably fast reproduction.

The devices are also known as (/proc/device-tree/compatible)
raspberrypi,3-model-bbrcm,bcm2837
raspberrypi,3-model-b-plusbrcm,bcm2837
I only use them in ARM64 mode, and I only run -generic kernels via Pete Batards EDKII builds.

(unreliable, slow) reproduction:
1. run affected aarch64 kernel
2. set low non-zero /proc/sys/kernel/hung_task_timeout_secs
3. utilize xorg through vc4 kms
4. check dmesg when display freezes

symptom 1 (always the same -62):
13:52:01.832230 kernel: raspberrypi-firmware soc:firmware: mbox_send_message returned -62
13:52:01.839310 kernel: raspberrypi-clk raspberrypi-clk: Failed to change pllb frequency: -62

Note: similar message in Bug 1889637

symptom 2 (varying hangs):
[Dez15 16:09] INFO: task kworker/2:1:36 blocked for more than 120 seconds.
[ +0,000019] Tainted: G C E 5.4.0-58-generic #64-Ubuntu
[ +0,000004] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ +0,000006] kworker/2:1 D 0 36 2 0x00000028
[ +0,000022] Workqueue: events dbs_work_handler
[ +0,000004] Call trace:
[ +0,000009] __switch_to+0xf8/0x1b0
[ +0,000008] __schedule+0x310/0x7d8
[ +0,000005] schedule+0x40/0xb8
[ +0,000004] schedule_timeout+0xa0/0x198
[ +0,000005] __wait_for_common+0xf0/0x230
[ +0,000005] wait_for_completion_timeout+0x38/0x48
[ +0,000006] mbox_send_message+0xd0/0x170
[ +0,000006] rpi_firmware_property_list+0xec/0x250
[ +0,000004] rpi_firmware_property+0x78/0xb8
[ +0,000010] raspberrypi_fw_pll_set_rate+0x60/0xe0 [clk_raspberrypi]
[ +0,000008] clk_change_rate+0xdc/0x420
[ +0,000004] clk_core_set_rate_nolock+0x1cc/0x1f0
[ +0,000003] clk_set_rate+0x3c/0xc0
[ +0,000006] dev_pm_opp_set_rate+0x3d4/0x520
[ +0,000004] set_target+0x4c/0x90
[ +0,000007] __cpufreq_driver_target+0x2c8/0x678
[ +0,000004] od_dbs_update+0x144/0x1a0
[ +0,000003] dbs_work_handler+0x48/0x80
[ +0,000006] process_one_work+0x1d0/0x468
[ +0,000005] worker_thread+0x154/0x4e0
[ +0,000004] kthread+0xf0/0x118
[ +0,000004] ret_from_fork+0x10/0x18

non-kernel-version explanations ruled out:
network adapter is neither idle nor saturated (~32MiB / hour on enxb827eb..)
4 different power supply models, 2 of them bad but not correlated with frequency of hang
swap size & usage obviously does change system responsiveness, but not correlated with frequency of hang
hang reported with & without use of wifi of device or any other device within 1m
hang reported with & without config.txt dtoverlay=disable-wifi (NOT confirmed that even works)
hang reported with cpu_thermal-virtual-0 between 59-67°C (3 copper plates, no fan)
hang reported with & without kernel cmdline memtest=2
hang reported with & without kernel cmdline dwc_otg.speed=1
hang reported with firmware-1.20200601 & firmware-1.20201201
hang reported with /proc/device-tree/model "Raspberry Pi 3 Model B Rev 1.2" & "Raspberry Pi 3 Model B Plus Rev 1.3"
hang reported with & without config.txt enable_uart=1,uart_2ndstage=1
hang reported with config.txt boot_delay=2 bootcode_delay=2
hang reported with EFI "rpi3 tianocore 1.27" & "rpi3 tianocore 1.31"

hang reported with 5.4.0-56-generic usually after a few hours of uptime
hang reported with 5.4.0-58-generic usually after a few hours of uptime
hang reported with (hwe-edge) 5.8.0-31-generic usually after a few hours of uptime
hang reported with 5.4.0-59-generic usually after a few hours of uptime
never seen in versions 5.4.0-26-generic through 5.4.0-53-generic, with one notable exception while investigating:
similiar hang reported with 5.4.0-53-generic if and only if attempting to unload i2c_bcm2835 module

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: linux-image-5.4.0-58-generic 5.4.0-58.64
ProcVersionSignature: Ubuntu 5.4.0-58.64-generic 5.4.73
Uname: Linux 5.4.0-58-generic aarch64
AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.4.0-58-generic.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu27.14
Architecture: arm64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/controlC1', '/dev/snd/pcmC1D1p', '/dev/snd/pcmC1D0p', '/dev/snd/by-path', '/dev/snd/seq', '/dev/snd/controlC0', '/dev/snd/pcmC0D0p', '/dev/snd/timer'] failed with exit code 1:
Card0.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card0.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
Card1.Amixer.info: Error: [Errno 2] No such file or directory: 'amixer'
Card1.Amixer.values: Error: [Errno 2] No such file or directory: 'amixer'
CasperMD5CheckResult: skip
Date: Tue Dec 15 16:46:43 2020
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lspci:

Lspci-vt: -[0000:00]-
Lsusb:
 Bus 001 Device 004: ID 046d:c52b Logitech, Inc. Unifying Receiver
 Bus 001 Device 003: ID 0424:ec00 Microchip Technology, Inc. (formerly SMSC) SMSC9512/9514 Fast Ethernet Adapter
 Bus 001 Device 002: ID 0424:9514 Microchip Technology, Inc. (formerly SMSC) SMC9514 Hub
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: Raspberry Pi Foundation Raspberry Pi 3 Model B
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
ProcFB: 0 vc4drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.4.0-58-generic root=UUID=e3bc3b23-fd02-4250-8350-5916df6d6fd1 ro consoleblank=0 usbcore.authorized_default=0 splash fsck.repair=yes cma=128M ip=dhcp printk.devkmsg=on vt.handoff=7
RelatedPackageVersions:
 linux-restricted-modules-5.4.0-58-generic N/A
 linux-backports-modules-5.4.0-58-generic N/A
 linux-firmware 1.187.6
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux
StagingDrivers: snd_bcm2835 vchiq
UpgradeStatus: No upgrade log present (probably fresh install)
acpidump:

dmi.bios.date: 11/13/2020
dmi.bios.vendor: https://github.com/pftf/RPi3
dmi.bios.version: UEFI Firmware v1.31
dmi.board.name: Raspberry Pi 3 Model B
dmi.board.vendor: Sony UK
dmi.board.version: A02082
dmi.chassis.type: 34
dmi.chassis.vendor: Sony UK
dmi.chassis.version: Raspberry Pi 3 Model B
dmi.modalias: dmi:bvnhttps//github.com/pftf/RPi3:bvrUEFIFirmwarev1.31:bd11/13/2020:svnRaspberryPiFoundation:pnRaspberryPi3ModelB:pvrA02082:rvnSonyUK:rnRaspberryPi3ModelB:rvrA02082:cvnSonyUK:ct34:cvrRaspberryPi3ModelB:
dmi.product.family: Raspberry Pi
dmi.product.name: Raspberry Pi 3 Model B
dmi.product.sku: 0000000000A02082
dmi.product.version: A02082
dmi.sys.vendor: Raspberry Pi Foundation
modified.conffile..etc.default.apport: enabled=0
mtime.conffile..etc.default.apport: 2020-07-06T13:01:13.327228