Blank screen after boot in raspberry pi 4

[Andre Ruiz]

I just flashed noble arm64+raspi image and booted it in my raspberry pi 4, and after initial rainbow screen kernel boots with 4 raspberries on the top and boot text appears as normal. But then, before it finishes booting, the screen goes blank.

Image is ubuntu-24.04-preinstalled-server-arm64+raspi.img.xz downloaded at Apr/30.

ubuntu@ubuntu:~$ uname -a
Linux ubuntu 6.8.0-1004-raspi #4-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 02:29:55 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

I know the system booted because I can access it over the network and I even plugged a usb-serial adapter in the uart GPIOs and from there I can see the boot does progress to completion and offers me a prompt where I can use the system normally. But the HDMI screen stays blank.

The funny thing is that I have been trying daily builds for quite some time now and they used to work well, this is the first one that does this (the official release).

After testing some config options I found that if I change "dtoverlay=vc4-kms-v3d" for "dtoverlay=vc4-fkms-v3d" in config.txt then it boots ok and the screen stays on (note the change from kms to fkms).

I also tried appending ",cma-128" to that line (it is present on other lines -- like for pi3 and pi0 -- but not on this one), it does not help.

I don't think the issue is with my monitor since it works well for a long time. Also I checked EDIDs, they are detected when using fkms but they are blank when using kms:

With KMS:
ubuntu@ubuntu:~$ edid-decode /sys/class/drm/card0-HDMI-A-1/edid
EDID of '/sys/class/drm/card0-HDMI-A-1/edid' was empty.

With fKMS:
ubuntu@ubuntu:~$ edid-decode /sys/class/drm/card0-HDMI-A-1/edid
edid-decode (hex):

00 ff ff ff ff ff ff 00 49 f7 00 00 00 00 00 00
01 1a 01 03 80 00 00 00 0a d7 a5 a2 59 4a 96 24
......
(supressed)

I do not use a graphical environment on this unit so I don't know the consequences of this change (to me it just fixes the issue, but I guess it was there for a reason).

Let me know if you need any other information from the environment.

[Andre Ruiz (andre-ruiz)]

I just found a usb flash key with my last installation (from a daily image) from a few days ago, and it does not have a line for that dtoverlay at all (neither fkms nor kms).

This line seems to have appeared on the official release.

[Dave Jones (waveform)]

You are correct it was introduced for noble as, without the KMS driver, there's no audio output over HDMI on the server image under the Pi 5 (LP: #2038924). KMS is also the only supported stack on the Pi 5 upstream, and rather than trying to support different stacks on different models I opted to move everything to the KMS stack instead (which we were already using on the desktop images).

Unfortunately, it seems that as well as causing issues on the older models (LP: #2063255, LP: #2060300 -- incidentally the cma-128 workaround is for this latter bug), it's now even causing issues on the Pi 4. Obviously I didn't see these issues during testing, but then I've only two different models of monitor here that I test with: a BenQ and a Samsung.

From the description it sounds like the kernel is either failing to retrieve, or failing to parse the EDID from your monitor. However, from reading a potentially related upstream bug (https://github.com/raspberrypi/linux/issues/5368#issuecomment-1453144837) it may be that the monitor is failing to assert hotplug. I would first try 6by9's work-arounds from that thread, specifically:

* Forcing hotplug by appending "vc4.force_hotplug=N (where N is 0x01 for the first HDMI port, or 0x02 for the second) to cmdline.txt on the boot partition:

$ sudo sed -i -e 's/$/ vc4.force_hotplug=0x01' /boot/firmware/cmdline.txt
$ sudo reboot

* If that fails, reboot with the fkms overlay in place, dump the EDID to /lib/firmware/edid.bin, then force that to be parsed under kms:

$ sudo sed -i -e 's/vc4-kms-v3d/vc4-fkms-v3d/' /boot/firmware/config.txt
$ sudo reboot
... wait for the reboot ...
$ sudo cp /sys/class/drm/card1-HDMI-A-1/edid /lib/firmware/edid.bin
$ sudo sed -i -e 's/vc4-fkms-v3d/vc4-kms-v3d/' /boot/firmware/config.txt
$ sudo sed -i -e 's/$/ drm.edid_firmware=HDMI-A-1:edid.bin' /boot/firmware/cmdline.txt
$ sudo reboot
... see if the monitor works ...

If you could attach the edid dump from when running under FKMS that may also help the kernel team in their investigation.

[Andre Ruiz (andre-ruiz)]

Ok, running some tests.

First thing is that with fKMS overlay, right on boot (even before the kernel started) I can see these messages on the serial console:

RPi: BOOTLOADER release VERSION:0b7b6f28 DATE: 2024/04/17 TIME: 13:51:36
BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1713358296 serial a0940f46 boardrev b03111 stc
(lots of lines removed)
Boot mode: USB-MSD (04) order e
HDMI0 edid block 0 offset 0
00ffffffffffff0049f7000000000000
011a0103800000000ad7a5a2594a9624
145054afce0081c00101010101010101
0101010101017f2156aa51001e30468f
33003f432100001ea91a00a050001630
3020370020560000001a000000fd0032
4c1e5010000a202020202020000000fc
004141410a20202020202020200a016f
HDMI0 edid block 1 offset 128
020323744f109f140504131216111503
0702060123090f038301000066030c00
100080011d00bc52d01e20b8285540c4
8e2100001e011d80d0721c1620102c25
80c48e2100009e8c0ad08a20e02d1010
3e9600138e210000188c0ad090204031
200c405500138e210000180000000000
0000000000000000000000000000009d
HDMI0: best-mode 1 (limit 1) 1280x720 60 Hz CEA modes fe007f80000000000000000000000000 extensions 1
USB2[1] 400202e1 connected

If I switch to the KMS overlay, the message is exactly the same, so it means the EDID is accessible from the monitor in either case. I compared both outputs and they are the same.

But with KMS, as expected, the monitor went blank again by the end of the boot messages.

Another interesting thing is that I *could* dump edid even with the KMS overlay right after boot, once. When I tried again, it did not work anymore, saying it was empty:

ubuntu@ubuntu:~$ edid-decode /sys/class/drm/card0-HDMI-A-1/edid
edid-decode (hex):

00 ff ff ff ff ff ff 00 49 f7 00 00 00 00 00 00
01 1a 01 03 80 00 00 00 0a d7 a5 a2 59 4a 96 24
14 50 54 af ce 00 81 c0 01 01 01 01 01 01 01 01
01 01 01 01 01 01 7f 21 56 aa 51 00 1e 30 46 8f
33 00 3f 43 21 00 00 1e a9 1a 00 a0 50 00 16 30
30 20 37 00 20 56 00 00 00 1a 00 00 00 fd 00 32
4c 1e 50 10 00 0a 20 20 20 20 20 20 00 00 00 fc
00 41 41 41 0a 20 20 20 20 20 20 20 20 0a 01 6f

02 03 23 74 4f 10 9f 14 05 04 13 12 16 11 15 03
07 02 06 01 23 09 0f 03 83 01 00 00 66 03 0c 00
10 00 80 01 1d 00 bc 52 d0 1e 20 b8 28 55 40 c4
8e 21 00 00 1e 01 1d 80 d0 72 1c 16 20 10 2c 25
80 c4 8e 21 00 00 9e 8c 0a d0 8a 20 e0 2d 10 10
3e 96 00 13 8e 21 00 00 18 8c 0a d0 90 20 40 31
20 0c 40 55 00 13 8e 21 00 00 18 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 9d

----------------

Block 0, Base EDID:
  EDID Structure Version & Revision: 1.3
  Vendor & Product Identification:
    Manufacturer: ROW
    Model: 0
    Made in: week 1 of 2016
  Basic Display Parameters & Features:
    Digital display
    Image size is variable
    Gamma: 1.00
    RGB color display
    First detailed timing is the preferred timing
  Color Characteristics:
    Red : 0.6357, 0.3486
    Green: 0.2900, 0.5888
    Blue : 0.1425, 0.0800
    White: 0.3134, 0.3291
  Established Timings I & II:
    IBM : 720x400 70.081663 Hz 9:5 31.467 kHz 28.320000 MHz
    DMT 0x04: 640x480 59.940476 Hz 4:3 31.469 kHz 25.175000 MHz
    DMT 0x05: 640x480 72.808802 Hz 4:3 37.861 kHz 31.500000 MHz
    DMT 0x06: 640x480 75.000000 Hz 4:3 37...

[Andre Ruiz (andre-ruiz)]

Found one more bug report that may be related: https://github.com/raspberrypi/linux/issues/5195

[Andre Ruiz (andre-ruiz)]

This is the complete boot log, just in case something in it is useful (taken from the serial console including the firmware initialization part):

https://pastebin.canonical.com/p/2HMBbBdcFp/

This is the config file used in that boot (which btw is the stock file from image, unchanged):

https://pastebin.canonical.com/p/ySzZhbwPf2/

Note that the screen goes blank at the very end of that log, a few lines before the login prompt (cannot say exactly where, seems to be around the EXT4-fs (sda2): re-mounted message.

At the same time, this logs seems to be incomplete compared from what I can see on the HDMI output when it works (in that case, with fKMS), I can see many more lines between those last "rcu:" lines and the prompt which I cannot see on the serial. Edit: I just realised the difference is that I only see kernel messages on serial while I see kernel messages + systemd service messages on hdmi. No real problem here.

Unplugging power to the monitor after it went dark and pluggin it back does not help (this was suggested in one of the github issues).

[Andre Ruiz (andre-ruiz)]

Ok, adding vc4.force_hotplug=0x01 to the kernel command line _does_ fix the issue, the screen flickers for a second but comes back with normal image and stays on.

Also, now I seem to be able to decode edid many times in a row without it disappearing (appearing empty).

I'm not running the test of forcing an edid.bin file because now there seems to be no point in doing it, let me know if you need this or any other test.

[Dave Jones (waveform)]

> If I switch to the KMS overlay, the message is exactly the same, so it means the EDID is accessible from the monitor in either case.

Not quite the case I'm afraid. The output you're quoting there is from the pi's bootloader running on the GPU. So, what you've established there is that the GPU (which ignores hotplug, as noted in 6by9's comment in the upstream bug) reads the EDID. This is all that's needed in the FKMS case because there the kernel driver is a thin shim that just talks to the firmware running on the GPU and retrieves EDID data from there.

However in the KMS case, the bootloader eventually relinquishes control of the GPU to the linux kernel, and the kernel then attempts to retrieve the EDID independently (as least, this is my understanding of things -- I'm no expert in this area; corrections welcome!).

Glad to hear the force_hotplug option works; that largely gels with the idea that the monitor is not correctly asserting hot-plug. If it's not asserting hot-plug on boot, the would also explain why re-plugging the monitor doesn't work either. FKMS doesn't care, it just reads the EDID anyway, but the KMS driver is waiting for a signal to do so and there's none on boot, or later when the monitor is re-plugged.

[Andre Ruiz (andre-ruiz)]

Thank you for the explanation.

So that means that with different monitors I could potentially not need this vc4.force_hotplug option, right? I will (as time permits) try plugging this device in all my other monitors to compare, let's see how it goes (this monitor specifically is a laptop LCD using a lcd controller from ebay -- the others are all lg/samsung/dell ones, should behave better).

[Dave Jones (waveform)]

> So that means that with different monitors I could potentially not need this vc4.force_hotplug option, right?

That is certainly my suspicion; it doesn't much surprise me that an LCD controller isn't *quite* obeying the HDMI spec (or perhaps that it expects to be part of something else which handles asserting the hot-plug line). I would certainly hope a branded monitor behaves properly, though.

[Juerg Haefliger (juergh)]

Sounds like a regression given it worked before. Do you know which daily build was the last one that still worked for you?

[Juerg Haefliger (juergh)]

Could you try this kernel and let me know if it behaves any better?

https://people.canonical.com/~juergh/raspi/noble/linux-image-6.8.8-9001-raspi_6.8.8-9001-1+v8+git43db870c_arm64.deb

It's the foundation's 6.8 kernel compiled to run on Ubuntu.

$ wget ...
$ dpkg -i linux-image-6.8.8-9001-raspi_6.8.8-9001-1+v8+git43db870c_arm64.deb
$ flash-kernel --force 6.8.8-9001-raspi
$ reboot

If that kernel is not better, please give this one a try:

https://people.canonical.com/~juergh/raspi/noble/linux-image-6.6.29-9001-raspi_6.6.29-9001-1+v8+git3a8b7204_arm64.deb

That is the foundation's currently supported 6.6 kernel. Could be that something broke between 6.6 and 6.8.

[Andre Ruiz (andre-ruiz)]

@Dave so I tested this in 4 other monitors and they all worked as expected without the need for forcing hotplug.

@Juerg you could say it's a regression because it worked on that specific monitor and now it does not work anymore, but it is now understood what happend and the reason for that (there was a switch from fkms to kms right before official release and all my other tests were with daily builds which were not using kms yet). But since it still works ok with all my other monitors I'm not sure how much it is a regression or how much it is just a fault of my specific hardware.

Anyway, I'll test both kernels and let you know.

[Andre Ruiz (andre-ruiz)]

Well, I cannot explain why but my problematic monitor decided to work well even without forcing hotplug, so now I cannot test the new kernels because I can't see the problem anymore.

I'll revisit this if it starts misbehaving again.