Splash screen fails to display on recent pi core18 images

[Dave Jones]

The current core18 image [1] for the raspberry pi fails to display the "Core" splash screen on boot. This is because psplash fails to open the /dev/fb0 framebuffer device, because it doesn't exist. This appears to be due to a lack of supporting kernel modules in the initrd.img (fb_sys_fops, drm, vc4, etc.) which were formerly present but are missing from the version I'm testing (pi-kernel 4.15.0-1041.44, snap rev 42).

Steps to reproduce:

* Flash the image to a uSD card and boot the pi with it (preferably with a serial console attached).
* Note screen remains black instead of displaying the familiar "Core" text under an Ubuntu logo.
* If serial console is attached, note "Error opening /dev/fb0" in the output shortly after u-boot starts the kernel; this is output from psplash failing

[1] http://cdimage.ubuntu.com/ubuntu-core/18/current/ubuntu-core-18-armhf+raspi3.img.xz

[Oliver Grawert (ogra)]

note that it works fine if you comment dtoverlay=vc4-fkms-v3d in config.txt which makes it use the native linux frambuffer (i remember we had similar issues with the vc4 overlay in the past)

[Dave Jones (waveform)]

Tested Oliver's suggestion and indeed it does work if the overlay is removed but I assume that the overlay is there for a reason. Furthermore, extracting the psplash binary from the psplash initrd and running it after boot has concluded (at which point /dev/fb0 exists) works successfully which suggests there's no issue with the framebuffer itself, just that the framebuffer doesn't exist (presumably due to the aforementioned lack of kernel modules in the initrd) at the (very early) point in the boot where psplash is executed.

[Oliver Grawert (ogra)]

the overlay is needed for mir-kiosk to allow it to work at all (unaccelerated though)

to have mir-kiosk fully accelerated one needs to use -kms-v3d instead but this in turn disables all video decoding in the vc4 core. i originally picked -fkms-v3d for the gadget since it is the least evil (makes mir semi-work while still allowing to use omxplayer accelerated (see the omxplayer-pi snap for testing)).

long term it would be really cool if the graphics overlay selection could be a bit more dynamic (this is indeed orthogonal to this bug though) without the need to hardcode such stuff on boot or if mir-kiosk could simply work on top of vc4 without need for the overlay (the mir team was looking into that but i do not know the current status).

[Paolo Pisati (p-pisati)]

Last working image / kernel version? First broken image / kernel version? So we can compare the two.

[Dave Jones (waveform)]

Indeed - I'd rather not remove overlays without being certain we don't need them. For now though, let's just get the splash screen working again (as it's a little un-nerving for new users to be presented with a straight black screen during boot and have no indication anything's working other than a blinking LED).

In this case I'm certain all the kernel modules required are being built (because as mentioned psplash works *after* boot), but there's only a single kernel module (which looks related to i2c) present in the current initrd.img (which is part of the kernel snap). So I think the only thing required here is to ensure that the kernel modules which were present in the stable/current version (snap rev 21) are also present in the current version (snap rev 42).

Specifically, the kernel modules included in the initrd in snap rev 21 are as follows:

lib/modules/4.15.0-1031-raspi2/kernel/drivers/video/fbdev/core/sysfillrect.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/video/fbdev/core/syscopyarea.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/video/fbdev/core/sysimgblt.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/video/fbdev/core/fb_sys_fops.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/ipu-v3/imx-ipu-v3.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/arm/mali-dp.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/arm/hdlcd.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/vgem/vgem.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/drm_kms_helper.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/vc4/vc4.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/i2c/sil164.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/i2c/tda998x.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/i2c/ch7006.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/tve200/tve200_drm.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/tinydrm/repaper.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/tinydrm/mipi-dbi.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/tinydrm/st7586.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/tinydrm/mi0283qt.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/tinydrm/core/tinydrm.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/mxsfb/mxsfb.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/synopsys/dw-hdmi-cec.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/synopsys/dw-hdmi-i2s-audio.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/synopsys/dw-hdmi.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/synopsys/dw-hdmi-ahb-audio.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/nxp-ptn3460.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/adv7511/adv7511_drm.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/lvds-encoder.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/ti-tfp410.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/dumb-vga-dac.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/tc358767.ko
lib/modules/4.15.0-1031-raspi2/kernel/drivers/gpu/drm/bridge/sii902x.ko
lib/modules/4.1...

[Paolo Pisati (p-pisati)]

If you remove the dtoverlay line (just tested it) and it works fine during boot (before root is mounted), it means initrd has all necessary modules to make it work, so it's not like initrd is missing something.

[Dave Jones (waveform)]

@p-pisati: that assumes the overlay isn't required by anyone; as ogra notes above, the overlay is required for mir-kiosk to work. I don't know how important mir-kiosk is, but it seems it might be fairly relevant to core [1]. Either way, it seems less risky to me to add modules to initrd (which only enables functionality during early boot) than to remove an overlay (which potentially breaks existing functionality).

[1] https://ubuntu.com/blog/mir-kiosk-uses-mir

[Oliver Grawert (ogra)]

mir-kiosk is currently the only supported way to get grahical (mostly kiosk) applications up and running on ubuntu core, some of our digital-signage customers use it so currently the overlay is a requirement (until the mir team solves it with a pi specific mir backend or the vc4 driver changes in some way like stated abov)

[Łukasz Zemczak (sil2100)]

@p-pisati: as mentioned earlier, with pi-kernel rev 21 the splash support worked with dtoverlay line present, which does not anymore - meaning it's a regression. It's also quite a visible regression, since that's the first thing Ubuntu Core pi users see when they boot the device. This might be a 'release critical' regression for 18.04.3, which is planned to be released on the 1st of August. As mentioned by Olivier and Dave, removal of the dtoverlay line does not seem to be a feasible option as of now. And besides, as mentioned, the pi-kernel snap regressed by breaking something that worked fine previously.

Can we identify which modules from the list that Dave presented in comment #5 (which were present in rev 21 but are no longer in 42) would be required and get them back into the initrd? Could we revert the change that removed all those (possibly unneeded) modules from the initrd at least until we know nothing regresses?

[Paolo Pisati (p-pisati)]

Test kernel snap available here:

https://people.canonical.com/~ppisati/lp1837209/pi-kernel_4.15.0-1041.44_armhf.snap

and precooked image here:

https://people.canonical.com/~ppisati/lp1837209/ubuntu-core-18-pi_lp1837209fix.img.xz

[Dave Jones (waveform)]

Thanks Paolo - I can confirm the pre-cooked image displays the boot logo nicely (couldn't test the snap itself, but I suspect that's due to me lacking the correction snap incantations rather than anything wrong with it - snap complained about replacing a signed kernel with an unasserted one).

[Łukasz Zemczak (sil2100)]

When would it be possible to get this change released as part of the pi-kernel snap? I still need to poke Adam, but I suppose we wouldn't want to release our .3 core images without this fix. Will we have to delay the core image releases for .3, or maybe can these changes go into pi-kernel outside of the regular kernel cycle?

[Paolo Pisati (p-pisati)]

Fixed kernel snaps are building as we speak, and will be uploaded to the store -edge and -beta immediately after.

I'll build another test image first thing tomorrow morning, and then we can promote it to -stable.

[Paolo Pisati (p-pisati)]

Ok, tested and uploaded, let me know if you test images are fixed now.

[Łukasz Zemczak (sil2100)]

@p-pisati: Dave just confirmed that our latest edge-based images now have the Ubuntu splash logo back, thanks a lot! Could we get this version promoted to stable anytime soon?

[Zygmunt Krynicki (zyga)]

The bug is in progress but it is unclear who is working on it. Can the party responsible please self-assign and update the status of the issue.

Is there something to be done in the snappy project for this issue to be fixed?
Is this just waiting on a kernel snaps? Are those in stable channel now?