[Kernel 5.15] System messages (kernel and systemd) are not shown during boot until it reach a late stage

Bug #1976147 reported by fprietog
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux-raspi (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
Fix Released
Undecided
Juerg Haefliger
Kinetic
Fix Released
Undecided
Unassigned

Bug Description

# lsb_release -rd
Description: Ubuntu 22.04 LTS
Release: 22.04

# uname -a
Linux fpgrpi 5.15.0-1008-raspi #8-Ubuntu SMP PREEMPT Thu May 19 11:58:51 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Problem summary:
----------------
System is a Raspberry Pi 4 Model B Rev 1.4.

When you remove the 'quiet' and 'splash' options in file /boot/firmware/cmdline.txt in order to see "live" the kernel and systemd messages generated during boot these messages are not shown from the start of the boot process. It remains a blank screen until boot process reached a late stage (maybe the gpu driver init but I'm not sure).

It works with previous kernels 5.13 from Impish (it show all messages from the boot start) but is broken in kernels 5.15 from jammy.

What was expected:
------------------
As in kernel 5.13 the boot will show 4 raspberries in the top of the screen and the kernel and systemd messages generated will be shown from the start of the boot process.

How to reproduce:
-----------------
Remove the 'quiet' and 'splash' options in file /boot/firmware/cmdline.txt in order to see "live" the kernel and systemd messages generated during boot. With kernel 5.13 these messages are shown from the boot process start but with kernel 5.15 the screen remains blank until a late boot stage.

CVE References

fprietog (fprietog)
description: updated
Juerg Haefliger (juergh)
tags: added: kern-3419
Revision history for this message
Juerg Haefliger (juergh) wrote :

Works just fine for me with both 5.15.0-1006 and -1011. Do you see the rainbow splash? Is the monitor in stand-by and wakes up later during the boot process? Can you try a different monitor?

Changed in linux-raspi (Ubuntu):
status: New → Incomplete
Revision history for this message
Juerg Haefliger (juergh) wrote :

Can you try 5.15.0-1012.14 from -proposed?

$ sudo apt-add-repository -p proposed
$ sudo apt install linux-raspi

Revision history for this message
fprietog (fprietog) wrote :

> Do you see the rainbow splash?
Yes

> Is the monitor in stand-by and wakes up later during the boot process?
Yes. But in a "normal" boot (e.g. Ubuntu 21.10) it also made a quick jump from "on to stand-by to on" at middle of the boot. The difference is that in Ubuntu 21.10 boot messages are shown from the beginning so are shown before and after that jump. In Ubuntu 22.04 boot messages are not shown until that jump occurs.
I've made two videos showing the boot process in Ubuntu 21.0 and Ubuntu 22.04 (both fresh installs only removing quiet and splash options from cmdline.txt). I'll attach these videos in a later posts.

> Can you try a different monitor?
Yes, tried 3 devices (two monitors and a TV) and all of then shows the same problem.

> Can you try 5.15.0-1012.14 from -proposed?
Yes. Tested and still the same problem.

I've also tried Ubuntu's current critical bootloader (pieeprom-2022-01-25.bin) and the newest one available from Raspberry Pi Foundation (pieeprom-2022-04-26.bin) and got the same problem in both.

Also tested current Raspberry Pi OS (64 bits) that come with a 5.15 kernel as Ubuntu 22.10 and it works as expected: boot messages are shown from the beginning of the boot process.

Revision history for this message
fprietog (fprietog) wrote :

First video attached: booting Ubuntu 21.10:
- Rainbow screen.
- 4 raspberrys shown at left top of screen and boot messages shown from the beginning of the boot.
- Quick jump from on to stand by to on.
- Rest of boot messages shown until boot's end.

Revision history for this message
fprietog (fprietog) wrote :

Second video attached: booting Ubuntu 22.04:
- Rainbow screen.
- BLANK SCREEN!!!
- Quick jump from on to stand by to on.
- Rest of boot messages shown until boot's end.

Revision history for this message
Juerg Haefliger (juergh) wrote :

Can you attached the dmesg from a good and a bad boot?

Revision history for this message
fprietog (fprietog) wrote :

Attached the "good" boot: Ubuntu 21.10 kernel 5.13.0-1031-raspi.

Revision history for this message
fprietog (fprietog) wrote :

Attached the "bad" boot: Ubuntu 22.04 kernel 5.15.0-1011-raspi.

Revision history for this message
fprietog (fprietog) wrote :

I've compared both dmesg and find this that seems to be related to the problem:

In the "good" boot (Ubuntu 21.10) there is a framebuffer initialization at the very beginning of the boot:

[ 0.285108] simple-framebuffer 3e2fe000.framebuffer: framebuffer at 0x3e2fe000, 0x7f8000 bytes
[ 0.285147] simple-framebuffer 3e2fe000.framebuffer: format=a8r8g8b8, mode=1920x1080x32, linelength=7680
[ 0.296278] Console: switching to colour frame buffer device 240x67
[ 0.306251] simple-framebuffer 3e2fe000.framebuffer: fb0: simplefb registered!

But that is totally missing in the "bad" boot (Ubuntu 22.04).

Revision history for this message
Juerg Haefliger (juergh) wrote :

Yes, simplefb was turned into a module in 5.15 so that is expected. But I didn't expect this to make a difference given that it works just fine for me. Looks like your monitors can't render whatever the dummy console driver is outputting or something.

I flipped it back in a test kernel. Can you give this a try:
https://kernel.ubuntu.com/~juergh/lp1976147/

Revision history for this message
fprietog (fprietog) wrote :

The test kernel works. It doesn't show the 4 raspberries at the top left but the messages, that are the real important thing, are shown.

Regarding the monitor; I've tested the problem in 3 different FullHD devices (two monitors and a TV, all of them old devices) and none of them works. The only thing I've used in all tests (apart from the Raspberry itself) is the uHDMI to HDMI cable because I only have one. It's an Amazon Basics cable bought in December 2013. It's old and don't meet the newest HDMI specs, so it may be the source of the problem.

Revision history for this message
fprietog (fprietog) wrote :

I've also tested the newest Raspberry Pi OS (64 bits) that came with kernel 5.15.32-v8+ #1538. It works as expected (4 raspberries at top, messages, ...) and show the simple-framebuffer messages in the dmesg so I think it's using a non module simplefb.

Maybe the Raspberry OS developers faced this problem (or another) and revert back that change in their 5.15 kernel.

Revision history for this message
Juerg Haefliger (juergh) wrote :

Looks like the missing berries are due to FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER=y. Please try package versions 5.15.0-1012.14+20220701.gitb3a4f444 from:
https://kernel.ubuntu.com/~juergh/lp1976147/

Juerg Haefliger (juergh)
Changed in linux-raspi (Ubuntu):
status: Incomplete → Triaged
Changed in linux-raspi (Ubuntu Jammy):
status: New → In Progress
assignee: nobody → Juerg Haefliger (juergh)
Revision history for this message
fprietog (fprietog) wrote :

Tested 5.15.0-1012.14+20220701.gitb3a4f444: Perfect!!!

Thank you very much for your effort with this problem!

Revision history for this message
Juerg Haefliger (juergh) wrote :

Thanks for testing. Patch submitted to the ML:
https://lists.ubuntu.com/archives/kernel-team/2022-July/131475.html

Revision history for this message
fprietog (fprietog) wrote :

Thank you!

BTW: there is no reference to FRAMEBUFFER_CONSOLE_DEFERRED_TAKEOVER in the patch you submitted (?).

Revision history for this message
Juerg Haefliger (juergh) wrote :

Yeah, it was disabled in our kernel and I enabled it for your first test kernel because I thought we want it as well but that was obviously not the case. The second test kernel had it disabled again. So all need is FB_SIMPLE=y.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (75.5 KiB)

This bug was fixed in the package linux-raspi - 5.15.0-1014.16

---------------
linux-raspi (5.15.0-1014.16) jammy; urgency=medium

  * jammy/linux-raspi: 5.15.0-1014.16 -proposed tracker (LP: #1983893)

  * Jammy update: v5.15.46 upstream stable release (LP: #1981864)
    - [Packaging] raspi: add python3-dev as build dependency

  * [Kernel 5.15] System messages (kernel and systemd) are not shown during boot
    until it reach a late stage (LP: #1976147)
    - [Config] raspi: Disable DRM_SIMPLEDRM and set FB_SIMPLE=y

  [ Ubuntu: 5.15.0-47.51 ]

  * jammy/linux: 5.15.0-47.51 -proposed tracker (LP: #1983903)
  * Jammy update: v5.15.46 upstream stable release (LP: #1981864)
    - UBUNTU: [Packaging] Move python3-dev to build-depends
  * touchpad and touchscreen doesn't work at all on ACER Spin 5 (SP513-54N)
    (LP: #1884232)
    - x86/PCI: Eliminate remove_e820_regions() common subexpressions
    - x86: Log resource clipping for E820 regions
    - x86/PCI: Clip only host bridge windows for E820 regions
    - x86/PCI: Add kernel cmdline options to use/ignore E820 reserved regions
    - x86/PCI: Disable E820 reserved region clipping via quirks
    - x86/PCI: Revert "x86/PCI: Clip only host bridge windows for E820 regions"
  * [SRU][H/OEM-5.13/OEM-5.14/U][J/OEM-5.17/U] Fix invalid MAC address after
    hotplug tbt dock (LP: #1942999)
    - SAUCE: igc: wait for the MAC copy when enabled MAC passthrough
  * Mass Storage Gadget driver truncates device >2TB (LP: #1981390)
    - usb: gadget: storage: add support for media larger than 2T
  * AMD Rembrandt: DP tunneling fails with Thunderbolt monitors (LP: #1983143)
    - SAUCE: drm/amd: Fix DP Tunneling with Thunderbolt monitors
    - drm/amd/display: Fix for dmub outbox notification enable
    - Revert "drm/amd/display: Fix DPIA outbox timeout after S3/S4/reset"
    - drm/amd/display: Reset link encoder assignments for GPU reset
    - drm/amd/display: Fix DPIA outbox timeout after S3/S4/reset
    - drm/amd/display: Fix new dmub notification enabling in DM
    - SAUCE: thunderbolt: Add DP out resource when DP tunnel is discovered.
  * Fix sub-optimal I210 network speed (LP: #1976438)
    - igb: Make DMA faster when CPU is active on the PCIe link
  * e1000e report hardware hang (LP: #1973104)
    - e1000e: Enable GPT clock before sending message to CSME
    - Revert "e1000e: Fix possible HW unit hang after an s0ix exit"
  * ioam6.sh in net from ubuntu_kernel_selftests fails with 5.15 kernels in
    Focal (LP: #1982930)
    - selftests: net: fix IOAM test skip return code
  * Additional fix for TGL + AUO panel flickering (LP: #1983297)
    - Revert "UBUNTU: SAUCE: drm/i915/display/psr: Fix flicker on TGL + AUO panel"
    - drm/i915/display: Fix sel fetch plane offset calculation
    - drm/i915: Nuke ORIGIN_GTT
    - drm/i915/display: Drop PSR support from HSW and BDW
    - drm/i915/display/psr: Handle plane and pipe restrictions at every page flip
    - drm/i915/display/psr: Do full fetch when handling multi-planar formats
    - drm/i915/display: Drop unnecessary frontbuffer flushes
    - drm/i915/display: Handle frontbuffer rendering when PSR2 selective fetch is
      enabled
    - drm/i915/display: Fi...

Changed in linux-raspi (Ubuntu Jammy):
status: In Progress → Fix Released
Revision history for this message
fprietog (fprietog) wrote :

Just installed and tested linux-raspi 5.15.0-1014.16 from proposed and can confirm that problem is fixed.

Thank you very much!

Revision history for this message
fprietog (fprietog) wrote :

Seems that this is not fixed in kinetic 5.19 kernel:

# lsb_release -rd
Description: Ubuntu 22.10
Release: 22.10

# uname -a
Linux fpgrpi 5.19.0-1006-raspi #13-Ubuntu SMP PREEMPT Mon Oct 17 19:01:51 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

Is it a regression? Do I need to open a new bug?

Revision history for this message
Juerg Haefliger (juergh) wrote :

We seem to have lost the relevant config changes in Kinetic. Sigh.

Changed in linux-raspi (Ubuntu Kinetic):
status: New → Confirmed
Juerg Haefliger (juergh)
Changed in linux-raspi (Ubuntu Kinetic):
status: Confirmed → Fix Committed
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the linux-raspi/5.19.0-1010.17 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-kinetic' to 'verification-done-kinetic'. If the problem still exists, change the tag 'verification-needed-kinetic' to 'verification-failed-kinetic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: kernel-spammed-kinetic-linux-raspi verification-needed-kinetic
Revision history for this message
fprietog (fprietog) wrote :

Proposed linux-raspi/5.19.0-1010.17 kernel tested and verified that solves the problem.
Tag 'verification-needed-kinetic' changed to 'verification-done-kinetic'.

tags: added: verification-done-kinetic
removed: verification-needed-kinetic
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (103.1 KiB)

This bug was fixed in the package linux-raspi - 5.19.0-1011.18

---------------
linux-raspi (5.19.0-1011.18) kinetic; urgency=medium

  * kinetic/linux-raspi: 5.19.0-1011.18 -proposed tracker (LP: #1999742)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

  [ Ubuntu: 5.19.0-28.29 ]

  * kinetic/linux: 5.19.0-28.29 -proposed tracker (LP: #1999746)
  * mm:vma05 in ubuntu_ltp fails with '[vdso] bug not patched' on kinetic/linux
    5.19.0-27.28 (LP: #1999094)
    - fix coredump breakage

linux-raspi (5.19.0-1010.17) kinetic; urgency=medium

  * kinetic/linux-raspi: 5.19.0-1010.17 -proposed tracker (LP: #1997789)

  * [Kernel 5.15] System messages (kernel and systemd) are not shown during boot
    until it reach a late stage (LP: #1976147)
    - [Config] raspi: Disable DRM_SIMPLEDRM and set FB_SIMPLE=y
    - [Packaging] raspi: Remove simpledrm from modules ABI

  * Kinetic update: v5.19.17 upstream stable release (LP: #1994179)
    - [Config] raspi: updateconfigs for ARM64_ERRATUM_2441007
    - [Config] raspi: updateconfigs for AS_HAS_NON_CONST_LEB128

  * Kinetic update: v5.19.15 upstream stable release (LP: #1994078)
    - [Config] raspi: updateconfigs for CC_HAS_AUTO_VAR_INIT_ZERO_ENABLER

  * Kinetic update: v5.19.9 upstream stable release (LP: #1994068)
    - [Config] raspi: updateconfigs for ARM64_ERRATUM_2457168
    - [Config] raspi: updateconfigs for ARM64_BTI_KERNEL

  * kinetic/linux-raspi: Update to upstream raspberrypi rpi-5.19.y (2022-10-26)
    (LP: #1995968)
    - drm/vc4: hdmi: Constify drm_display_mode
    - drm/vc4: hdmi: Remove unused argument in vc4_hdmi_supports_scrambling
    - drm/vc4: hdmi: Remove mutex in detect
    - drm/vc4: hdmi: Simplify the hotplug handling
    - drm/vc4: hdmi: Switch to detect_ctx
    - drm/vc4: hdmi: Move vc4_hdmi_supports_scrambling() around
    - drm/vc4: hdmi: Reset link on hotplug
    - media: i2c: imx219: make HBLANK r/w to allow longer exposures
    - Revert "Add PHY_ID_BCM54213PE identifier."
    - ARM: dts: Add dtparams to disable PCIe, HDMI, and SD card/eMMC
    - usb: xhci: account for num_trbs_free when invalidating TDs
    - overlays: Add rpi-sense-v2
    - media: i2c: imx477: Allow dynamic horizontal blanking control
    - vc04_services: bcm2835-codec: Remove redundant role check
    - vc04_services: bcm2835-codec: Allow encoder_cmd on ISP and deinterlace
    - media: i2c: imx477: Reset hblank on mode switch
    - media: i2c: imx477: Do not unconditionally adjust hblank and vblank limits
    - nvmem: Use NVMEM_DEVID_AUTO
    - media: i2c: imx290: Correct min HBLANK.
    - overlays: i2c-rtc: Add RV3032 support
    - configs: Add RTC_DRV_RV3032=m
    - ARM: dts: Add nvmem node for BCM2711 bootloader public key
    - configs: Add FSIA6B driver module

  * armhf kernels are compiled with gcc-11 (LP: #1995853)
    - drivers: dwc_otg: stop GCC from patching FIQ functions
    - Revert "UBUNTU: [Packaging] raspi: Use gcc-11 for armhf"
    - [Config] raspi: updateconfigs for gcc-12 for armhf

  * remove circular dep between linux-image and modules (LP: #1989334)
    - [Packaging] remove circular dep between modules and image

  [ Ubuntu: 5.19.0-27.28 ]

  * kinetic...

Changed in linux-raspi (Ubuntu Kinetic):
status: Fix Committed → Fix Released
Revision history for this message
fprietog (fprietog) wrote :

linux-raspi/5.19.0-1011.18 kernel tested and verified that solves the problem.

Thanks and best regards!

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (36.1 KiB)

This bug was fixed in the package linux-raspi - 6.2.0-1003.3

---------------
linux-raspi (6.2.0-1003.3) lunar; urgency=medium

  * lunar/linux-raspi: 6.2.0-1003.3 -proposed tracker (LP: #2009847)

  * Packaging resync (LP: #1786013)
    - [Packaging] update update.conf

  * Miscellaneous upstream changes
    - media: bcm2835-v4l2-codec: Add profile & level ctrls to decode
    - drm/vc4_plane: Add support for YUV444 formats
    - drm/vc4: Calculate bpc based on max_requested_bpc
    - media: dt-bindings: ak7375: Convert to DT schema
    - media: dt-bindings: ak7375: Add supplies
    - media: i2c: ak7375: Add regulator management
    - configs: Add CONFIG_VIDEO_AK7375=m
    - dtoverlays: Add VCM option to imx519
    - dtoverlays: Add IMX519 support to camera mux overlays
    - vc04_services: bcm2835_codec: Ignore READ_ONLY ctrls in s_ctrl
    - ASoC: bcm: Fix Rpi-PROTO and audioinjector.net Pi
    - dtoverlays: Add VCM option to Arducam64MP
    - dtoverlays: Add Arducam64MP support to camera mux overlays

  [ Ubuntu: 6.2.0-13.13 ]

  * lunar/linux: 6.2.0-13.13 -proposed tracker (LP: #2009704)
  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/master)
  * mt7921: add support of MTFG table (LP: #2009642)
    - wifi: mt76: mt7921: add support to update fw capability with MTFG table

  [ Ubuntu: 6.2.0-12.12 ]

  * lunar/linux: 6.2.0-12.12 -proposed tracker (LP: #2009698)
  * Miscellaneous Ubuntu changes
    - SAUCE: enforce rust availability only on x86_64
    - [Config] update CONFIG_RUST_IS_AVAILABLE

  [ Ubuntu: 6.2.0-11.11 ]

  * lunar/linux: 6.2.0-11.11 -proposed tracker (LP: #2009697)
  * Miscellaneous Ubuntu changes
    - [Packaging] do not stop the build if rust is not available

linux-raspi (6.2.0-1002.2) lunar; urgency=medium

  * Miscellaneous Ubuntu changes
    - [Packaging] raspi: Drop etc/kernelconfig
    - [Config] raspi: Update CC_VERSION_TEXT
    - [Config] raspi: Change non-essential options from 'y' to 'm'
    - [Config] raspi: Change unsupported options from 'y' to 'n'
    - [Config] raspi: Change undesired options from 'y' to 'n'

  [ Ubuntu: 6.2.0-10.10 ]

  * lunar/linux-unstable: 6.2.0-10.10 -proposed tracker (LP: #2007818)
  * Built-in camera device dies after runtime suspended (LP: #2007530)
    - SAUCE: usb: xhci: Workaround for runpm issue on AMD xHC
  * Miscellaneous Ubuntu changes
    - [Config] update annotations after rebase to v6.2
  * Rebase to v6.2

linux-raspi (6.2.0-1001.1) lunar; urgency=medium

  * IO errors when writing large amounts of data to USB storage in eoan on
    RPI2/3 (armhf kernel) (LP: #1852510)
    - Add dwc_otg driver

  * Miscellaneous Ubuntu changes
    - [Config] raspi: updateconfigs after import of rpi-6.2.y patchset

  * Miscellaneous upstream changes
    - raspberrypi-firmware: Update mailbox commands
    - media: uapi: Add some RGB bus formats for VC4 DPI output
    - Support RPi DPI interface in mode6 for 18-bit color
    - drm/vc4: Add FKMS as an acceptable node for dma ranges.
    - drm/vc4: Add the 2711 HVS as a suitable DMA node
    - drm/vc4: Change the default DPI format to being 18bpp, not 24.
    - drm/atomic: Don't fixup modes th...

Changed in linux-raspi (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.