[SRU] CM4 NVMe boot fails due to old start4.elf / fixup4.dat

[ww898]

[ Impact ]

CM4 users wishing to boot directly from NVMe find the initramfs is reported as corrupted. This is due to a bug in the boot firmware reported upstream as:

https://github.com/raspberrypi/firmware/issues/1731
https://github.com/raspberrypi/firmware/issues/1746

These issues were fixed upstream but Ubuntu now needs an updated linux-firmware-raspi package incorporating the fixed boot firmware (and ideally this should land prior to 22.04.2 to ensure the updated firmware can go into the images for that release).

[ Test Plan ]

* sudo add-apt-repository ppa:waveform/firmware

* sudo apt install linux-firmware-raspi

* sudo reboot

* Verify that the system boots successfully

* Run through the full battery of ISO tests for the relevant image. Specifically, Ubuntu Desktop arm64+raspi, Ubuntu Server arm64+raspi, and Ubuntu Server armhf+raspi under:

  - Kinetic dailies: http://iso.qa.ubuntu.com/qatracker/milestones/433/builds

  - Jammy dailies: http://iso.qa.ubuntu.com/qatracker/milestones/429/builds

[ Regression Potential ]

The updated package replaces the tertiary boot firmware. An obvious regression potential is therefore that one or more Pi models fail to boot. However, in addition we've encountered other issues which ultimately boiled down to boot firmware in the past (the boot firmware is also responsible for patching the device-tree). Hence the test plan needs to be comprehensive and should be tested on as many models of Pi as is reasonably possible.

Several kind users have already tested the back-ported firmware on the CM4 with NVMe boot (see comments 7 and 8), and I've tested on Pi 3 and 4 models under armhf and arm64 kernels, hence I'm quietly confident but I haven't run the full gamut of ISO tests yet. Still, SRU verification should include all supported models, and ISO tests to guard against regressions due to interactions with differing kernel versions on the SRU'd releases.

[ Original Description ]

Hi there,

there is the issue connected with "Initramfs unpacking failed: Decoding failed" error during Ubuntu boot. See https://github.com/raspberrypi/firmware/issues/1746 and https://github.com/raspberrypi/firmware/issues/1731. The Raspberry Pi have the update for start4.elf / fixup4.dat with fix it. Could you please include update for these files in new ubuntu release ASAP?

Thanks in advance!

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dave Jones (waveform)]

Marking linux-firmware-raspi2 invalid for now; will be used if we decide to backport to focal. Currently prepping an upload for ppa:waveform/firmware with a test updated firmware. If that works well, will upload to lunar then start SRU to kinetic and jammy.

[Dave Jones (waveform)]

Test firmware packages now built in ppa:waveform/firmware, including tentative backports for kinetic and jammy:

https://launchpad.net/~waveform/+archive/ubuntu/firmware

I would be grateful if anyone could test these on their particular Pi setups, whether using NVMe or otherwise (to guard against regressions). I've tested locally on a jammy and kinetic setup without issue so far, booting one from USB and one from SD card but I don't have a CM4 NVMe setup to test with.

Theoretically we *should* be testing this on lunar, but there aren't any daily images for that yet and given this will need SRU'ing anyway (and the boot firmware is *relatively* disconnected from the release it happens to be installed upon) we can take jammy and kinetic tests as indicative.

I would recommend the following testing procedure (given we're messing with the boot partition), if you care about the content of your Pi (i.e. assuming this is anything other than an ephemeral installation):

* Shut down your Pi

* Connect your current storage to another computer, and backup the entirety of the boot partition. Assuming it mounts as /media/$USER/system-boot the following should suffice:

  $ tar -C /media/$USER/system-boot -cvzf ~/system-boot-backup.tar.gz .

* Safely remove the storage, connect it back to your Pi, then boot it

* Add the PPA specified above:

  $ sudo add-apt-repository -y ppa:waveform/firmware

* Upgrade the linux-firmware-raspi package and reboot:

  $ sudo apt install linux-firmware-raspi
  $ sudo reboot

* Please report success or failure here (including release tested, boot media used, and model of Pi)

* If everything breaks, the worst-case scenario (short of actually catching fire!) is your Pi cannot boot. To restore the old content of the boot partition, attach your storage to the machine on which you took the backup. Again, assuming the boot partition is mounted as /media/$USER/system-boot:

  $ tar -C /media/$USER/system-boot -xvzf ~/system-boot-backup.tar.gz

* Safely remove the storage, re-connect it to the Pi, and boot it

* Note at this point we've restored the boot partition, but the newer package is still installed, as is the PPA. This means any kernel upgrade (or anything that causes flash-kernel to run) will once again copy the new firmware over to the boot partition. We need to correct that:

  $ sudo add-apt-repository -r -y ppa:waveform/firmware
  $ sudo apt install linux-firmware-raspi/$(lsb_release -sc)

* At this point your Pi should be back to the currently shipping linux-firmware-raspi package on jammy/kinetic/lunar

[ww898 (ww8987)]

I have some CM4 with NVMe, so I can try this week.

[Dave Jones (waveform)]

@ww8987 Excellent, thank you -- I look forward to hearing any results!

[François (francois-bonelle)]

Hello,

I have tested nvme boot (one CM4 on tofu board - https://store.oratek.com/products/tofu). All is OK, boot successful :)

Thanks very much

[ww898 (ww8987)]

Hi, I have just tested it on CM4 with NVMe. Everything works!!! Thanks a lot! I have tested on https://uplab.pro/2022/03/compute-blade-changelog/ if you interesting.

[Dave Jones (waveform)]

Excellent news; combined with the testing I've done here on a few boards & archs I'd say this is good to land. I'll upload to lunar and convert this ticket for SRU to kinetic & jammy. Many thanks for all the testing help!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware-raspi - 8-0ubuntu1

---------------
linux-firmware-raspi (8-0ubuntu1) lunar; urgency=medium

  * New upstream boot firmware, 1.20220331
    - Fixes issues with NVME boot (LP: #1993937, LP: #1989713)
  * New upstream wifi firmware, 20210315-3+rpt7

    [ Serge Schneider ]
    - Add update-alternatives for cyfmac43455-sdio.bin

    [ Phil Elwell ]
    - Add license from Synaptics
    - Update SYN43436P firmware
      - The shipping firmware for the SYN43436P does not support 4-way
        handshake offloading. This new firmware (version string "Version:
        9.88.4.77 CRC: 143f9f15 Date: Thu 2022-03-31 17:25:16 CST Ucode
        Ver: 1043.20743 FWID: 01-3b307371") fixes that.
    - Updated SYN43436S firmware
      - The embedded clm_blob in the previous SYN43436S firmware did not offer
        any channels when the country code was set to KR (Korea). This
        firmware fixes that.
      - Full version string:
        Version: 7.45.96.141 (g53ba0094) CRC: 6a9f22a0 Date: Tue 2022-07-05
        21:30:23 CST Ucode Ver: 1043.20739 FWID: 01-eef9b3bf

  * New upstream bluetooth firmware release

    [ Phil Elwell ]

    - This updated Bluetooth firmware contains Braktooth fixes for CYW43455,
      addressing the following vulnerabilities:
      CVE-2021-34145
      CVE-2021-34146
      CVE-2021-34147
      CVE-2021-34148
    - Add Synaptics firmware:
      SYN43430A1.hcd and SYN43430B0.hcd are firmwares for the two Zero 2 W
      variants.
    - Add board-specific links for the Zero 2 W firmwares

 -- Dave Jones <email address hidden> Thu, 28 Jul 2022 13:50:22 +0100

[Dave Jones (waveform)]

That took a while to migrate -- sorry about the delay! I'll get on with prepping the SRUs this week.

[Łukasz Zemczak (sil2100)]

Hello ww898, or anyone else affected,

Accepted linux-firmware-raspi into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware-raspi/8-0ubuntu1~22.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Dave - are we sure we can safely drop linux-firmware-raspi2 from jammy as well?

[François (francois-bonelle)]

Hello,

I am using the patched firmware for several weeks on Ubuntu Jammy (CM4). I don't have any problem. I prefer a lot to use LTS release than non LTS ;)

Regards

[Dave Jones (waveform)]

> Dave - are we sure we can safely drop linux-firmware-raspi2 from jammy as well?

Oh damn, I did drop that in the backport, didn't I. No, it's not safe to drop that package -- it needs to be there for upgraders. Can you reject that upload, and I'll re-push it with the transitional package re-instated?

[Łukasz Zemczak (sil2100)]

Sure thing! Rejecting ;)

[Dave Jones (waveform)]

Thanks! And thank you "git revert" for making it such a trivial task to re-instate :)

It should be re-uploaded now, with the transitional package re-instated.

[Łukasz Zemczak (sil2100)]

Hello ww898, or anyone else affected,

Accepted linux-firmware-raspi into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware-raspi/8-0ubuntu1~22.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dave Jones (waveform)]

Thanks to @francois-bonelle and @ww8987 for all their testing help!

I've also tested on several Pi models here (both SD and SSD booting, and boot + wifi + bt functionality although I only tested the bluetooth bits on a couple of models but still should have covered both the 4343x and 4345x chipsets). Anyway, no issues here, so I'm going to mark the verification as done on kinetic and jammy at this point.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware-raspi - 8-0ubuntu1~22.10

---------------
linux-firmware-raspi (8-0ubuntu1~22.10) kinetic; urgency=medium

  * New upstream boot firmware, 1.20220331
    - Fixes issues with NVME boot (LP: #1993937, LP: #1989713)
  * New upstream wifi firmware, 20210315-3+rpt7

    [ Serge Schneider ]
    - Add update-alternatives for cyfmac43455-sdio.bin

    [ Phil Elwell ]
    - Add license from Synaptics
    - Update SYN43436P firmware
      - The shipping firmware for the SYN43436P does not support 4-way
        handshake offloading. This new firmware (version string "Version:
        9.88.4.77 CRC: 143f9f15 Date: Thu 2022-03-31 17:25:16 CST Ucode
        Ver: 1043.20743 FWID: 01-3b307371") fixes that.
    - Updated SYN43436S firmware
      - The embedded clm_blob in the previous SYN43436S firmware did not offer
        any channels when the country code was set to KR (Korea). This
        firmware fixes that.
      - Full version string:
        Version: 7.45.96.141 (g53ba0094) CRC: 6a9f22a0 Date: Tue 2022-07-05
        21:30:23 CST Ucode Ver: 1043.20739 FWID: 01-eef9b3bf

  * New upstream bluetooth firmware release

    [ Phil Elwell ]

    - This updated Bluetooth firmware contains Braktooth fixes for CYW43455,
      addressing the following vulnerabilities:
      CVE-2021-34145
      CVE-2021-34146
      CVE-2021-34147
      CVE-2021-34148
    - Add Synaptics firmware:
      SYN43430A1.hcd and SYN43430B0.hcd are firmwares for the two Zero 2 W
      variants.
    - Add board-specific links for the Zero 2 W firmwares

 -- Dave Jones <email address hidden> Thu, 28 Jul 2022 13:50:22 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for linux-firmware-raspi has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware-raspi - 8-0ubuntu1~22.04

---------------
linux-firmware-raspi (8-0ubuntu1~22.04) jammy; urgency=medium

  * New upstream boot firmware, 1.20220331
    - Fixes issues with NVME boot (LP: #1993937, LP: #1989713)
  * New upstream wifi firmware, 20210315-3+rpt7

    [ Serge Schneider ]
    - Add update-alternatives for cyfmac43455-sdio.bin

    [ Phil Elwell ]
    - Add license from Synaptics
    - Update SYN43436P firmware
      - The shipping firmware for the SYN43436P does not support 4-way
        handshake offloading. This new firmware (version string "Version:
        9.88.4.77 CRC: 143f9f15 Date: Thu 2022-03-31 17:25:16 CST Ucode
        Ver: 1043.20743 FWID: 01-3b307371") fixes that.
    - Updated SYN43436S firmware
      - The embedded clm_blob in the previous SYN43436S firmware did not offer
        any channels when the country code was set to KR (Korea). This
        firmware fixes that.
      - Full version string:
        Version: 7.45.96.141 (g53ba0094) CRC: 6a9f22a0 Date: Tue 2022-07-05
        21:30:23 CST Ucode Ver: 1043.20739 FWID: 01-eef9b3bf

  * New upstream bluetooth firmware release

    [ Phil Elwell ]

    - This updated Bluetooth firmware contains Braktooth fixes for CYW43455,
      addressing the following vulnerabilities:
      CVE-2021-34145
      CVE-2021-34146
      CVE-2021-34147
      CVE-2021-34148
    - Add Synaptics firmware:
      SYN43430A1.hcd and SYN43430B0.hcd are firmwares for the two Zero 2 W
      variants.
    - Add board-specific links for the Zero 2 W firmwares

 -- Dave Jones <email address hidden> Thu, 28 Jul 2022 13:50:22 +0100