"Synchronous Abort" handler and reboot loop on rpi

[Paul Larson]

[ Impact ]

The issue results in an unbootable system on certain Pi hardware. While this hardware (Pi 400) wasn't originally supported at Focal's release, as the current LTS it needs to be supported. Furthermore, this version of the bootloader also fixes another issue (LP: #1900693) and thus needs back-porting to Bionic to support Core 18's gadget.

[ Test Case ]

* Flash a focal image to a card and boot it
* Enable proposed (https://wiki.ubuntu.com/Testing/EnableProposed)
* Upgrade the u-boot-rpi package to the proposed version: sudo apt install -t focal-proposed u-boot-rpi
* Reboot the system: sudo reboot
* Check the system boots correctly to login prompt

[ Regression Potential ]

As with anything that messes with the bootloader, the potential is there for an unbootable system. However, the update fixes a variety of serious issues (LP: #1906551, LP: #1900693, and indirectly LP: #1900879) and enables hardware on the current LTS.

[ Original Description ]

I first noticed this on a focal daily image (http://cdimage.ubuntu.com/ubuntu-server/focal/daily-preinstalled/20201202/ or the current pending one should be able to reproduce it). On the rpi400 device, it goes into a reboot loop. I'm also seeing this behavior with the current uc20 arm64 images on rpi400, and I suspect the same problem could be happening on cm4, but I can't confirm that yet because I don't have one in front of me.

Here's the output I'm getting from serial before it goes into the reboot loop (it gives this output each time it reboots):
U-Boot 2020.10+dfsg-1ubuntu0~20.04.1 (Nov 12 2020 - 17:05:18 +0000)

DRAM: 3.9 GiB
RPI: Board rev 0x13 outside known range
RPI Unknown model (0xc03130)
MMC: mmcnr@7e300000: 1, emmc2@7e340000: 0
Loading Environment from FAT... ** No partition table - mmc 0 **
In: serial
Out: vidconsole
Err: vidconsole
Net: eth0: ethernet@7d580000
PCIe BRCM: link up, 5.0 Gbps x1 (SSC)
starting USB...
Bus xhci_pci: probe failed, error -110
No working controllers found
Hit any key to stop autoboot: 0
switch to partitions #0, OK
mmc0 is current device
** No partition table - mmc 0 **
Card did not respond to voltage select!
starting USB...
Bus xhci_pci: probe failed, error -110
No working controllers found
USB is stopped. Please issue 'usb start' first.
starting USB...
Bus xhci_pci: probe failed, error -110
No working controllers found
"Synchronous Abort" handler, esr 0x96000044
elr: 000000000009c1c0 lr : 0000000000091edc (reloc)
elr: 000000003b3751c0 lr : 000000003b36aedc
x0 : 0000000000000031 x1 : 0000000000000051
x2 : 20d51c1000580003 x3 : 409400002fd519f2
x4 : 20d2800800d51e10 x5 : 0000000000000014
x6 : 000000003b3d2e40 x7 : 20d2800800d51df0
x8 : 0000000000000000 x9 : 0000000000000008
x10: 00000000ffffffd0 x11: 000000000000000d
x12: 0000000000000006 x13: 000000000001869f
x14: 000000003af2ceb0 x15: 0000000000000002
x16: 000000003b377278 x17: 33a26b1e70c6e1df
x18: 000000003af38d90 x19: 0000000000000021
x20: 000000003b3d2d00 x21: 000000003afcac50
x22: 000000000000000b x23: 00000000ffffffff
x24: 0000000000000002 x25: 000000003b3e5594
x26: 000000003b3bee1b x27: 000000003b3c74cb
x28: 0000000000000020 x29: 000000003af2c4f0

Code: a9410803 8b130001 b2400273 f9000413 (f9000c62)
Resetting CPU ...

resetting ...

[Paul Larson (pwlars)]

I think this one could be due to the newer u-boot - 2020.10+dfsg-1
It's coming into focal dailies because proposed is enabled there, but I also tried to boot an rpi400 with bionic on an older uboot, then enable proposed and *only* update u-boot. After updating, I get this boot loop there too.

[Dave Jones (waveform)]

Finally managed to replicate this and confirmed it also occurs on the Pi 4 8Gb, but *only when there's no monitor attached*. Stick a monitor on it, it works (which was the reason I having troubles reproducing this). Couldn't replicate it on the CM4, but did find another issue with the latest u-boot being unable to identify the eMMC device.

Anyway, this issue turns out to be caused by the addition of PCI support to the u-boot config; disable that and everything's fine again. I'm currently prepping a trivial patch to the hirsute version (and another one to fix the aforementioned CM4 issue), which we can then SRU if I haven't managed to break something *else* in the meantime :)

[Dave Jones (waveform)]

Attaching a patch against the hirsute version which fixes this (and the aforementioned CM4 failure); I'm prepping separate patches for SRU to bionic, focal, and groovy.

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2020.10+dfsg-1ubuntu6

---------------
u-boot (2020.10+dfsg-1ubuntu6) hirsute; urgency=medium

  * Add d/p/rpi-8gb-pci.patch for Pi400 and Pi4-8Gb support (LP: #1906552)
  * Add d/p/rpi-cm4-sdhci.patch for CM4 eMMC support
  * Add d/p/rpi-maxargs.patch for new Core 18 boot-env (LP: #1910094)
  * Remove redundant d/targets entries

 -- Dave Jones <email address hidden> Thu, 10 Dec 2020 23:44:09 +0000

[Łukasz Zemczak (sil2100)]

Hello Paul, or anyone else affected,

Accepted u-boot into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/u-boot/2020.10+dfsg-1ubuntu0~20.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Paul, or anyone else affected,

Accepted u-boot into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/u-boot/2020.10+dfsg-1ubuntu0~20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Paul, or anyone else affected,

Accepted u-boot into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/u-boot/2020.10+dfsg-1ubuntu0~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Paul Larson (pwlars)]

Tested on focal with rpi400. After updating, I'm now able to boot.

[Paul Larson (pwlars)]

Confirmed bionic boots after the update to proposed on rpi400

[Paul Larson (pwlars)]

Current uc20 candidate image has the updated uboot - this boots fine on both cm4-lite and rpi400

[Paul Larson (pwlars)]

Worked on groovy before the update, but also works after updating to proposed

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2020.10+dfsg-1ubuntu0~20.10.1

---------------
u-boot (2020.10+dfsg-1ubuntu0~20.10.1) groovy; urgency=medium

  * SRU of changes through to 2020.10+dfsg-1ubuntu6 to fix Pi4-8GB & CM4 support,
    and add support for risc-v unleashed board:

  [ Dave Jones ]
  * Add d/p/rpi-8gb-pci.patch for Pi400 and Pi4-8Gb support (LP: #1906552)
  * Add d/p/rpi-cm4-sdhci.patch for CM4 eMMC support
  * Add d/p/rpi-maxargs.patch for new Core 18 boot-env (LP: #1910094)
  * Remove redundant d/targets entries

  [ Dimitri John Ledkov ]
  * Enable u-boot spl for unleashed. LP: #1905274

u-boot (2020.10+dfsg-1ubuntu0~20.10) groovy; urgency=low

  * SRU of 2020.10+dfsg-1 to support Raspberry Pi 4 ethernet (LP: #1903054)

  [ Dave Jones ]
  * Merge from Debian unstable. Remaining changes:
    - Enable Ubuntu support for the Nitrogen6x board (LP: #1838064)
      - Add d/p/ubuntu-nitrogen6q2g-config-tweaks.patch to tweak the
        nitrogen6q2g configs to better fit our Ubuntu usage.
      - Start building the nitrogen6x2g target for u-boot.
    - Enable FIT signing support (LP: #1831942)
      - Enable CONFIG_FIT_SIGNATURE so we can sign FIT images.
      - Add libssl-dev to Build-Depends: to enable crypto functionality.
      - Limit key names to keys within the keydir.
    - d/p/rpi-config-tweaks.patch: Configuration adjustments to the RPi
      configs
    - d/p/rpi-board-dt.patch: use the board's device-tree instead of an
      embedded one

  * Removed obsolete patches/changes:
    - d/p/lzo-to-lzno.patch: use gzip instead of lzo compression for FIT
      images as lzop in Ubuntu is in universe. This should be temporary and in
      the next releases ideally we should follow what Debian does.

u-boot (2020.10+dfsg-1) unstable; urgency=medium

  * New upstream release.

u-boot (2020.10~rc5+dfsg-1) experimental; urgency=medium

  [ Vagrant Cascadian ]
  * New upstream release candidate.
  * debian/control: u-boot-qemu: Set Multi-Arch: foreign.

  [ Uwe Kleine-König ]
  * control: Use https for upstream homepage

u-boot (2020.10~rc3+dfsg-1) experimental; urgency=medium

  * New upstream release candidate.
  * debian/patches: Refresh and remove obsolete patches.
  * [armhf] u-boot-sunxi: Add Bananapi_M2_Ultra (Closes: #962931). Thanks
    to Bernhard Wörner.
  * [arm64] u-boot-rockchip: Update rock-pi-4 .dtb names.
  * u-boot-qemu: Build firmware for qemu ppc e500 (Closes: #966624).

u-boot (2020.07+dfsg-2) unstable; urgency=medium

  [ Vagrant Cascadian ]
  * u-boot-install-rockchip: Update with additional RockPro64 names from
    linux 5.7 device-trees.

  [ Denis Pynkin ]
  * u-boot-rpi: Add rpi_arm64 target for arm64 (Closes: #966078).

  [ Vagrant Cascadian ]
  * debian/patches: Fix reproducibility of mx6cuboxi target.

u-boot (2020.07+dfsg-1) unstable; urgency=medium

  * New upstream release.
  * [armhf] u-boot-omap: Drop omap3_pandora, removed upstream.
  * [arm64] u-boot-amlogic: Install new board-specific documentation.
  * debian/patches: Fix riscv64 support for extlinux.
  * debian/control:
    - Upgrade to debhelper compat 13.
    - Update Build-Depends on arm-trusted-firmware.

u-boot (2020.07~rc4+dfsg-1) ex...

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for u-boot has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2020.10+dfsg-1ubuntu0~20.04.2

---------------
u-boot (2020.10+dfsg-1ubuntu0~20.04.2) focal; urgency=medium

  * SRU of changes through to 2020.10+dfsg-1ubuntu6 to fix Pi4-8GB & CM4 support,
    and add support for risc-v unleashed board:

  [ Dave Jones ]
  * Add d/p/rpi-8gb-pci.patch for Pi400 and Pi4-8Gb support (LP: #1906552)
  * Add d/p/rpi-cm4-sdhci.patch for CM4 eMMC support
  * Add d/p/rpi-maxargs.patch for new Core 18 boot-env (LP: #1910094)
  * Remove redundant d/targets entries

  [ Dimitri John Ledkov ]
  * Enable u-boot spl for unleashed. LP: #1905274

u-boot (2020.10+dfsg-1ubuntu0~20.04.1) focal; urgency=medium

  * SRU of 2020.10+dfsg-1 to support Raspberry Pi 4 (LP: #1903054)
  * Re-added d/p/lzo-to-lzno.patch as lzop is in universe prior to Groovy
  * Updated d/targets for compatibility with focal arm-trusted-firmware
  * Updated d/control for compatibility with focal debhelper
  * Reinstate rpi-config-migration script for focal

  [ Dave Jones ]
  * Merge from Debian unstable. Remaining changes:
    - Enable Ubuntu support for the Nitrogen6x board (LP: #1838064)
      - Add d/p/ubuntu-nitrogen6q2g-config-tweaks.patch to tweak the
        nitrogen6q2g configs to better fit our Ubuntu usage.
      - Start building the nitrogen6x2g target for u-boot.
    - Enable FIT signing support (LP: #1831942)
      - Enable CONFIG_FIT_SIGNATURE so we can sign FIT images.
      - Add libssl-dev to Build-Depends: to enable crypto functionality.
      - Limit key names to keys within the keydir.
    - d/p/rpi-config-tweaks.patch: Configuration adjustments to the RPi
      configs
    - d/p/rpi-board-dt.patch: use the board's device-tree instead of an
      embedded one

  * Removed obsolete patches/changes:
    - d/p/lzo-to-lzno.patch: use gzip instead of lzo compression for FIT
      images as lzop in Ubuntu is in universe. This should be temporary and in
      the next releases ideally we should follow what Debian does.

u-boot (2020.10+dfsg-1) unstable; urgency=medium

  * New upstream release.

u-boot (2020.10~rc5+dfsg-1) experimental; urgency=medium

  [ Vagrant Cascadian ]
  * New upstream release candidate.
  * debian/control: u-boot-qemu: Set Multi-Arch: foreign.

  [ Uwe Kleine-König ]
  * control: Use https for upstream homepage

u-boot (2020.10~rc3+dfsg-1) experimental; urgency=medium

  * New upstream release candidate.
  * debian/patches: Refresh and remove obsolete patches.
  * [armhf] u-boot-sunxi: Add Bananapi_M2_Ultra (Closes: #962931). Thanks
    to Bernhard Wörner.
  * [arm64] u-boot-rockchip: Update rock-pi-4 .dtb names.
  * u-boot-qemu: Build firmware for qemu ppc e500 (Closes: #966624).

u-boot (2020.07+dfsg-2) unstable; urgency=medium

  [ Vagrant Cascadian ]
  * u-boot-install-rockchip: Update with additional RockPro64 names from
    linux 5.7 device-trees.

  [ Denis Pynkin ]
  * u-boot-rpi: Add rpi_arm64 target for arm64 (Closes: #966078).

  [ Vagrant Cascadian ]
  * debian/patches: Fix reproducibility of mx6cuboxi target.

u-boot (2020.07+dfsg-1) unstable; urgency=medium

  * New upstream release.
  * [armhf] u-boot-omap: Drop omap3_pandora, removed upstream.
  * [arm64] ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2020.10+dfsg-1ubuntu0~18.04.2

---------------
u-boot (2020.10+dfsg-1ubuntu0~18.04.2) bionic; urgency=medium

  * SRU of changes through to 2020.10+dfsg-1ubuntu6 to fix Pi4-8GB & CM4 support:

  * Add d/p/rpi-8gb-pci.patch for Pi400 and Pi4-8Gb support (LP: #1906552)
  * Add d/p/rpi-cm4-sdhci.patch for CM4 eMMC support
  * Add d/p/rpi-maxargs.patch for new Core 18 boot-env (LP: #1910094)
  * Remove redundant d/targets entries

u-boot (2020.10+dfsg-1ubuntu0~18.04.1) bionic; urgency=low

  * SRU of 2020.10+dfsg-1 to support Raspberry Pi 4 (LP: #1903054), and
    fix a failure to write issue (LP: #1900693)
  * Re-added d/p/lzo-to-lzno.patch as lzop is in universe prior to Groovy
  * Updated d/targets to remove arm-trusted-firmware references
  * Updated d/control for compatibility with bionic debhelper
  * Reinstate rpi-config-migration script for bionic

  [ Dave Jones ]
  * Merge from Debian unstable. Remaining changes:
    - Enable Ubuntu support for the Nitrogen6x board (LP: #1838064)
      - Add d/p/ubuntu-nitrogen6q2g-config-tweaks.patch to tweak the
        nitrogen6q2g configs to better fit our Ubuntu usage.
      - Start building the nitrogen6x2g target for u-boot.
    - Enable FIT signing support (LP: #1831942)
      - Enable CONFIG_FIT_SIGNATURE so we can sign FIT images.
      - Add libssl-dev to Build-Depends: to enable crypto functionality.
      - Limit key names to keys within the keydir.
    - d/p/rpi-config-tweaks.patch: Configuration adjustments to the RPi
      configs
    - d/p/rpi-board-dt.patch: use the board's device-tree instead of an
      embedded one

  * Removed obsolete patches/changes:
    - d/p/lzo-to-lzno.patch: use gzip instead of lzo compression for FIT
      images as lzop in Ubuntu is in universe. This should be temporary and in
      the next releases ideally we should follow what Debian does.

u-boot (2020.10+dfsg-1) unstable; urgency=medium

  * New upstream release.

u-boot (2020.10~rc5+dfsg-1) experimental; urgency=medium

  [ Vagrant Cascadian ]
  * New upstream release candidate.
  * debian/control: u-boot-qemu: Set Multi-Arch: foreign.

  [ Uwe Kleine-König ]
  * control: Use https for upstream homepage

u-boot (2020.10~rc3+dfsg-1) experimental; urgency=medium

  * New upstream release candidate.
  * debian/patches: Refresh and remove obsolete patches.
  * [armhf] u-boot-sunxi: Add Bananapi_M2_Ultra (Closes: #962931). Thanks
    to Bernhard Wörner.
  * [arm64] u-boot-rockchip: Update rock-pi-4 .dtb names.
  * u-boot-qemu: Build firmware for qemu ppc e500 (Closes: #966624).

u-boot (2020.07+dfsg-2) unstable; urgency=medium

  [ Vagrant Cascadian ]
  * u-boot-install-rockchip: Update with additional RockPro64 names from
    linux 5.7 device-trees.

  [ Denis Pynkin ]
  * u-boot-rpi: Add rpi_arm64 target for arm64 (Closes: #966078).

  [ Vagrant Cascadian ]
  * debian/patches: Fix reproducibility of mx6cuboxi target.

u-boot (2020.07+dfsg-1) unstable; urgency=medium

  * New upstream release.
  * [armhf] u-boot-omap: Drop omap3_pandora, removed upstream.
  * [arm64] u-boot-amlogic: Install new board-specific documentation.
  * debian/patches: Fix riscv64 suppor...