riscv64 images fail to boot in qemu

[Dimitri John Ledkov]

[Impact]

 * u-boot may crash when attempting to boot extlinux.conf which specifies fdtdir; the given u-boot config doesn't specify a dtb filename to load; autodetection tries to make one up using $soc & $board variables; and if one or both of them are not set (as it is the case for qemu) it would crash. Fix this crash by doing validation / checking when quering for $soc & $board variables in autodetectionn.

[Test Plan]

 * Use qemu & uboot produced by the build that fixes this bug report properly and attempt to boot hirsute's riscv64+unmatched image.

 * It should boot correctly without a crash / qemu backtrace.

[Where problems could occur]

 * This patch is bein upstreamed. If one is using external uboot (i.e. provided by some other vendor) it may still crash when trying to boot Ubuntu's riscv64 rootfs or cloud image.

[Other Info]

 * Patch is being reviewed upstream https://lists.denx.de/pipermail/u-boot/2021-May/449176.html

[Dimitri John Ledkov (xnox)]

This is reproducible with any uboot from focal/groovy/hirsute.

[Dimitri John Ledkov (xnox)]

It appears that although fdtdir option from u-boot-menu works on bare metal, it breaks qemu booting.

Thus revert it from now, seeking more details from u-boot upstream.

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot-menu - 4.0.2ubuntu4

---------------
u-boot-menu (4.0.2ubuntu4) hirsute; urgency=medium

  * Revert "Add support for ubuntu kernels location of device trees." as
    it breaks booting riscv64 images with qemu, as the qemu in-ram dtb
    should be used not the one from disk. LP: #1923162

u-boot-menu (4.0.2ubuntu3) hirsute; urgency=medium

  * Add support for ubuntu kernels location of device trees.

 -- Dimitri John Ledkov <email address hidden> Mon, 12 Apr 2021 15:50:07 +0100

[Dimitri John Ledkov (xnox)]

So the issue around providing fdtdir was that qemu did not know what fdt file to attempt loading, and hence was failing to boot.

Now that we dropped fdtdir in extlinux.conf, all riscv64 boards need to use preboot to pick up fdt from the .itb. This was already the case on unleashed and smode, but not on unmatched.

Add USE_PREBOOT config option to unmatched too, thus bringing it inline with unleashed & qemu.

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2021.01+dfsg-3ubuntu6

---------------
u-boot (2021.01+dfsg-3ubuntu6) hirsute; urgency=medium

  * Set USE_PREBOOT on unmatched board too (just like unleashed & qemu),
    otherwise u-boot's fdtfile from itb is not used (i.e. when
    extlinux.conf does not specify fdtdir). LP: #1923162

 -- Dimitri John Ledkov <email address hidden> Tue, 13 Apr 2021 14:52:32 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2021.01+dfsg-3ubuntu8

---------------
u-boot (2021.01+dfsg-3ubuntu8) hirsute; urgency=medium

  * Skip processing fdtdir on qemu-riscv64_smode target, as it crashes the
    riscv qemu VM. LP: #1925267 LP: #1923162

 -- Dimitri John Ledkov <email address hidden> Wed, 21 Apr 2021 01:25:13 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2021.01+dfsg-3ubuntu10

---------------
u-boot (2021.01+dfsg-3ubuntu10) impish; urgency=medium

  * Revert "Skip processing fdtdir on qemu-riscv64_smode target, as it crashes the
    riscv qemu VM."
  * Apply upstreamed patch "cmd: pxe_utils: sysboot: fix crash if either
    board or soc are not set." This fixes a crash booting any
    extlinux.conf with fdtdir specified, and dtbfile not set and
    "soc"/"board" not set in the environment LP: #1923162

 -- Dimitri John Ledkov <email address hidden> Fri, 14 May 2021 16:54:50 +0100

[Brian Murray (brian-murray)]

Hello Dimitri, or anyone else affected,

Accepted u-boot into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/u-boot/2021.01+dfsg-3ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[William Wilson (jawn-smith)]

This patch backports the changes from hirsute in u-boot-menu to focal.

[Łukasz Zemczak (sil2100)]

Ok, leaving a trace here as well, regarding the u-boot upload in the focal Unapproved queue, other than direct messages: currently in the .changes file there are 3 bugs attached, 2 of which do not have SRU templates, so we'd need that + since this is a backport, I feel a bit sad that there's no backport tracking bug (since it's quite a big u-boot bump). So I'd recommend either modifying one of the bugs into a 'backport latest u-boot to focal' with a proper test case of all supported platforms, or creating a new one, attaching it to the changelog and then re-uploading.
Do we still use u-boot for focal raspi images? I guess we do? If yest, we'd need to have testing of the pi images as well in the testcase of this SRU tracking bug.

Also, quickly browsing through the debdiff of the new version I see this line in debian/control:
+Breaks: flash-kernel (<< 3.104)
...but focal doesn't have flash-kernel 3.104. So this would need the new flash-kernel backported? Or at least some changes done + changing the breaks.

I guess I'll reject the upload for now. We can always re-upload or fetch it from the rejected queue.

[Łukasz Zemczak (sil2100)]

An upload of u-boot to focal-proposed has been rejected from the upload queue for the following reason: "Please see https://bugs.launchpad.net/ubuntu/+source/u-boot/+bug/1923162/comments/13 for the rationale. Might need some more work or re-upload.".

[Dave Jones (waveform)]

@sil2100 sorry - that's my fault for skimming this (and the breaks: bit too -- that's actually not necessary as the backport re-instates the postinst installing the binaries to the boot partition; the breaks was there as later releases ditched the postinst and passed the responsibility for copying u-boot to flash-kernel). I'll work with jawn-smith to get this fixed up.

[Dimitri John Ledkov (xnox)]

Re: u-boot-menu backport, imho debian/changelog should have documented that debhelper-compat version is lowered from 13 to 12, to make it build on focal. But also it is self-explanatory (i did try to get debhelper 13 into focal before it released, but that upload got rejected).

Otherwise no changes from hirsute version that is stable and works. The changes in impish are minor and not needed for hardware support yet.

[Dimitri John Ledkov (xnox)]

re: u-boot-menu backport, imho the debian/changelog should be merged and focal entries should not be lost.

[Dimitri John Ledkov (xnox)]

@jawn-smith https://paste.ubuntu.com/p/hjqFz46frm/ is what i'm sponsoring, after fix ups.

[Brian Murray (brian-murray)]

Hello Dimitri, or anyone else affected,

Accepted u-boot into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/u-boot/2021.01+dfsg-3ubuntu0~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Dimitri, or anyone else affected,

Accepted u-boot-menu into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/u-boot-menu/4.0.2ubuntu5~20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dimitri John Ledkov (xnox)]

$ dpkg-query -W u-boot-qemu
u-boot-qemu 2021.01+dfsg-3ubuntu9.1

$ qemu-system-riscv64 -machine virt -nographic -m 2048 -smp 4 -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf -kernel /usr/lib/u-boot/qemu-riscv64_smode/uboot.elf -device virtio-net-device,netdev=eth0 -netdev user,id=eth0 -drive file=ubuntu-21.04-preinstalled-server-riscv64+unmatched.img,format=raw,if=virtio

OpenSBI v0.9
   ____ _____ ____ _____
  / __ \ / ____| _ \_ _|
 | | | |_ __ ___ _ __ | (___ | |_) || |
 | | | | '_ \ / _ \ '_ \ \___ \| _ < | |
 | |__| | |_) | __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name : riscv-virtio,qemu
Platform Features : timer,mfdeleg
Platform HART Count : 4
Firmware Base : 0x80000000
Firmware Size : 124 KB
Runtime SBI Version : 0.2

Domain0 Name : root
Domain0 Boot HART : 0
Domain0 HARTs : 0*,1*,2*,3*
Domain0 Region00 : 0x0000000080000000-0x000000008001ffff ()
Domain0 Region01 : 0x0000000000000000-0xffffffffffffffff (R,W,X)
Domain0 Next Address : 0x0000000080200000
Domain0 Next Arg1 : 0x0000000082200000
Domain0 Next Mode : S-mode
Domain0 SysReset : yes

Boot HART ID : 0
Boot HART Domain : root
Boot HART ISA : rv64imafdcsu
Boot HART Features : scounteren,mcounteren,time
Boot HART PMP Count : 16
Boot HART PMP Granularity : 4
Boot HART PMP Address Bits: 54
Boot HART MHPM Count : 0
Boot HART MHPM Count : 0
Boot HART MIDELEG : 0x0000000000000222
Boot HART MEDELEG : 0x000000000000b109

U-Boot 2021.01+dfsg-3ubuntu9.1 (May 14 2021 - 15:54:50 +0000)

CPU: rv64imafdcsu
Model: riscv-virtio,qemu
DRAM: 2 GiB
In: uart@10000000
Out: uart@10000000
Err: uart@10000000
Net: eth0: virtio-net#0
Hit any key to stop autoboot: 0

Device 0: 1af4 VirtIO Block Device
            Type: Hard Disk
            Capacity: 3584.0 MB = 3.5 GB (7340032 x 512)
... is now current device
Scanning virtio 0:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
743 bytes read in 1 ms (725.6 KiB/s)
U-Boot menu
1: Ubuntu 21.04 5.11.0-1007-generic
2: Ubuntu 21.04 5.11.0-1007-generic (rescue target)
Enter choice: 1: Ubuntu 21.04 5.11.0-1007-generic
Retrieving file: /boot/initrd.img-5.11.0-1007-generic
182268553 bytes read in 2852 ms (60.9 MiB/s)
Retrieving file: /boot/vmlinuz-5.11.0-1007-generic
27355136 bytes read in 6 ms (4.2 GiB/s)
append: root=LABEL=cloudimg-rootfs ro earlycon
Retrieving file: /lib/firmware/5.11.0-1007-generic/device-tree/qemu-riscv.dtb
Failed to load '/lib/firmware/5.11.0-1007-generic/device-tree/qemu-riscv.dtb'
Skipping l0 for failure retrieving fdt
2: Ubuntu 21.04 5.11.0-1007-generic (rescue target)
Retrieving file: /boot/initrd.img-5.11.0-1007-generic
182268553 bytes read in 40 ms (4.2 GiB/s)
Retrieving file: /boot/vmlinuz-5.11.0-1007-generic
27355136 bytes read in 4 ms (6.4 GiB/s)
append: root=LABEL=cloudimg-rootfs ro earlycon single
Retrieving file: /lib/firmware/5.11.0-1007-generic/device-tree...

[Dimitri John Ledkov (xnox)]

Trying to see if impish u-boot is also broken at the moment.

[Dimitri John Ledkov (xnox)]

Impish is not affected, as it still skips fdtdir under qemu completely.

[Dimitri John Ledkov (xnox)]

Oh https://launchpad.net/ubuntu/+source/u-boot/2021.01+dfsg-3ubuntu9.1 is an orphan, as it did not make it into impish.

[William Wilson (jawn-smith)]

The verification passed for focal. The daily images are building with u-boot-menu-4.0.2ubuntu5~20.04.1. I have successfully booted these daily images in qemu and Unmatched hardware.

Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.8.0-29-generic riscv64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

  System information as of Tue Jul 20 17:31:58 UTC 2021

  System load: 0.89
  Usage of /: 82.1% of 3.22GB
  Memory usage: 9%
  Swap usage: 0%
  Processes: 114
  Users logged in: 0
  IPv4 address for eth0: 10.0.2.15
  IPv6 address for eth0: fec0::5054:ff:fe12:3456

 * Super-optimized for small spaces - read how we shrank the memory
   footprint of MicroK8s to make it the smallest full K8s around.

   https://ubuntu.com/blog/microk8s-memory-optimisation

0 updates can be applied immediately.

Last login: Fri Jul 16 13:46:13 UTC 2021 on ttyS0
ubuntu@ubuntu:~$ apt-cache policy u-boot-menu
u-boot-menu:
  Installed: 4.0.2ubuntu5~20.04.1
  Candidate: 4.0.2ubuntu5~20.04.1
  Version table:
 *** 4.0.2ubuntu5~20.04.1 500
        500 http://ports.ubuntu.com/ubuntu-ports focal-proposed/main riscv64 Packages
        100 /var/lib/dpkg/status
     4.0.1ubuntu0.1 500
        500 http://ports.ubuntu.com/ubuntu-ports focal-updates/universe riscv64 Packages
     4.0.1 500
        500 http://ports.ubuntu.com/ubuntu-ports focal/universe riscv64 Packages

[Łukasz Zemczak (sil2100)]

Hey William! Any reason why the verification-failed-hirsute tag has been removed? Did you discuss with Dimitri and all is good already, or is the update still incorrect for hirsute?

[Łukasz Zemczak (sil2100)]

Per my discussion with Dimitri on IRC, I'm switching the tag to verification-failed for hirsute again and then proceeding with dropping the hirsute-proposed u-boot package.

[Łukasz Zemczak (sil2100)]

The version of u-boot in the proposed pocket of Hirsute that was purported to fix this bug report has been removed because one or more bugs that were to be fixed by the upload have failed verification and been in this state for more than 10 days.

[William Wilson (jawn-smith)]

sil2100 sorry for the delayed response. I originally thought that the verification-failed-hirsute tag was left over from a previous regression we had in this package. I will investigate what is needed in hirsute.

[Brian Murray (brian-murray)]

I'm sorry to be a bother here but I don't understand how u-boot failed verification for hirsute but passed for focal when what are presumably the same patches exist in both versions of the package. How could this happen?

[Dimitri John Ledkov (xnox)]

On Tue, 27 Jul 2021, 05:45 Brian Murray, <email address hidden> wrote:

> I'm sorry to be a bother here but I don't understand how u-boot failed
> verification for hirsute but passed for focal when what are presumably
> the same patches exist in both versions of the package. How could this
> happen?

Focal is on par with hirsute-release.
Hirsute-proposed had newer version of patches that were incomplete and did
not make it into impish either. Hence the hirsute-proposed was an orphan
upload, neither forward ported nor backported.

[Brian Murray (brian-murray)]

The changes in ubuntu9.1 did make it into impish in ubuntu10.

https://launchpadlibrarian.net/538731561/u-boot_2021.01+dfsg-3ubuntu9_2021.01+dfsg-3ubuntu10.diff.gz

However, it looks like they were later removed from impish.

https://launchpadlibrarian.net/539465636/u-boot_2021.01+dfsg-3ubuntu10_2021.01+dfsg-4ubuntu1.diff.gz

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2021.01+dfsg-3ubuntu0~20.04.1

---------------
u-boot (2021.01+dfsg-3ubuntu0~20.04.1) focal; urgency=medium

  * SRU of 2021.01+dfsg-3 from hirsute to enable booting on HiFive Unmatched
    LP: #1934791, LP: #1925267, LP: #1923162

  [ Dimitri John Ledkov ]
  * Import meta-sifive u-boot patches.
  * Enable sifive_hifive_unmatched_fu740 target.
  * Build-depend on opensbi with fu740 errata fix.
  * Set default FDT files for the sifive boards.
  * Set USE_PREBOOT on unmatched board too (just like unleashed & qemu),
    otherwise u-boot's fdtfile from itb is not used (i.e. when
    extlinux.conf does not specify fdtdir). LP: #1923162
  * Unapply unmatched patches, whilst building unleashed platform. Fixes
    failure to boot from sd-card on Unleashed. LP: #1924761
  * sifive-unleashed-default-fdt-files.patch: split into unleashed &
    unmatched separate patches, for ease of upstreaming to meta-sifive and
    u-boot upstreams. Also this ensures that unleashed target is built
    with fdtfile= set, as unleashed target unapplies lots of patches.
  * Apply upstreamed patch "cmd: pxe_utils: sysboot: fix crash if either
    board or soc are not set." This fixes a crash booting any
    extlinux.conf with fdtdir specified, and dtbfile not set and
    "soc"/"board" not set in the environment LP: #1923162

  [ William 'jawn-smith' Wilson ]
  * Re-added d/p/lzo-to-lzno.patch as lzop is in universe prior to Groovy
  * Updated d/targets for compatibility with focal arm-trusted-firmware
  * Updated d/control for compatibility with focal debhelper
  * Reinstate rpi-config-migration script for focal
  * Prevent stripping of qemu binaries under focal where strip doesn't
    recognize the format

  [ Dave Jones ]
  * In d/control update opensbi dependency for focal

 -- William 'jawn-smith' Wilson <email address hidden> Fri, 25 Jun 2021 18:21:47 +0000

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for u-boot has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot-menu - 4.0.2ubuntu5~20.04.1

---------------
u-boot-menu (4.0.2ubuntu5~20.04.1) focal; urgency=medium

  * SRU changes from hirsute to enable booting on the HiFive Unmatched
    LP: #1925267 LP: #1923162

u-boot-menu (4.0.2ubuntu5) hirsute; urgency=medium

  * Go back to generating fdtdir option in extlinux.conf, instead patch
    u-boot qemu-riscv64_smode target to not process or load fdtdir
    option. Currently the dtb for riscv64 unmatched board in kernel &
    u-boot are _unmatched_ meaning u-boot must use its dtb, and kernel
    must use its dtb. Hence fdtdir option in extlinux.conf is
    required. LP: #1925267 LP: #1923162

u-boot-menu (4.0.2ubuntu4) hirsute; urgency=medium

  * Revert "Add support for ubuntu kernels location of device trees." as
    it breaks booting riscv64 images with qemu, as the qemu in-ram dtb
    should be used not the one from disk. LP: #1923162

u-boot-menu (4.0.2ubuntu3) hirsute; urgency=medium

  * Add support for ubuntu kernels location of device trees.

u-boot-menu (4.0.2ubuntu2) hirsute; urgency=medium

  * Make default U_BOOT_PARAMETERS to be "ro earlycon". If one is using
    embedded boards, most likely they want verbosity.

u-boot-menu (4.0.2ubuntu1) hirsute; urgency=medium

  * u-boot-update: use PRETTY_NAME from os-release as menu label.

u-boot-menu (4.0.2) unstable; urgency=medium

  [ Vagrant Cascadian ]
  * install example script for syncing .dtb files
    for systems with a split /boot partition
  * fix comment handling in in /etc/fstab

  [ Jonas Smedegaard ]
  * declare compliance with Debian Policy 4.5.0
  * use debhelper compatibility level 13 (not 10);
    build-depend on debhelper-compat (not debhelper)
  * copyright: extend coverage

 -- William 'jawn-smith' Wilson <email address hidden> Thu, 24 Jun 2021 16:04:00 -0500