Failed to boot

[Yung Shen]

[Impact]

 * Unable to boot into system after upgrade to 4.4.0-1030.37

[Test Case]

 * fresh install ubuntu-16.04.1-preinstalled-server-armhf+raspi2.img.xz

 * enable -proposed and 'apt full-upgrade'

[Regression Potential]

 * red led still on, but few green led blinks then stay at the black screen(hdmi on)

[Other Info]

 * restore the working boot partition backup image could recover the situation

[Solution]

* The latest X/raspi2 kernel in -proposed blows the space the
bootloader reserves for the DTB, making the board unbootable (it only
happens with classic rpi2 images created by the Foundation, Snappy is
fine)

* Ppa:p-pisati/misc has these 3 new packages that fix it:

 - linux-firmware-raspi2 - contains the latest Broadom fw release, and
   upon installation checks that /boot/firmware/config.txt contains the
   new DTB memory location, and if not, it does the necessary
   modification on behalf of the user

 - flash-kernel - modifies uboot.env to find the DTB in the new memory location

 - uboot - update uboot to work with the new boot firmware (aka skip
   serial initialization if already done by the fw), contains the
   mkknlimg required when installing a new u-boot binary and correctly
   installs the uboot binary in the vfat partition

[How to test it]

* Install a fresh foundation image, add my misc ppa and the xenial -proposed repository, apt-get update && dist-upgrade, once the board completes the upgrade reboot it.

[Paolo Pisati (p-pisati)]

Can you connect a serial cable to the board? You need a TTL serial and it connects like this:

http://workshop.raspberrypiaustralia.com/usb/ttl/connecting/2014/08/31/01-connecting-to-raspberry-pi-via-usb/

then on you pc:

screen /dev/ttyUSB0 115200

then plug the power in the board, and you should see the bootloader:

U-Boot 2015.07 (Aug 31 2015 - 15:50:21 +0200)

DRAM: 944 MiB
WARNING: Caches not enabled
RPI 2 Model B
MMC: bcm2835_sdhci: 0
reading uboot.env
In: serial
Out: lcd
Err: lcd
Net: Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot: 0

ubuntu@raspi2:~$ uname -a
Linux raspi2 4.4.0-1029-raspi2 #36-Ubuntu SMP Wed Oct 19 14:38:48 UTC 2016 armv7l armv7l armv7l GNU/Linux
ubuntu@raspi2:~$ ls -la /boot/firmware/*.dtb
-rwxr-xr-x 1 root root 13430 Oct 26 15:18 /boot/firmware/bcm2709-rpi-2-b.dtb

this on -1029.36, while if i update to -1030.37 the new dtb is bigger and it blows the space at the beginning of the memory:

ubuntu@raspi2:~$ ls -la /boot/firmware/*.dtb
-rwxr-xr-x 1 root root 15356 Oct 26 18:18 /boot/firmware/bcm2709-rpi-2-b.dtb

and i bet on console you will end up with something like this:

U-Boot 2015.07 (Aug 31 2015 - 15:50:21 +0200)

DRAM: 944 MiB
WARNING: Caches not enabled
RPI 2 Model B
MMC: bcm2835_sdhci: 0
reading uboot.env
In: serial
Out: lcd
Err: lcd
Net: Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot: 0
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
No FDT memory address configured. Please configure
the FDT address via "fdt addr <address>" command.
Aborting!
6682376 bytes read in 12755 ms (510.7 KiB/s)
Kernel image @ 0x1000000 [ 0x000000 - 0x65f708 ]
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
U-Boot>

is that the case?

[Yung Shen (kaxing)]

@p-pisati hey, i think you are correct, this is what I get:

U-Boot 2016.01+dfsg1-2ubuntu1 (Feb 12 2016 - 05:55:38 +0000)

DRAM: 944 MiB
WARNING: Caches not enabled
RPI 2 Model B (0xa21041)
MMC: bcm2835_sdhci: 0
reading uboot.env
In: serial
Out: serial
Err: serial
Net: Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot: 0
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
No FDT memory address configured. Please configure
the FDT address via "fdt addr <address>" command.
Aborting!
reading vmlinuz
6682376 bytes read in 9273 ms (703.1 KiB/s)
reading initrd.img
12675295 bytes read in 17685 ms (699.2 KiB/s)
Kernel image @ 0x1000000 [ 0x000000 - 0x65f708 ]
ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
U-Boot>

[Paolo Pisati (p-pisati)]

In ppa:p-pisati/dtb-lp1636838 are the packages for all the releases affected: Xenial, Yakkety and Zesty.

[Yung Shen (kaxing)]

removed the verification-done-xenial tag, since I'm only tested it with ppa:p-pisati/misc.

[Launchpad Janitor (janitor)]

This bug was fixed in the package flash-kernel - 3.0~rc.4ubuntu65

---------------
flash-kernel (3.0~rc.4ubuntu65) zesty; urgency=medium

  * Adjust rpi2/3 dtb address from 0x100 to 0x02000000 (LP: #1636838)

 -- Paolo Pisati <email address hidden> Wed, 09 Nov 2016 15:41:50 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware-raspi2 - 1.20161020

---------------
linux-firmware-raspi2 (1.20161020) zesty; urgency=medium

  * Update to release 1.20161020
  * Check that config.txt sets the $device_tree_address, or forcefully set it to 0x02000000
    (LP: #1636838)

 -- Paolo Pisati <email address hidden> Fri, 28 Oct 2016 14:41:31 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2016.03+dfsg1-6ubuntu2

---------------
u-boot (2016.03+dfsg1-6ubuntu2) zesty; urgency=medium

  * From upstream u-boot: (LP: #1636838)
    - debian/patches/rpi-import-mkknlimg.patch: import tools/mkknlimg from
      Xenial/raspi2 - scripts/mkknlimg @ 83a3ebb
    - debian/u-boot-rpi.postinst: pass u-boot.bin through mkknlimg before
      installing it as /boot/firmware/uboot.bin
    - debian/patches/serial-pl01x-Add-support-for-devices-with-the-rate-p.patch:
      Skip serial clock initialization when it's done by the firmware.

 -- Paolo Pisati <email address hidden> Wed, 09 Nov 2016 17:09:29 +0200

[Chris J Arges (arges)]

Hello Yung, or anyone else affected,

Accepted u-boot into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/u-boot/2016.01+dfsg1-2ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Yung, or anyone else affected,

Accepted flash-kernel into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/flash-kernel/3.0~rc.4ubuntu62.1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Hello Yung, or anyone else affected,

Accepted linux-firmware-raspi2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware-raspi2/1.20161020-0ubuntu1~0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Yung, or anyone else affected,

Accepted u-boot into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/u-boot/2016.03+dfsg1-6ubuntu2~1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Yung, or anyone else affected,

Accepted linux-firmware-raspi2 into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/linux-firmware-raspi2/1.20161020-0ubuntu1~1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Yung, or anyone else affected,

Accepted flash-kernel into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/flash-kernel/3.0~rc.4ubuntu64.1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Paolo Pisati (p-pisati)]

This is to get the three packages (flash-kernel, linux-raspi2-firmware and uboot) moving, while the kernel will land later.

[Robie Basak (racb)]

I don't understand what verification has been done here, or if there is any special process required, so I'm skipping releasing it today.

[Paolo Pisati (p-pisati)]

No, there's no special process required:

-take the rpi2 image: http://cdimage.ubuntu.com/ubuntu/releases/16.04/release/ubuntu-16.04-preinstalled-server-armhf+raspi2.img.xz

-enable proposed, and do a dist upgrade and you'll get all the above packages

-reboot

-does it come back? yes, fine -> verification-done

[Paolo Pisati (p-pisati)]

Ok, here's a more verbose version of my testing procedure:

-take the rpi2 image: http://cdimage.ubuntu.com/ubuntu/releases/16.04/release/ubuntu-16.04-preinstalled-server-armhf+raspi2.img.xz

-after a fresh install, these are the pkgs version:

ubuntu@ubuntu:~$ dpkg -l | grep -e uboot -e flash-kernel -e firmware -e linux-raspi2
ii flash-kernel 3.0~rc.4ubuntu62 armhf utility to make certain embedded devices bootable
ii linux-firmware-raspi2 1.20151118+b70b451-0ubuntu1 armhf RaspberryPi2 GPU firmware and bootloaders
ii linux-raspi2 4.4.0.1009.9 armhf Complete Linux kernel for the BCM2709 architecture.

-enable proposed, apt-get update and apt-get dist-upgrade:

ubuntu@ubuntu:~$ dpkg -l | grep -e uboot -e flash-kernel -e firmware -e linux-raspi2
ii flash-kernel 3.0~rc.4ubuntu62.1.1 armhf utility to make certain embedded devices bootable
ii linux-firmware-raspi2 1.20161020-0ubuntu1~0.1 armhf RaspberryPi2 GPU firmware and bootloaders
ii linux-raspi2 4.4.0.1038.37 armhf Complete Linux kernel for the BCM2709 architecture.

same goes for 'apt-get upgrade' instead of 'dist-upgrade', except this time you won't get the newer kernel.

[Paolo Pisati (p-pisati)]

And for uboot:

ubuntu@ubuntu:~$ dpkg -l | grep u-boot
ii u-boot-rpi:armhf 2016.01+dfsg1-2ubuntu2 armhf A boot loader for Raspberry PI systems
ii u-boot-tools 2016.01+dfsg1-2ubuntu2 armhf companion tools for Das U-Boot bootloader

[Paolo Pisati (p-pisati)]

And here is the same for Yakkety:

ubuntu@ubuntu:~$ dpkg -l | grep -e uboot -e flash-kernel -e firmware -e linux-raspi2 -e u-boot
ii flash-kernel 3.0~rc.4ubuntu64 armhf utility to make certain embedded devices bootable

ii linux-firmware-raspi2 1.20151118+b70b451-0ubuntu1 armhf RaspberryPi2 GPU firmware and bootloaders

ii linux-raspi2 4.8.0.1013.16 armhf Complete Linux kernel for the BCM2709 architecture.

ii u-boot-rpi:armhf 2016.03+dfsg1-6ubuntu1 armhf A boot loader for Raspberry PI systems
ii u-boot-tools 2016.03+dfsg1-6ubuntu1 armhf companion tools for Das U-Boot bootloader

after a dist-upgrade:

ubuntu@ubuntu:~$ dpkg -l | grep -e uboot -e flash-kernel -e firmware -e linux-raspi2 -e u-boot
ii flash-kernel 3.0~rc.4ubuntu64.1.1 armhf utility to make certain embedded devices bootable

ii linux-firmware-raspi2 1.20161020-0ubuntu1~1.1 armhf RaspberryPi2 GPU firmware and bootloaders

ii linux-raspi2 4.8.0.1021.24 armhf Complete Linux kernel for the BCM2709 architecture.

ii u-boot-rpi:armhf 2016.03+dfsg1-6ubuntu2~1.1 armhf A boot loader for Raspberry PI systems
ii u-boot-tools 2016.03+dfsg1-6ubuntu2~1.1 armhf companion tools for Das U-Boot bootloader

[Taihsiang Ho (tai271828)]

proposed Xenial looks good. I could boot into the system with the proposed stack.

The detailed of the system info:

=================================================================================================
Kernel: 4.4.0-1038-raspi2
Ubuntu 16.04.1 LTS
=================================================================================================
### dpkg -l | grep -e uboot -e flash-kernel -e firmware -e linux-raspi2 -e u-boot
ii flash-kernel 3.0~rc.4ubuntu62.1.1 armhf utility to make certain embedded devices bootable
ii linux-firmware 1.157.6 all Firmware for Linux kernel drivers
ii linux-firmware-raspi2 1.20161020-0ubuntu1~0.1 armhf RaspberryPi2 GPU firmware and bootloaders
ii linux-raspi2 4.4.0.1038.37 armhf Complete Linux kernel for the BCM2709 architecture.
ii linux-raspi2-headers-4.4.0-1009 4.4.0-1009.10 armhf Header files related to Linux kernel version 4.4.0
ii linux-raspi2-headers-4.4.0-1034 4.4.0-1034.41 armhf Header files related to Linux kernel version 4.4.0
ii linux-raspi2-headers-4.4.0-1038 4.4.0-1038.45 armhf Header files related to Linux kernel version 4.4.0
ii u-boot-rpi:armhf 2016.01+dfsg1-2ubuntu2 armhf A boot loader for Raspberry PI systems
ii u-boot-tools 2016.01+dfsg1-2ubuntu2 armhf companion tools for Das U-Boot bootloader
=================================================================================================

=================================================================================================
Please refer to this URL to access full dpkg info.
http://paste.ubuntu.com/23637373/
=================================================================================================

[Paolo Pisati (p-pisati)]

And the upgrade from Xenial (vannila or dist-upgrade) to Yakkety is fine too - i think i covered all the possible upgrade paths combination, please release these packages.

[Launchpad Janitor (janitor)]

This bug was fixed in the package flash-kernel - 3.0~rc.4ubuntu62.1.1

---------------
flash-kernel (3.0~rc.4ubuntu62.1.1) xenial; urgency=medium

  * Adjust rpi2 dtb address from 0x100 to 0x02000000 (LP: #1636838)

 -- Paolo Pisati <email address hidden> Fri, 28 Oct 2016 13:39:00 +0200

[Adam Conrad (adconrad)]

The verification of the Stable Release Update for flash-kernel has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2016.01+dfsg1-2ubuntu2

---------------
u-boot (2016.01+dfsg1-2ubuntu2) xenial; urgency=medium

  * From upstream u-boot: (LP: #1636838)
    - debian/patches/rpi-import-mkknlimg.patch: import tools/mkknlimg from
      Xenial/raspi2 - scripts/mkknlimg @ 83a3ebb
    - debian/u-boot-rpi.postinst: pass u-boot.bin through mkknlimg before
      installing it as /boot/firmware/uboot.bin
    - debian/patches/serial-pl01x-Add-support-for-devices-with-the-rate-p.patch:
      Skip serial clock initialization when it's done by the firmware.

 -- Paolo Pisati <email address hidden> Fri, 28 Oct 2016 12:12:25 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package flash-kernel - 3.0~rc.4ubuntu64.1.1

---------------
flash-kernel (3.0~rc.4ubuntu64.1.1) yakkety; urgency=medium

  * Adjust rpi2/3 dtb address from 0x100 to 0x02000000 (LP: #1636838)

 -- Paolo Pisati <email address hidden> Wed, 09 Nov 2016 15:41:50 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package u-boot - 2016.03+dfsg1-6ubuntu2~1.1

---------------
u-boot (2016.03+dfsg1-6ubuntu2~1.1) yakkety; urgency=medium

  * From upstream u-boot: (LP: #1636838)
    - debian/patches/rpi-import-mkknlimg.patch: import tools/mkknlimg from
      Xenial/raspi2 - scripts/mkknlimg @ 83a3ebb
    - debian/u-boot-rpi.postinst: pass u-boot.bin through mkknlimg before
      installing it as /boot/firmware/uboot.bin
    - debian/patches/serial-pl01x-Add-support-for-devices-with-the-rate-p.patch:
      Skip serial clock initialization when it's done by the firmware.

 -- Paolo Pisati <email address hidden> Wed, 09 Nov 2016 17:09:29 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware-raspi2 - 1.20161020-0ubuntu1~1.1

---------------
linux-firmware-raspi2 (1.20161020-0ubuntu1~1.1) yakkety; urgency=medium

  * Update to release 1.20161020
  * Check that config.txt sets the $device_tree_address, or forcefully set it to 0x02000000
    (LP: #1636838)

 -- Paolo Pisati <email address hidden> Fri, 28 Oct 2016 14:41:31 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-firmware-raspi2 - 1.20161020-0ubuntu1~0.1

---------------
linux-firmware-raspi2 (1.20161020-0ubuntu1~0.1) xenial; urgency=medium

  * Update to release 1.20161020
  * Check that config.txt sets the $device_tree_address, or forcefully set it to 0x02000000
    (LP: #1636838)

 -- Paolo Pisati <email address hidden> Fri, 28 Oct 2016 14:41:31 +0200

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Chris Carlin (crcarlin)]

Booting with -proposed isn't fixed on Raspberry Pi 3

I don't know if this was expected to be fixed on that device, but I at least wanted to save someone else from trying.

Should I file a separate bug about it, or does the unsupported warning mean it's not reportable?

[Simon Kropf (simon-kropf)]

This bug has already been marked as fixed & Ubuntu Stable Release Updates Team has already been unsubscribed as it is written here: https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1636838/comments/23

New bug report at https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1680605

[Simon Kropf (simon-kropf)]

This bug has already been marked as fixed & Ubuntu Stable Release Updates Team has already been unsubscribed as it is written here: https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1636838/comments/23

New bug report at https://bugs.launchpad.net/ubuntu/+source/linux-raspi2/+bug/1680605