LAVA lacks ability to test bootloader/u-boot

[Alexander Sack]

This is a follow up bug for discussion about u-boot testing in bug 833832. Filing against lava lab project as this seems to involve hardware plumming.

Background: The bootloader currently needs to be rock solid for boards in rack. Basically, if the bootloader fails the board is gone from the pool and requires manual maintenance. That's not scalable and until we have a way to automate replacing bootloader in a failsafe way we cannot offer bootloader testing in the lab.

Please discuss.

[warmcat (andy-warmcat)]

The difficulty is that as it stands, Panda naturally always wants to boot into the X-loader / U-Boot it finds on the SD Card.

If we teleported in a broken X-loader / U-Boot because that's what we wanted to have tested, then we kind of brick the Panda until someone comes and rewrites the SD by hand or swaps in a good card. That's not really acceptable.

One solution to this is to provide a fake SD card PCB which comes out into two SD sockets. One contains a stable reference boot image and initrd, the other contains the test image. By one out-of-band means or another, we allow the test system to select which of the SD cards is visible to the Panda after reset.

If it selects the reference SD, the initrd is loaded and the SD card switched back by timeout to the test image SD, From the initrd userland, the test environment can nuke and prepare the test image SD for the test action over the network. Since it's an initrd, the test image SD is not mounted so you have full control over it.

If it selects the test image SD, then that just boots normally, if it's workable the tests are run otherwise the test session times out and it can be rebooted into the reference SD environment again to prepare for the next test image. That's the case even if, eg, x-loader is corrupted or missing in the test SD image.

[Michael Hudson-Doyle (mwhudson)]

On Tue, 30 Aug 2011 04:15:29 -0000, warmcat <email address hidden> wrote:
> The difficulty is that as it stands, Panda naturally always wants to
> boot into the X-loader / U-Boot it finds on the SD Card.
>
> If we teleported in a broken X-loader / U-Boot because that's what we
> wanted to have tested, then we kind of brick the Panda until someone
> comes and rewrites the SD by hand or swaps in a good card. That's not
> really acceptable.
>
> One solution to this is to provide a fake SD card PCB which comes out
> into two SD sockets. One contains a stable reference boot image and
> initrd, the other contains the test image. By one out-of-band means or
> another, we allow the test system to select which of the SD cards is
> visible to the Panda after reset.

Do you know of such a thing? If one exists, it would be mightily useful
for all sorts of tricks.

I believe most boards we are interested in support ways of supplying the
boot loader dynamically, for instance OMAP4 has something called "USB
Peripheral Boot Mode". I have no idea how this works in practice though
:-) Zygmunt has been thinking about this, I believe.

Cheers,
mwh

[warmcat (andy-warmcat)]

Here's where I got with it back in March.

In this version it has a short flat cable connecting a uSD header with a larger board that intercepts the serial console cable and switches power to the board.

After some more discussion IIRC it became clear we could dispense with using the handshake from the serial console if we used duration of power interruption as the signal whether to come up in the recovery / reference SD or not. So if you turn off power for >10s, say, you guarantee the next boot comes up in recovery SD. For less than 5s say, you guarantee to boot in the test SD.

It was also discussed changing the power arrangements to use SD card socket power, which is much more convenient. At the moment it includes a MOSFET to switch the card's main power but that wouldn't be needed then. The downside is that a malicious test image could switch the SD Card power rail to select the reference / recovery SD.

I think these things are quite possible to build, especially as I am now in Taipei.

[warmcat (andy-warmcat)]

USB peripheral boot mode has some advantages and disadvantages. It should be possible to inject a recovery image using it. However you need a smart USB host for each board then; and it's specific to Panda, whereas the dual SD scheme will work on any SD-boot board.

[Zygmunt Krynicki (zyga)]

W dniu 30.08.2011 07:01, Michael Hudson-Doyle pisze:
> On Tue, 30 Aug 2011 04:15:29 -0000, warmcat<email address hidden> wrote:
>> The difficulty is that as it stands, Panda naturally always wants to
>> boot into the X-loader / U-Boot it finds on the SD Card.
>>
>> If we teleported in a broken X-loader / U-Boot because that's what we
>> wanted to have tested, then we kind of brick the Panda until someone
>> comes and rewrites the SD by hand or swaps in a good card. That's not
>> really acceptable.
>>
>> One solution to this is to provide a fake SD card PCB which comes out
>> into two SD sockets. One contains a stable reference boot image and
>> initrd, the other contains the test image. By one out-of-band means or
>> another, we allow the test system to select which of the SD cards is
>> visible to the Panda after reset.
> Do you know of such a thing? If one exists, it would be mightily useful
> for all sorts of tricks.
>
> I believe most boards we are interested in support ways of supplying the
> boot loader dynamically, for instance OMAP4 has something called "USB
> Peripheral Boot Mode". I have no idea how this works in practice though
> :-) Zygmunt has been thinking about this, I believe.

Omap3, omap4 and imx53 can 100% boot from USB. The only thing needed is
to sit down and automate the hell out of it. There may be caveats but at
the very least the capability is there. As for other boards it is
virtually guaranteed that they support some form of remote booting.

This may be the better method eventually but at this time it does not
work for us. I personally started working in imx53loco (quickstart) usb
boot and I got to a point where I can ping-pong with the boot rom over
raw USB protocol (hey, from python :-) in response to plugging a imx53
and turning it on.

There is a method for delivering boot payload but I did not attempt it
yet as I'm certain stock uboot will not work this way (or maybe it
would, did not try). If anyone is willing to help I'll gladly share my
code and work towards doing this for _at least_ one board. I started
with loco because panda is kind of there already and all the hard work
has been done. Everything works on panda nowadays ;-P

Thanks
ZK

[Alan Bennett (akbennett)]

Bootloader test capability is now on the LAVA roadmap and tracked. http://cards.linaro.org/browse/CARD-491