Need support for Snowball board

[Hans Odeberg]

Need a new BoardConfig class for the Snowball board (ux500-based). The class should:
* Specialize get_sfdisk_cmd() to provide an additional partition at address 0x20000
* Create the data that should be stored in that partition out of boot files stored in the corresponding HW pack

[James Westby (james-w)]

On Thu, 31 Mar 2011 14:42:29 -0000, Hans Odeberg <email address hidden> wrote:
> * Create the data that should be stored in that partition out of boot files stored in the corresponding HW pack

These files are packaged? If so then you can follow the existing way
that it finds u-boot files and copy that.

If not then we'll have to find a way around that, as currently hwpacks
just contain packages.

The easiest solution from linaro-image-tools point of view will be to
package the files. If that's not possible then we'll have to discuss the
constraints on those files so that we can design a solution.

Thanks,

James

[Loïc Minier (lool)]

On Fri, Apr 01, 2011, James Westby wrote:
> If not then we'll have to find a way around that, as currently hwpacks
> just contain packages.
>
> The easiest solution from linaro-image-tools point of view will be to
> package the files. If that's not possible then we'll have to discuss the
> constraints on those files so that we can design a solution.

 (This is also my preference.)

 I had an email exchange about these files with ST-Ericsson folks;
 unfortunately they contain binary blobs. I suggested to seek
 redistribution permission so that we could at least include them in the
 archive, but it might take some time.

 We could also add a command-line flag taking the various blobs
 directly.

--
Loïc Minier

[James Westby (james-w)]

On Fri, 01 Apr 2011 15:53:50 -0000, Loïc Minier <email address hidden> wrote:
> On Fri, Apr 01, 2011, James Westby wrote:
> > If not then we'll have to find a way around that, as currently hwpacks
> > just contain packages.
> >
> > The easiest solution from linaro-image-tools point of view will be to
> > package the files. If that's not possible then we'll have to discuss the
> > constraints on those files so that we can design a solution.
>
> (This is also my preference.)
>
> I had an email exchange about these files with ST-Ericsson folks;
> unfortunately they contain binary blobs. I suggested to seek
> redistribution permission so that we could at least include them in the
> archive, but it might take some time.

Right. As the bug mentioned hwpacks I assumed that we would have at
least that permission. If we don't then Linaro can't provide hwpacks
that will have the board fully working.

> We could also add a command-line flag taking the various blobs
> directly.

That would work, but it's a poorer user experience, so I'd prefer a
hwpack-based solution.

If we don't have redistribution permission then we will have to go this
way I think.

Thanks,

James

[Hans Odeberg (hans-odeberg)]

Yes, the boot files are binary blobs. Not only that, some of them are signed with security keys only available to a few individuals at ST-Ericsson. So even if source was available they could not be rebuilt.

Our goal is still to create a HW pack containing all those binary boot files, and we would love to be able to upload it to where HW packs normally go.

Today we can only provide these binaries to people who have signed an NDA. Clearly, this is completely unacceptable for the Snowball boards. So we are working very tight with our licensing department right now, to make these binaries redistributable to anybody. And I really cannot imagine any other scenario than that we manage to do this before the Snowball boards are released.

The current status at the ST-Ericsson end is that:
* I have added code to board.py to copy the boot binaries into the image
* A colleague is working on the HW pack
Then we just need to build an image, flash it to a Snowball board and keep reflashing and retuning the Snowball board config until the board boots. I suspect that will be what takes the most time.

Hans

[James Westby (james-w)]

On Sat, 02 Apr 2011 15:01:13 -0000, Hans Odeberg <email address hidden> wrote:
> Yes, the boot files are binary blobs. Not only that, some of them are
> signed with security keys only available to a few individuals at ST-
> Ericsson. So even if source was available they could not be rebuilt.
>
> Our goal is still to create a HW pack containing all those binary boot
> files, and we would love to be able to upload it to where HW packs
> normally go.
>
> Today we can only provide these binaries to people who have signed an
> NDA. Clearly, this is completely unacceptable for the Snowball boards.
> So we are working very tight with our licensing department right now, to
> make these binaries redistributable to anybody. And I really cannot
> imagine any other scenario than that we manage to do this before the
> Snowball boards are released.
>
> The current status at the ST-Ericsson end is that:
> * I have added code to board.py to copy the boot binaries into the image
> * A colleague is working on the HW pack
> Then we just need to build an image, flash it to a Snowball board and
> keep reflashing and retuning the Snowball board config until the board
> boots. I suspect that will be what takes the most time.

Hi,

Thanks for the information.

Although you can't show us those files yet, you could presumably show us
the support you are adding to linaro-image-tools and the approach you
are taking to put the files in the hwpack?

I suggest you do this sooner rather than later. If you leave it until
the day before the Snowball boards are available then it's very unlikely
that linaro-image-tools will support them the day after.

Also, we can comment on the approach without testing, and agree in
principle to that, but we'll need access to all the needed files before
we can merge anything, and we'll need redistribution rights before
Linaro can support the snowball as we do other boards. In fact to be
clear, in order to support the board like we do others, we need the
rights to distribute the files, with the rights for others to
redistribute them as well.

Thanks,

James

[Hans Odeberg (hans-odeberg)]

Yes, showing the source early is a good idea. So I enclose a patch of what I have written so far. I have focused on creating an filling the bootloader partition.
So far, I have not tested running the tool - I don't have a HW pack to test it against. What I have tested is that I created some dummy boot files, an empty image file, and invoked SnowballConfig._make_boot_files(), verifying that the boot files ended up at the places in the image file that I expected.

What I have neither coded for or tested is that the u-boot environment variables are correct for booting the Snowball board. So don't expect this to work...

/Hans

[James Westby (james-w)]

On Mon, 04 Apr 2011 14:46:10 -0000, Hans Odeberg <email address hidden> wrote:
> Yes, showing the source early is a good idea. So I enclose a patch of
> what I have written so far. I have focused on creating an filling the
> bootloader partition.

Hi,

This approach looks reasonable to me, and it's good that it is contained
entirely within the Snowball config class.

It looks as though this assumes that the blobs are in the current
working directory with specific names though, which is obviously not
ideal. Knowing what to do there is at least partly dependent on the
license question though.

Thanks,

James

[Hans Odeberg (hans-odeberg)]

No, I really hope I don't have to put the binary blobs in the current directory - I suspect that would be slightly surprising to anyone who has used l-m-c to build for another board.

Currently I prepend the value of the variable uboot_parts_dir to the listed filenames, somewhere in the function get_file_info(). uboot_parts_dir is set to /boot somewhere else in the file, and that is where I expect the HW pack to unpack the boot blobs.

Yes, hardcoded names might be a problem - if we decide to encode version names in the filenames. But in that case I could just steal an idea I saw elsewhere in the file - to glob for a filename pattern instead.

/Hans

[Hans Odeberg (hans-odeberg)]

Sorry for being silent for so long...

Thanks to Tony Månsson, we are beginning to have something locally that seems to work; we are building raw images, flashing and booting.

In doing this, we had to patch the linaro-media-create script itself; something I'd love to get some feedback on. As you might be able to infer from the changes, we added a class variable to the BoardConfig class, specifying whether the targeted media is a block device or not. And to set that variable, a minor change to linaro-media-create was needed.

We then use that variable in the SnowballConfig class, to only create and populate a loader partition when the target media is a raw image file.

Then I have also started looking at getting linaro-android-media-create to work for Snowball, and have not gotten further than a couple of questions:
1) In partitions.py/get_android_partitions_for_media(), there is an assert for media.is_block_device. But Snowball really needs to build for raw images; the board cannot boot from SD card. Will this limitation go away in the future?
2) android_boards.py/AndroidBoardConfig::get_sfdisk_cmd() defines the following partitions (assuming I got sfdisk right):
  boot partition, FAT (same as before)
  system partition, Linux
  cache partition, Linux
  an extended partition
    userdata partition, Linux
    sdcard partition
If I am to add a loader partition, as I did in SnowballConfig, I then run into the minor problem that the partition table is full. So the question, at long last, is: Can you rearrange the default partition table to move more things to the extended partition, leaving a free entry, or is that my job when I define an AndroidSnowballlConfig::get_sfdisk_cmd()?

[Guilherme Salgado (salgado)]

This feels very weird to me as the media we're writing to does not vary
according to the board -- it varies according to a command line switch.
For that reason I think it doesn't make sense to have it as a
BoardConfig attribute.

Also, we have plans of dropping the ability to create images on block
devices (bug 744449), so media.is_block_device and all the code specific
to block devices is going away soon.

It seems to me like the best alternative here is to have two separate
BoardConfigs for snowball as Loïc suggested
(http://<email address hidden>/msg03974.html)

On Mon, 2011-05-09 at 08:39 +0000, Hans Odeberg wrote:
[...]
> Then I have also started looking at getting
> linaro-android-media-create to work for Snowball, and have not gotten
> further than a couple of questions:
> 1) In partitions.py/get_android_partitions_for_media(), there is an
> assert for media.is_block_device. But Snowball really needs to build
> for raw images; the board cannot boot from SD card. Will this
> limitation go away in the future?

I think that assert is there only because when l-a-m-c was written it
didn't make much sense to generate image files for android. If now it
does, we can just remove the assert and make sure l-a-m-c knows how to
generate image files. We'll probably have to do that anyway as part of
the work to fix the bug I mentioned above.

> 2) android_boards.py/AndroidBoardConfig::get_sfdisk_cmd() defines the
> following partitions (assuming I got sfdisk right):
> boot partition, FAT (same as before)
> system partition, Linux
> cache partition, Linux
> an extended partition
> userdata partition, Linux
> sdcard partition
> If I am to add a loader partition, as I did in SnowballConfig, I then
> run into the minor problem that the partition table is full. So the
> question, at long last, is: Can you rearrange the default partition
> table to move more things to the extended partition, leaving a free
> entry, or is that my job when I define an
> AndroidSnowballlConfig::get_sfdisk_cmd()?

I think it doesn't make sense to always leave free space at the
beginning of the disk for just one or two boards that might need a
loader partition. It is common for board configs to override the
standard get_sfdisk_cmd(), and that's probably what we should do here.

[Hans Odeberg (hans-odeberg)]

> This feels very weird to me as the media we're writing to does not vary
> according to the board -- it varies according to a command line switch.
> For that reason I think it doesn't make sense to have it as a
> BoardConfig attribute.

Well, the point of putting media type as an attribute in the board class was that the Snowball board class needs to do slightly different things depending on the media type.
Yes, having two board classes is an alternative. To me, that also feels very weird. Then it will be possible for a user to create an image using the Snowball_sdcard device, and an SD card using the Snowball_image device, with interesting results.
But, if you feel that that is what will fit best with future designs, then I will create two board classes. Should be simple.

> I think it doesn't make sense to always leave free space at the
> beginning of the disk for just one or two boards that might need a
> loader partition. It is common for board configs to override the
> standard get_sfdisk_cmd(), and that's probably what we should do here.

Yes, that is exactly the way we're doing in the SnowballConfig: overloading get_sfdisk_cmd(). My point was that the standard get_sfdisk_cmd() for Android creates four primary partitions, which is the maximum number allowed in the MBR. Thus I see two options:
1) Change the standard get_sfdisk_cmd() to return three primary partitions, by moving one of the existing primary partitions into the extended partition. That way I can still just add a partition in front by overriding get_sfdisk_cmd().
2) Let the boards that need an extra partition in front be responsible for doing the reshuffling in 1) themselves.

[Guilherme Salgado (salgado)]

On Mon, 2011-05-09 at 18:08 +0000, Hans Odeberg wrote:
> > This feels very weird to me as the media we're writing to does not vary
> > according to the board -- it varies according to a command line switch.
> > For that reason I think it doesn't make sense to have it as a
> > BoardConfig attribute.
>
> Well, the point of putting media type as an attribute in the board
> class was that the Snowball board class needs to do slightly different
> things depending on the media type.
> Yes, having two board classes is an alternative. To me, that also
> feels very weird. Then it will be possible for a user to create an
> image using the Snowball_sdcard device, and an SD card using the
> Snowball_image device, with interesting results.

We can easily prevent that by aborting the script if incompatible
options are passed in, though. I'm not very fond of this either, but it
would be temporary anyway as we plan to drop the --mmc option.

> But, if you feel that that is what will fit best with future designs,
> then I will create two board classes. Should be simple.
>
> > I think it doesn't make sense to always leave free space at the
> > beginning of the disk for just one or two boards that might need a
> > loader partition. It is common for board configs to override the
> > standard get_sfdisk_cmd(), and that's probably what we should do
> here.
>
> Yes, that is exactly the way we're doing in the SnowballConfig:
> overloading get_sfdisk_cmd(). My point was that the standard
> get_sfdisk_cmd() for Android creates four primary partitions, which is
> the maximum number allowed in the MBR. Thus I see two options:

But you're overriding get_sfdisk_cmd() so you can create the 5
partitions you need, anyway you want, no? Maybe what you're trying to
do is extend the standard get_sfdisk_cmd() in your custom config class
by prepending to its return value the sfdisk commands to create an extra
partition at the beginning?

> 1) Change the standard get_sfdisk_cmd() to return three primary
> partitions, by moving one of the existing primary partitions into the
> extended partition. That way I can still just add a partition in front
> by overriding get_sfdisk_cmd().

I don't see a problem with that, but then you might as well move all the
partitions, with the exception of the boot one, into the extended
partition?

> 2) Let the boards that need an extra partition in front be responsible
> for doing the reshuffling in 1) themselves.