Need to build the DT blob for Origen, Linux Linaro Tracking project in jenkins

[Andrey Konovalov]

In LAVA Origen is set up to use the device tree. The hwpack created by the CI job doesn't contain the dtb file. Hence the boot failure (http://validation.linaro.org/lava-server/scheduler/job/34934/log_file):

-----8<-----
ORIGEN # setenv bootcmd 'fatload mmc 0:5 0x40007000 uImage; fatload mmc 0:5 0x42000000 uInitrd; fatload mmc 0:5 0x41f00000 board.dtb; bootm 0x40007000 0x42000000 0x41f00000'
ORIGEN # setenv bootargs 'console=ttySAC2,115200n8 root=LABEL=testrootfs rootwait ro'
ORIGEN # boot
reading uImage

3878424 bytes read
reading uInitrd

1532483 bytes read
reading board.dtb

** Unable to read "board.dtb" from mmc 0:5 **
<snip>

ERROR: Did not find a cmdline Flattened Device Tree
Could not find a valid device tree
-----8<-----

The llct and the ll trees have the required change in the builddeb script. So adding the "make dtbs" step to the build_instructions() in jenkins_common_lib should be enough.

[Milo Casagrande (milo)]

Hi Andrey, the lined branch adds support for a parametrized build (you will need to set a parameter in the CI job) in oder to use a further step in the compilation and run 'make dtbs'.

I have a question though: I was looking at the hwpack used by the LAVA job you pointed us, but that hardware pack does not contain any information about dtb file (if you open the hwpack and look at the metadata file, no DTB file is listed there). Is this supposed to be so for Origen hwpacks? The hwpack configuration file (both version 2 and the latest version 3) supports the definition of a DTB file that will be copied in the boot directory.

I guess the "make dtbs" is an extra step to compile the correct stuff and have a package with the DTB file included, that will be installed correctly. Is that all that is needed?

[Ricardo Salveti (rsalveti)]

On Fri, Oct 12, 2012 at 5:33 AM, Milo Casagrande
<email address hidden> wrote:
> I have a question though: I was looking at the hwpack used by the LAVA
> job you pointed us, but that hardware pack does not contain any
> information about dtb file (if you open the hwpack and look at the
> metadata file, no DTB file is listed there). Is this supposed to be so
> for Origen hwpacks? The hwpack configuration file (both version 2 and
> the latest version 3) supports the definition of a DTB file that will be
> copied in the boot directory.

You'd need the dtb_file to be part of the hwpack, like how it's done
for the leb-origen hwpack:
dtb_file: lib/firmware/*-lt-origen/device-tree/exynos4210-origen.dtb

Guess if you're not creating the hwpack yourself, we'd need to use one
as reference that correctly points out the device tree at the meta
file.

[Andrey Konovalov (andrey-konovalov)]

The linux-linaro-tracking_origen-exynos4 uses the most recent hwpack from http://snapshots.linaro.org/precise/hwpacks/origen/ .
Indeed, the hwpacks there don't have information about dtb file in the metadata file.

The url to get the hwpack from is (nearly) hardcoded in
http://bazaar.launchpad.net/~linaro-infrastructure/linaro-ci/lci-build-tools/view/head:/find_latest.py, near line 88:

    if hwpack == "panda" or hwpack == "snowball":
        url="http://snapshots.linaro.org/precise/hwpacks/lt-%s/" % hwpack
    else:
        url="http://snapshots.linaro.org/precise/hwpacks/%s/" % hwpack

Should we just change "snapshots.linaro.org/precise/hwpacks" to "http://snapshots.linaro.org/precise/hwpacks/leb-origen" for the Linux Linaro Tracking Origen project only (and some more "if"s to the code snippet above), or make it more flexible (add some parameter)?
Would it make sense to create the hwpack (vs just replacing the single kernel image package in the hwpack), as the dtb name specification in the leb-origen hwpack is quite strict (lib/firmware/*-lt-origen/device-tree/exynos4210-origen.dtb), and could lead to boot failures if it changes at some point later?

[Milo Casagrande (milo)]

On Fri, Oct 12, 2012 at 11:21 PM, Andrey Konovalov
<email address hidden> wrote:
> The linux-linaro-tracking_origen-exynos4 uses the most recent hwpack from http://snapshots.linaro.org/precise/hwpacks/origen/ .
> Indeed, the hwpacks there don't have information about dtb file in the metadata file.
>
> The url to get the hwpack from is (nearly) hardcoded in
> http://bazaar.launchpad.net/~linaro-infrastructure/linaro-ci/lci-build-tools/view/head:/find_latest.py, near line 88:
>
> if hwpack == "panda" or hwpack == "snowball":
> url="http://snapshots.linaro.org/precise/hwpacks/lt-%s/" % hwpack
> else:
> url="http://snapshots.linaro.org/precise/hwpacks/%s/" % hwpack
>
> Should we just change "snapshots.linaro.org/precise/hwpacks" to "http://snapshots.linaro.org/precise/hwpacks/leb-origen" for the Linux Linaro Tracking Origen project only (and some more "if"s to the code snippet above), or make it more flexible (add some parameter)?

I would rather make it more flexible at this point: we would still be
doing some code changes there to accomodate this, and it wouldn't be
much more work.

> Would it make sense to create the hwpack (vs just replacing the single kernel image package in the hwpack), as the dtb name specification in the leb-origen hwpack is quite strict (lib/firmware/*-lt-origen/device-tree/exynos4210-origen.dtb), and could lead to boot failures if it changes at some point later?

The dtb name specification in the hwpack supports basic "globbing", so
it shouldn't be difficult to specify a more general name (I never
tested glob in the DTB file name though, just the path).

--
Milo Casagrande
Infrastructure Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

[Milo Casagrande (milo)]

The "make dtbs" step has been added to the linaro-ci branch, it needs a boolean parameter to be set in the Jenkins build. The name of the parameter has to be "MAKE_DTBS", accepted values are "true" and "false".

I was looking at the Jenkins build:

https://ci.linaro.org/jenkins/view/Linux%20Linaro%20Tracking/job/linux-linaro-tracking-llct-branch_origen-exynos4/configure

and I noted that it is not using trunk versione of linaro-ci (lp:linaro-ci), but instead it is using one branch from Deepti:

bzr branch lp:~deeptik/linaro-ci/llt-config-fragment-changes lci-build-tools

Are there any particular reasons to use that branch?

[Milo Casagrande (milo)]

Regarding previous comment, I guess I picked the wrong Jenkins build. This one should be the correct one:

https://ci.linaro.org/jenkins/view/Linux%20Linaro%20Tracking/job/linux-linaro-tracking_origen-exynos4

And it is using trunk version of linaro-ci.

[Milo Casagrande (milo)]

On Mon, Oct 15, 2012 at 5:24 PM, Milo Casagrande
<email address hidden> wrote:
> On Fri, Oct 12, 2012 at 11:21 PM, Andrey Konovalov
>>
>> if hwpack == "panda" or hwpack == "snowball":
>> url="http://snapshots.linaro.org/precise/hwpacks/lt-%s/" % hwpack
>> else:
>> url="http://snapshots.linaro.org/precise/hwpacks/%s/" % hwpack
>>
>> Should we just change "snapshots.linaro.org/precise/hwpacks" to "http://snapshots.linaro.org/precise/hwpacks/leb-origen" for the Linux Linaro Tracking Origen project only (and some more "if"s to the code snippet above), or make it more flexible (add some parameter)?
>
> I would rather make it more flexible at this point: we would still be
> doing some code changes there to accomodate this, and it wouldn't be
> much more work.

I was looking into this, it looks like it could be made without even
touching the code.
The CI build uses environment variables to define which is the
board/hwpack to get: right now it is set to 'origen', we can switch to
'leb-origen' in the CI build just for this one.
If you cannot change the parameters of the build, we can do that for
you if you want.

>> Would it make sense to create the hwpack (vs just replacing the single kernel image package in the hwpack), as the dtb name specification in the leb-origen hwpack is quite strict (lib/firmware/*-lt-origen/device-tree/exynos4210-origen.dtb), and could lead to boot failures if it changes at some point later?
>
> The dtb name specification in the hwpack supports basic "globbing", so
> it shouldn't be difficult to specify a more general name (I never
> tested glob in the DTB file name though, just the path).

Regarding this, it could be good to update the hwpack configuration
file, using new 3.0 format and maybe even adding the dtb_files field.

--
Milo Casagrande
Infrastructure Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

[Milo Casagrande (milo)]

I created a copy of a Jenkins build to test out the new parameters:

https://ci.linaro.org/jenkins/job/milo-linux-linaro-tracking_origen-exynos4/

I had to tweak the make command (make dtbs does not simply work), and pass it the necessary parameters as with the previous make calls. The resulting call is:

make ARCH=arm O=$pkg_dir KERNELVERSION="$kernel_version" KERNELRELEASE="$kernel_version" CROSS_COMPILE="$TOOLCHAIN_PREFIX" dtbs

Are all those parameters necessary, or do we need to pass other?

The resulting dtb file, at least from the log is 'exynos4210-origen.dtb'. It should be safe also to use leb-origen in this case, but the appropriate hwpack config should be updated and used.

[Ricardo Salveti (rsalveti)]

I ported the 'origen' hwpack to V3, and also added the new dtb path as 'dtb_file: lib/firmware/*-origen/device-tree/exynos4210-origen.dtb'.

As it was still using the 3.5 kernel, I had to update the meta package to get the 3.6 instead, which should be done by Launchpad in a few minutes. Should be able to respin a new hwpack once that's done.

Andrey, would lib/firmware/*-origen/device-tree/exynos4210-origen.dtb work for you? We could change the package name (*-origen) for something that also works for you if needed (guess even a generic firmware/*/device-tree would be enough).

[Andrey Konovalov (andrey-konovalov)]

Milo,
you are right, all these parameters are needed:
"make ARCH=arm O=$pkg_dir KERNELVERSION="$kernel_version" KERNELRELEASE="$kernel_version" CROSS_COMPILE="$TOOLCHAIN_PREFIX" dtbs"

Ricardo,
yes, lib/firmware/*-origen/device-tree/exynos4210-origen.dtb works just fine. (I've switch https://ci.linaro.org/jenkins/job/milo-linux-linaro-tracking_origen-exynos4/ from leb-origin back to http://snapshots.linaro.org/precise/hwpacks/origen)
See https://ci.linaro.org/jenkins/job/milo-linux-linaro-tracking_origen-exynos4/4/ .
I've started http://validation.linaro.org/lava-server/scheduler/job/36092 with the hwpack created by this "Build #4". Let's see how it goes.

[Andrey Konovalov (andrey-konovalov)]

> Let's see how it goes.
Failed. The ci script needs fixing.

[Milo Casagrande (milo)]

Hello Andrey,

On Fri, Oct 19, 2012 at 11:28 PM, Andrey Konovalov
<email address hidden> wrote:
> Failed. The ci script needs fixing.

thanks for the comments here and in the merge proposal.
I updated the code in the merge request as per your comment there.

Branch where it is possible to test it is:
https://code.launchpad.net/~milo/linaro-ci/bug1064686

I will update the CI build with the new branch in order to test it.
Ciao.

--
Milo Casagrande
Infrastructure Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

[Andrey Konovalov (andrey-konovalov)]

Fixed in linux-linaro-tracking_origen-exynos4 build #232 (Oct 24, 2012 6:06:12 AM) :
https://ci.linaro.org/jenkins/view/Linux%20Linaro%20Tracking/job/linux-linaro-tracking_origen-exynos4/232/
The resulting kernel booted OK with the dtb:
http://validation.linaro.org/lava-server/scheduler/job/36584

[Milo Casagrande (milo)]

Thanks Andrey for testing it and confirming it now works.
I'm going to remove my personal copy of the CI build.