missing support for dtb_files (needed for vexpress)

[John Rigby]

I use the attached config file to create a hwpack then when I use the result with linaro-media-create I get an error:

`/tmp/tmpy9d2TP/u_boot/usr/lib/u-boot/vexpress_ca9x4/u-boot.bin' -> `/tmp/tmplbI8wE/boot-disc/u-boot.bin'
Unpacked package /tmp/tmpQ9K0Af/pkgs/linux-image-linaro-vexpress_3.6.0.1.1_armhf.deb.
Traceback (most recent call last):
  File "./linaro-media-create", line 240, in <module>
    args.is_live, args.is_lowmem, args.consoles)
  File "/home/jcrigby/work/bzr-trees/milo/bug1054422/linaro_image_tools/media_create/boards.py", line 960, in populate_boot
    cls.copy_files(boot_disk)
  File "/home/jcrigby/work/bzr-trees/milo/bug1054422/linaro_image_tools/media_create/boards.py", line 984, in copy_files
    source_path, source_package)
  File "/home/jcrigby/work/bzr-trees/milo/bug1054422/linaro_image_tools/media_create/boards.py", line 348, in get_file_from_package
    file_path)
  File "/home/jcrigby/work/bzr-trees/milo/bug1054422/linaro_image_tools/hwpack/builder.py", line 92, in get_file
    "not found in the package '%s'." % (file, package)
AssertionError: The file 'lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca9.dtb' was not found in the package '/tmp/tmpQ9K0Af/pkgs/linux-image-linaro-vexpress_3.6.0.1.1_armhf.deb'.

[John Rigby (jcrigby)]

The config file when invoking linaro-hwpack-create

[John Rigby (jcrigby)]

I have extracted the resulting hwpack and then used dpkg -c to verify that the file is in the .deb:

dpkg -c pkgs/linux-image-3.6.0-1-linaro-vexpress_3.6.0-1.1~ci+120924145659_armhf.deb
drwxr-xr-x root/root 0 2012-09-24 10:18 ./
drwxr-xr-x root/root 0 2012-09-24 10:18 ./usr/
drwxr-xr-x root/root 0 2012-09-24 10:18 ./usr/share/
drwxr-xr-x root/root 0 2012-09-24 10:18 ./usr/share/doc/
drwxr-xr-x root/root 0 2012-09-24 10:20 ./usr/share/doc/linux-image-3.6.0-1-linaro-vexpress/
-rw-r--r-- root/root 1292 2012-09-24 09:48 ./usr/share/doc/linux-image-3.6.0-1-linaro-vexpress/copyright
-rw-r--r-- root/root 904 2012-09-24 09:48 ./usr/share/doc/linux-image-3.6.0-1-linaro-vexpress/changelog.Debian.gz
-rw-r--r-- root/root 904 2012-09-24 10:18 ./usr/share/doc/linux-image-3.6.0-1-linaro-vexpress/changelog.Debian.old.gz
drwxr-xr-x root/root 0 2012-09-24 10:18 ./boot/
-rw------- root/root 1577863 2012-09-24 10:18 ./boot/System.map-3.6.0-1-linaro-vexpress
-rw-r--r-- root/root 282173 2012-09-24 10:18 ./boot/abi-3.6.0-1-linaro-vexpress
-rw-r--r-- root/root 49345 2012-09-24 10:18 ./boot/config-3.6.0-1-linaro-vexpress
-rw------- root/root 3662208 2012-09-24 10:18 ./boot/vmlinuz-3.6.0-1-linaro-vexpress
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/firmware/
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/firmware/3.6.0-1-linaro-vexpress/
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/firmware/3.6.0-1-linaro-vexpress/device-tree/
-rw-r--r-- root/root 7736 2012-09-24 10:18 ./lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca5s.dtb
-rw-r--r-- root/root 7629 2012-09-24 10:18 ./lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca15_a7.dtb
-rw-r--r-- root/root 9838 2012-09-24 10:18 ./lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca15-tc2.dtb
-rw-r--r-- root/root 7771 2012-09-24 10:18 ./lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca15-tc1.dtb
-rw-r--r-- root/root 8178 2012-09-24 10:18 ./lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca9.dtb
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/modules/
drwxr-xr-x root/root 0 2012-09-24 10:20 ./lib/modules/3.6.0-1-linaro-vexpress/
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/modules/3.6.0-1-linaro-vexpress/initrd/
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/modules/3.6.0-1-linaro-vexpress/kernel/
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/modules/3.6.0-1-linaro-vexpress/kernel/drivers/
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/modules/3.6.0-1-linaro-vexpress/kernel/drivers/char/
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/modules/3.6.0-1-linaro-vexpress/kernel/drivers/char/hw_random/
-rw-r--r-- root/root 6552 2012-09-24 10:18 ./lib/modules/3.6.0-1-linaro-vexpress/kernel/drivers/char/hw_random/rng-core.ko
drwxr-xr-x root/root 0 2012-09-24 10:18 ./lib/modules/3.6.0-1-linaro-vexpress/kernel/drivers/gator/
-rw-r--r-- root/root 64684 2012-09-24 10:18 ./lib/modules/3.6.0-1-linaro-vexpress/kernel/drivers/...

[John Rigby (jcrigby)]

I have tried with and without the ./:
        - ./lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca9.dtb : v2p-ca9.dtb
and
        - lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca9.dtb : v2p-ca9.dtb

and neither works

[Ricardo Salveti (rsalveti)]

I think that's probably because the copy_files support was created to only copy files available at the bootloader package, and not generically at the rootfs.

We'd need such support at the root layer of the hwpack v3 format, and support to copy any file from the filesystem to a partition or path, which in this case we'd need to use for the dtb files available at /lib/firmware/...

[Ricardo Salveti (rsalveti)]

Looking further, our issue would probably be resolved by the dtb_files support, which is not yet properly implemented:

        if self.dtb_file is not None:
            # XXX In V3 this one should be a list, called dtb_files.
            metadata += dump({DTB_FILE_FIELD: self.dtb_file})

[John Rigby (jcrigby)]

https://wiki.linaro.org/HardwarePacksV3#bootloaders

says this about copy_files:

A list of files to copy from an optionally named packages to an optionally named destination. If the destination isn't set, files will be copied to /boot. There are two formats:
Without explicit package name, in this case the bootloader package will be used:

[Ricardo Salveti (rsalveti)]

On Tue, Sep 25, 2012 at 12:04 AM, John Rigby <email address hidden> wrote:
> https://wiki.linaro.org/HardwarePacksV3#bootloaders
>
> says this about copy_files:
>
> A list of files to copy from an optionally named packages to an optionally named destination. If the destination isn't set, files will be copied to /boot. There are two formats:
> Without explicit package name, in this case the bootloader package will be used:

Problem is that the kernel package that provide those files is always
the one with a specific version and ABI (not the meta as used by the
hwpack), so even if we put the correct package there, we'd need to
change the hwpack every time a new kernel ABI/version is published.

That's why the dtb_files makes more sense at this case.

[Ryan Harkin (ryanharkin)]

I believe the original intention of the copy_files feature was to be able to copy from any package to the /boot partition, although that meaning may have got lost on the way.

Ricardo is probably correct that the dtb_files feature should be used to copy them in this case. However, the copy_files tag should still be able to do the same job and will be needed when we add some other files to /boot in the near future (like the Boot Monitor script that ARM requested).

[Данило Шеган (danilo)]

Yes, copy_files is there only for /boot.

Regarding dtb_files, perhaps we can add it, though it's, imho, risky to include it this late in the cycle. However, if we do it as a separate field, it's probably not too bad.

[Данило Шеган (danilo)]

I've clarified the documentation to say that only paths under /boot are supported for copy_files option. Please let us know how urgent dtb_files support is, though we should get people to test l-i-t earlier on in the cycle if it's important for 2012.09 cycle.

[Ryan Harkin (ryanharkin)]

Hey Danilo,

I believe that John was trying to copy the files into /boot, that's where we want them. The problem he is having is that the script cannot find the files in the source package. He was thinking that the source package can only be the bootloader package, but I think the copy files feature is supposed to allow any source package. Is this not correct?

But using dtb_files is a cleaner approach for copying the dtbs files, of course.

[John Rigby (jcrigby)]

Whatever is done don't forget Ricardo's comment in #7 the path to the dtb file will change with the kernel version so we need some limited wildcard support.

[Fathi Boudra (fboudra)]

As discussed, lower priority and schedule to be fixed for next cycle (12.10)

[Paul Sokolovsky (pfalcon)]

From Ryan:

"but I think the copy files feature is supposed to allow any source package."

This is correct.

But:

John's config:

    copy_files:
      linux-image-linaro-vexpress:
        - lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca9.dtb : v2p-ca9.dtb

That asks to get file out of "linux-image-linaro-vexpress". Later, John shows that such file exists in "linux-image-3.6.0-1-linaro-vexpress_3.6.0-1.1~ci+120924145659_armhf.deb". Well, but "linux-image-linaro-vexpress_3.6.0-1.1*.deb" and "linux-image-3.6.0-1-linaro-vexpress_3.6.0-1.1~ci+120924145659_armhf.deb" are two different packages! Yes, "linux-image-linaro-vexpress" is likely dummy package which refers to "linux-image-3.6.0-1-linaro-vexpress", but files are in the latter, and that's what should be used in config:

    copy_files:
      linux-image-3.6.0-1-linaro-vexpress:
        - lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca9.dtb : v2p-ca9.dtb

[Fathi Boudra (fboudra)]

On 27 September 2012 12:34, Paul Sokolovsky <email address hidden> wrote:
> >From Ryan:
>
> "but I think the copy files feature is supposed to allow any source
> package."
>
> This is correct.
>
> But:
>
> John's config:
>
> copy_files:
> linux-image-linaro-vexpress:
> - lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca9.dtb : v2p-ca9.dtb
>
> That asks to get file out of "linux-image-linaro-vexpress". Later, John
> shows that such file exists in "linux-image-3.6.0-1-linaro-
> vexpress_3.6.0-1.1~ci+120924145659_armhf.deb". Well, but "linux-image-
> linaro-vexpress_3.6.0-1.1*.deb" and "linux-image-3.6.0-1-linaro-
> vexpress_3.6.0-1.1~ci+120924145659_armhf.deb" are two different
> packages! Yes, "linux-image-linaro-vexpress" is likely dummy package
> which refers to "linux-image-3.6.0-1-linaro-vexpress", but files are in
> the latter, and that's what should be used in config:
>
> copy_files:
> linux-image-3.6.0-1-linaro-vexpress:
> - lib/firmware/3.6.0-1-linaro-vexpress/device-tree/vexpress-v2p-ca9.dtb : v2p-ca9.dtb

The problem is explained by Ricardo in comments#7
https://bugs.launchpad.net/linaro-image-tools/+bug/1055800/comments/7

We don't want to change the hwpack configuration every time a new
kernel ABI/version is published.

[Milo Casagrande (milo)]

The linked branch adds support for dtb_files filed (with basic glob support).

The wiki page has been updated to reflect the new syntax:
https://wiki.linaro.org/HardwarePacksV3

If somebody can test it, it would be great.
Thanks.

[Milo Casagrande (milo)]

dtb_files support is now in linaro-image-tools as of rev569.

The syntax is explained in:
https://wiki.linaro.org/HardwarePacksV3