linaro-hwpack-create fails to build arndale-octa on Ubuntu 14.04

[Sean Payne]

linaro-hwpack-create on 14.04 crashes. I expected the linaro image tools to work on 14.04.

Below shows how to reproduce the problem.

Also I included two proposed patches, which I think resolve the problem. I tested the proposed fixes on arndale-octa on 14.04 and 12.04.

To reproduce the crash:
  On a 14.04 machine:
    sudo apt-get update
    sudo apt-get upgrade
    mkdir repro
    cd repro
    sudo apt-get install --reinstall python-linaro-image-tools
    git clone https://git.linaro.org/hwpack/arndale-octa.git
    linaro-hwpack-create --debug arndale-octa/hwpacks/linaro-arndale-octa 42

Expected result:
The following files should be created in the working directory
  hwpack_linaro-arndale-octa_42_armhf_supported.manifest.txt
  hwpack_linaro-arndale-octa_42_armhf_supported.tar.gz

Actual Result:
Python assertion:
Traceback (most recent call last):
  File "/usr/bin/linaro-hwpack-create", line 57, in <module>
    builder.build()
  File "/usr/lib/pymodules/python2.7/linaro_image_tools/hwpack/builder.py", line 230, in build
    fetcher.ignore_packages(self.config.assume_installed)
  File "/usr/lib/pymodules/python2.7/linaro_image_tools/hwpack/packages.py", line 682, in ignore_packages
    ", ".join(broken))
AssertionError: Weirdly unable to satisfy dependencies of dh-python

Proposed Fix 1: (multi-arch.patch)
This dh-python Assertion is thrown while the builder.py is making a list of packages to ignore by feeding python-apt with a generated dpkg/status. python-apt on 14.04 marks dh-python is_inst_broken.

Note that the dh-python package includes a Multi-Arch Depends: header (Depends: python3:any).
Note that in packages.py the Multi-Arch: header is not written to the generated dpkg/status
I do not know what specifically has changed in Multi-Arch or python-apt between 12.04 to 14.04 that causes this change in behaviour, however I do think passing along Multi-Arch: headers is logical.

The fix: (multi_arch.patch)
Any time a pre_depends clause is read or written in packages.py, make sure multi_arch is also read or written.

cd /usr/share/pyshared
sudo patch -p1 < multi_arch.patch

The result:
After applying the patch, the above problem is not reproducible.

However linaro-hwpack-create continues to fail for a separate reason:
Traceback (most recent call last):
  File "/usr/bin/linaro-hwpack-create", line 57, in <module>
    builder.build()
  File "/usr/lib/pymodules/python2.7/linaro_image_tools/hwpack/builder.py", line 240, in build
    self._add_packages_to_hwpack(local_packages)
  File "/usr/lib/pymodules/python2.7/linaro_image_tools/hwpack/builder.py", line 317, in _add_packages_to_hwpack
    self.hwpack.add_dependency_package(self.config.packages)
  File "/usr/lib/pymodules/python2.7/linaro_image_tools/hwpack/hardwarepack.py", line 535, in add_dependency_package
    self.packages.append(FetchedPackage.from_deb(deb_file_path))
  File "/usr/lib/pymodules/python2.7/linaro_image_tools/hwpack/packages.py", line 403, in from_deb
    debcontrol = DebFile(deb_file_path).control.debcontrol()
  File "/usr/lib/python2.7/dist-packages/debian/debfile.py", line 271, in __init__
    compressed_part_name(DATA_PART)))
  File "/usr/lib/python2.7/dist-packages/debian/debfile.py", line 256, in compressed_part_name
    " (expected one of: %s)" % candidates)
debian.debfile.DebError: missing required part in given .deb (expected one of: ['data.tar.gz', 'data.tar.bz2', 'data.tar.lzma'])

Fix 2 (gzip.patch attached in a separate update)
The error above is thrown when debian.Debfile is trying to unpack the DATA_PART of something built by dpkg-deb.
Note that the DebError expects one of .gz, bz2, lzma.

Note that in 14.04, default compression for dpkg-deb has changed:
man dpkg-deb on 14.04:
  -Zcompress-type
              ...(default is xz).
man dpkg-deb on 12.04:
  -Zcompress-type
              ...(default is gzip).

So the proposed fix is to specify gzip when building the .deb file in PackageMaker::make_package:
ie change this argument list:
['dpkg-deb', '-b', packaging_dir],
to
['dpkg-deb', '-b', '-Zgzip', packaging_dir],

cd /usr/share/pyshared
sudo patch -p1 < gzip.patch

The result:
After applying both packages the above problems are no longer reproducible.
The hwpack is created in the working directory as described in Expected Results above.

[Sean Payne (sean-d-payne)]

attached the second fix

[Fathi Boudra (fboudra)]

multi-arch patch reviewed and merged: https://review.linaro.org/#/c/1711/

[Fathi Boudra (fboudra)]

gzip patch reviewed and merged: https://review.linaro.org/#/c/1713/

Thanks for your contribution.

[Fathi Boudra (fboudra)]

Reopening, we still have some failures with linaro-hwpack-replace:
https://ci.linaro.org/jenkins/view/engineering-builds/job/ubuntu-armhf-hwpacks/hwpack=vexpress64-be,label=build/634/console

linaro-hwpack-replace -t hwpack_linaro_arm64_supported.tar.gz -p linux-headers-3.14.0-1-linaro-vexpress64-be_3.14.0-1.1ubuntu1~ci+140421074033_arm64.deb -r linux-headers -d -i
2014-04-24 10:37:59,206 - linaro_image_tools - ERROR - Error Details: missing required part in given .deb (expected one of: ['data.tar.gz', 'data.tar.bz2', 'data.tar.lzma'])
2014-04-24 10:37:59,216 - linaro_image_tools - ERROR - Injecting the debian package 'linux-headers-3.14.0-1-linaro-vexpress64-be_3.14.0-1.1ubuntu1~ci+140421074033_arm64.deb' failed

[Fathi Boudra (fboudra)]

The BE and arm64 configurations have failed:
* arndale-be
* panda-be
* lsk-vexpress64
* lsk-vexpress64-be
* vexpress64
* vexpress64-be

[Sean Payne (sean-d-payne)]

For the linaro-hwpack-replace issue you mention above, I believe the problem is that the linux-headers-3.14.0-1-linaro-vexpress64-be_3.14.0-1.1ubuntu1~ci+140421074033_arm64.deb artifact was built with .xz compression.

I believe it's built in the ubuntu-kernel-ci/scripts/package_kernel and this is calling dpkg-buildpackage without the -Z flag.

I added a proposed patch to add to ubuntu-kernel-ci. It is *untested*

Then once the source artifacts get regenerated, these problems should go away.