Kernel CI fails if hwpack isn't available

[Fathi Boudra]

Kernel CI on Origen fails since the hwpacks aren't available anymore (per pruning policy).

As we want to ensure Origen is still working on mainline and that Linaro doesn't support Origen anymore, we should fix find_latest.py script to grab latest available hwpack from releases (fallback).

[Naresh Kamboju (naresh-kamboju)]

This issue re-produced on panda build=30

https://ci.linaro.org/jenkins/job/linux-next/hwpack=panda,label=precise_hwpack_cloud/30/console

01:31:55 ++ python lci-build-tools/get_latest_slo_hwpack
01:31:55 Traceback (most recent call last):
01:31:55 File "lci-build-tools/get_latest_slo_hwpack", line 16, in <module>
01:31:55 hwpack_url = find_latest_hwpack(hwpack_type)
01:31:55 File "/mnt/ci_build/workspace/linux-next/hwpack/panda/label/precise_hwpack_cloud/lci-build-tools/find_latest.py", line 96, in find_latest_hwpack
01:31:55 filename = find_latest(hwpack_url)
01:31:55 File "/mnt/ci_build/workspace/linux-next/hwpack/panda/label/precise_hwpack_cloud/lci-build-tools/find_latest.py", line 72, in find_latest
01:31:55 lastdate = dates.pop()
01:31:55 IndexError: pop from empty list

[Naresh Kamboju (naresh-kamboju)]

This issue re-produced on below builds

origen build=30
https://ci.linaro.org/jenkins/job/linux-next/hwpack=origen,label=precise_hwpack_cloud/30/console

panda build=30
https://ci.linaro.org/jenkins/job/linux-next/hwpack=panda,label=precise_hwpack_cloud/30/console

[Naresh Kamboju (naresh-kamboju)]

Kernel CI failed summart report:
Failed to build linux-next and linux-arm-soc-for-next: snowball hw-pack.
kernel image (zImage) build successful but failed to create hw-pack.

Failed logs:
- https://ci.linaro.org/jenkins/job/linux-next/hwpack=snowball,label=kernel_cloud/113/consoleFull
- https://ci.linaro.org/jenkins/job/linux-arm-soc-for-next/hwpack=snowball,label=kernel_cloud/115/consoleFull

Error log:
02:18:33 dpkg-deb: building package
`linux-image-3.11.0-rc2-linaro-u8500' in
`../linux-image-3.11.0-rc2-linaro-u8500_3.11.0-rc2-linaro-u8500-1_armhf.deb'.
02:18:38 ++ date +%s
02:18:38 + END=1375064317
02:18:38 + EXECUTION_TIME_IN_SEC=666
02:18:38 + '[' 1 -eq 1 ']'
02:18:38 ++ python lci-build-tools/get_latest_slo_hwpack
02:18:39 Traceback (most recent call last):
02:18:39 File "lci-build-tools/get_latest_slo_hwpack", line 16, in <module>
02:18:39 hwpack_url = find_latest_hwpack(hwpack_type)
02:18:39 File
"/mnt/ci_build/workspace/linux-next/hwpack/snowball/label/kernel_cloud/lci-build-tools/find_latest.py",
line 96, in find_latest_hwpack
02:18:39 filename = find_latest(hwpack_url)
02:18:39 File
"/mnt/ci_build/workspace/linux-next/hwpack/snowball/label/kernel_cloud/lci-build-tools/find_latest.py",
line 72, in find_latest
02:18:39 lastdate = dates.pop()
02:18:39 IndexError: pop from empty list
02:18:39 + latest_kernel=
02:18:39 + cleanup_exit
02:18:39 + cd /mnt/ci_build/workspace/linux-next/hwpack/snowball/label/kernel_cloud
02:18:39 + rm -rf lci-build-tools
02:18:39 + rm -rf builddir-3.11.0-rc2-linaro-u8500
02:18:40 Build step 'Execute shell' marked build as failure
02:18:40 Finished: FAILURE

[Naresh Kamboju (naresh-kamboju)]

Snowball hwpack build failure summary report:
-------------------------------------------------------------------
https://ci.linaro.org/jenkins/job/linux-next/hwpack=snowball,label=kernel_cloud/165/consoleFull
https://ci.linaro.org/jenkins/job/linux-next/hwpack=snowball,label=kernel_cloud/166/consoleFull

Build error log:
---------------------
02:31:14 ++ python lci-build-tools/get_latest_slo_hwpack
02:31:14 Traceback (most recent call last):
02:31:14 File "lci-build-tools/get_latest_slo_hwpack", line 16, in <module>
02:31:14 hwpack_url = find_latest_hwpack(hwpack_type)
02:31:14 File "/mnt/ci_build/workspace/linux-next/hwpack/snowball/label/kernel_cloud/lci-build-tools/find_latest.py", line 96, in find_latest_hwpack
02:31:14 filename = find_latest(hwpack_url)
02:31:14 File "/mnt/ci_build/workspace/linux-next/hwpack/snowball/label/kernel_cloud/lci-build-tools/find_latest.py", line 72, in find_latest
02:31:14 lastdate = dates.pop()
02:31:14 IndexError: pop from empty list

[Naresh Kamboju (naresh-kamboju)]

STE Snowball Kernel CI build failed on all following 3 branches since 19 September 2013:

linux-arm-soc-for-next: https://ci.linaro.org/jenkins/job/linux-arm-soc-for-next/hwpack=snowball,label=kernel_cloud/167/console
linux-mainline: https://ci.linaro.org/jenkins/job/linux-mainline/hwpack=snowball,label=kernel_cloud/165/console
linux-next: https://ci.linaro.org/jenkins/job/linux-next/hwpack=snowball,label=kernel_cloud/165/console

The error looks same:

++ python lci-build-tools/get_latest_slo_hwpack
Traceback (most recent call last):
  File "lci-build-tools/get_latest_slo_hwpack", line 16, in <module>
    hwpack_url = find_latest_hwpack(hwpack_type)
  File "/mnt/ci_build/workspace/linux-mainline/hwpack/snowball/label/kernel_cloud/lci-build-tools/find_latest.py", line 96, in find_latest_hwpack
    filename = find_latest(hwpack_url)
  File "/mnt/ci_build/workspace/linux-mainline/hwpack/snowball/label/kernel_cloud/lci-build-tools/find_latest.py", line 72, in find_latest
    lastdate = dates.pop()
IndexError: pop from empty list

So looks like the hardware pack is missing again. I also found another 2 errors in build log:

[1] ERROR: ld.so: object 'libfakeroot-sysv.so' from LD_PRELOAD cannot be preloaded: ignored.
[2] stat: cannot stat `arch/arm/boot/*.dtb': No such file or directory

I noticed that at the end of build log, it says:
Sending email to: <email address hidden>, <email address hidden>, <email address hidden>
Therefore I assume you have received notification email.

[Naresh Kamboju (naresh-kamboju)]

The project linux-next » snowball,kernel_cloud Build # 180 - Still Failing.
Console output at https://ci.linaro.org/jenkins/job/linux-next/./hwpack=snowball,label=kernel_cloud/180/

[Naresh Kamboju (naresh-kamboju)]

The project linux-next » snowball,kernel_cloud Build # 190 - Still Failing.
Console output at https://ci.linaro.org/jenkins/job/linux-next/./hwpack=snowball,label=kernel_cloud/190/

[Naresh Kamboju (naresh-kamboju)]

linux-next Build # 198 - Still Failing.

console output at https://ci.linaro.org/jenkins/job/linux-next/198/

[Naresh Kamboju (naresh-kamboju)]

 linux-next » snowball,kernel_cloud Build # 224 - Still Failing.
console output at https://ci.linaro.org/jenkins/job/linux-next/./hwpack=snowball,label=kernel_cloud/224/

[Naresh Kamboju (naresh-kamboju)]

linux-next » snowball,kernel_cloud Build # 240 - Still Failing.
Console output at https://ci.linaro.org/jenkins/job/linux-next/./hwpack=snowball,label=kernel_cloud/240/

[Naresh Kamboju (naresh-kamboju)]

linux-next » snowball,kernel_cloud Build # 256 - Still Failing
console output at https://ci.linaro.org/jenkins/job/linux-next/./hwpack=snowball,label=kernel_cloud/256

[Chase Qi (chase-qi)]

Migrated to bugzilla: https://bugs.linaro.org/show_bug.cgi?id=492