Normal builds start to fail due to build timeout

[Paul Sokolovsky]

We had issues with some new jobs (e.g. gcc-trunk) hitting our build timeout limit, which I had to adjust, but today we had few standard build aborted due to it:

https://android-build.linaro.org/jenkins/job/linaro-android_origen-ics-gcc47-samsunglt-stable-blob/22/console
https://android-build.linaro.org/jenkins/job/linaro-android_origen-ics-gcc47-samsunglt-tracking-blob/21/console
https://android-build.linaro.org/jenkins/job/linaro-android_panda-ics-gcc44-kwg-upstream-open/151/console

Looks, like we need to bump timeout up from current 2.5hrs.

[Paul Sokolovsky (pfalcon)]

Ok, bumped build timeout to 170 mins.

[Paul Sokolovsky (pfalcon)]

We're still having few builds fail even with 170 mins (couple I saw were failing at publisher step). Before blindly bumping timeout even further, would be nice to figure out why we have longer runtimes for them (because we still have builds running 2:10 for example).

[Paul Sokolovsky (pfalcon)]

Ok, here're few examples:

https://android-build.linaro.org/jenkins/job/doanac_tracking-panda-linaro4.6/9/
Total time: 2:51:00 (aborted per timeout)

Build stage durations:
real 11m37.175s
real 8m24.365s
real 8717.50 (2.42h)

1st time - seed tarball download+expand time, 2nd - repo sync time, 3rd - build time

https://android-build.linaro.org/jenkins/job/doanac_tracking-panda-linaro4.6/10/?
Total time: 2:01:00
real 9m4.020s
real 4m46.267s
real 5877.23 (1.63h)

[Paul Sokolovsky (pfalcon)]

Another example:

https://android-build.linaro.org/jenkins/job/linaro-android_vexpress-ics-gcc47-armlt-tracking-open-12.05-release/4/
Total time: 2:51:00 (aborted per timeout)

real 23m6.947s
real 11m20.535s
real 6824.17

https://android-build.linaro.org/jenkins/job/linaro-android_vexpress-ics-gcc47-armlt-tracking-open-12.05-release/5/
Total time: 2:06:00

real 8m44.342s
real 11m54.440s
real 5750.48

[Paul Sokolovsky (pfalcon)]

I.e., the slowdown can't be attributed to just one stage (like seed download/update) - we just had time, when all stages of builds process ran slower than usual, leading to build timeouts. Then, it went back to normal (2hr + several minutes, our usual "normal" time). This hard to attribute to anything else than EC2 disturbances that happen from time to time.

Proposed action: bump timeout extra 10 mins (to 3hrs) to account currently known "worst" cases (both cases above timed out during artifact transfer, so 10 mins will make difference).

[Paul Sokolovsky (pfalcon)]

Bumped, and closing.