mir builds seem to be taking an eternity

[kevin gunn]

we were previously around 20 to 30 minutes for ci runs.
but now we are taking multiple hours.
can anything be done ?

[kevin gunn (kgunn72)]

https://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-amd64-ci/486/ took an hour to run
 https://jenkins.qa.ubuntu.com/job/mir-team-mir-development-branch-trusty-armhf-ci/490/? took an hour and a half
 Mir just isn't that big

[Paul Larson (pwlars)]

I'm not really familiar with this job or how it's set up, but it is on a calxeda host. Any chance we've been given less to work with on these nodes? The build I'm watching in progress right now seems to be mostly just taking a long time on the compile.

[Chris Johnston (cjohnston)]

How long ago was this taking only 30 minutes?

[Paul Larson (pwlars)]

I talked to fginther about this, and he told me:
"they only thing that we've changed in the past month is moving the armhf builds from raring to saucy. As a result of this a we are still down 3 armhf nodes which still need to be updated. Getting these other three up again should improve things a little bit, as the builds should spread out a bit."
He also mentioned that he's working on getting the other armhf nodes this week, and is hopeful this should improve things a bit.
Francis, can you update this when those are back online?

[Francis Ginther (fginther)]

2 more armhf nodes have been updated and are back in the pool.

[Alexandros Frantzis (afrantzis)]

We could potentially get a big boost if we could manage to use ccache for the builds, since in most cases only a small number of files change. I have no idea if/how this could be implemented in our builders, but I think it's worth investigating.

[Alan Griffiths (alan-griffiths)]

I'm not sure what the limiting factor is, but compilation are mentioned above.

One approach to speeding compiles is to distribute them. As the compile nodes don't need to have all the dependencies installed for each build (just a suitable compiler) they can be shared effectively. E.g. https://code.google.com/p/distcc/

[Alan Griffiths (alan-griffiths)]

@"How long ago was this taking only 30 minutes?"

This was quite some time ago - certainly before Christmas. It has been getting steadily worse since early December.

Here's a build from last August that took 12 min

    https://jenkins.qa.ubuntu.com/job/mir-android-saucy-i386-build/1767/?

[Francis Ginther (fginther)]

The most recent version of the mir-android build took 11 minutes:
https://jenkins.qa.ubuntu.com/job/mir-android-trusty-i386-build/914/

But this lead me to look for the 'saucy' version for the other jobs and I found a comparison:

https://jenkins.qa.ubuntu.com/job/mir-saucy-armhf-ci/103/ == 30 minutes
https://jenkins.qa.ubuntu.com/job/mir-trusty-armhf-ci/35/ == 94 minutes

So there is data to support these builds used to take a *lot* less time.

Exploring alternative compilation methods such as distcc is completely out of scope at this time. Ccache might be a little more attainable as we've used it in the past. There may be something off in the trusty pbuilders, it's possible the slowness started with that transistion. That's something I can try to investigate.

[Alexander Sack (asac)]

could we add a few timestamp echos in log for the various steps we do in such a build so we know what takes so much time now? If we are lucky we can spot something that stands out and investigate.

[kevin gunn (kgunn72)]

Just adding some information recently collected & discussed in our team
=======================================
I also notice that we appear to build for utopic-armhf twice (in
parallel). Vis:

http://s-jenkins.ubuntu-ci:8080/job/mir-team-mir-development-branch-utopic-armhf-ci/316/consoleText
and

http://s-jenkins.ubuntu-ci:8080/job/generic-mediumtests-builder-utopic-armhf/1195/consoleText

As these are (by far) the two slowest jobs we could halve our impact on
the infrastructure if we could de-duplicating them.
========================================

wondering if this is something that could easily be addressed