Ubuntu Build Service jobs stuck in the build queue

[Fathi Boudra]

Since yesterday (2012-03-15), UBS[1] jobs are stuck in the build queue because they can't get a node (they're all offline).

It's seems caused by the main Ubuntu archive for Precise. As a result, we can't download packages and setup the node.
Could you take a look? Also, ensure we're proactive on those connectivity issues and get notified (timeout available?).

[1] https://ci.linaro.org/jenkins/view/Ubuntu%20Build%20Service/

[Deepti B. Kalakeri (deeptik)]

Ah! Fathi. The reason for your builds getting queued is not because of the packages non availability this time.
Its because nodes are not provisioned because the Instance Cap of 20 instances were reached

Here is the jenkins log information:
INFO: Instance cap reached, not provisioning.
Mar 16, 2012 7:50:07 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Instance cap reached, not provisioning.
Mar 16, 2012 7:50:17 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Instance cap reached, not provisioning.
Mar 16, 2012 7:50:17 AM hudson.plugins.ec2.EC2Cloud provision
INFO: Instance cap reached, not provisioning.

There seems to be high usages of ec2 instances at the same time because of which we are facing this problem.
I could only wish we increased the instance cap of ec2 instances now across linaro from 20 to something higher.
Until we have this cap limit we will tend to get such problems often.

Another way to reduce this is provisioning your builds at different intervals so that they dont get stuck in the queue at the same time. Between I try to use more executors on the instances so that more parallel builds can work on the same slave.
Hope that will help to solve the problem to some extent.
Could you reduce the severity of the problem to medium?

Thanks!!!
Deepti.

[Deepti B. Kalakeri (deeptik)]

Fathi,

Do you still see the problem of builds piling up because they are not able to get the node ?

[Fathi Boudra (fboudra)]

We've seen this issue last week, during 12.04 release week. Since, I haven't seen the issue.

[Deepti B. Kalakeri (deeptik)]

As mentioned by Fathi, the problem seems to happen during the release week and was requested to wait to see if it reappears during 12.05 release time before we close this bug.

Thanks!!
Deepti.

[Deepti B. Kalakeri (deeptik)]

I believe that there was no build queuing up during the release week this time.
Fathi, let me know if it is not the same.

Otherwise I would like to close this bug and reopen if it reappears or should we continue to monitor for one more release month? If we need to monitor for one more month , please change the milestone information of the bug.

[Deepti B. Kalakeri (deeptik)]

We did not see build queue up reappear during the release week of 12.05 and 12.06. Hence closing this bug.
If it appears we will re open the same.