build failure caused by java.io.IOException/java.lang.LinkageError

[Fathi Boudra]

Build failure:
Caused by: java.io.IOException
Caused by: java.lang.LinkageError

Build affected:
https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/5/
https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/8/
https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/14/
https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/15/
https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/21/
https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/22/
https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/24/
https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/25/
https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/28/
https://ci.linaro.org/jenkins/job/precise-armhf-alip/7/
https://ci.linaro.org/jenkins/job/precise-armhf-developer/10/
https://ci.linaro.org/jenkins/job/precise-armhf-nano/29/
https://ci.linaro.org/jenkins/job/precise-armhf-ubuntu-desktop/41/
https://ci.linaro.org/jenkins/job/sid-armhf-debian/25/

[Paul Sokolovsky (pfalcon)]

Upstream bug: https://issues.jenkins-ci.org/browse/JENKINS-6604

[Paul Sokolovsky (pfalcon)]

I never saw such exception on https://android-build.linaro.org/jenkins/

Initial suspicion was that these errors happened after installing 4 new plugins at ci.linaro.org, but Fathi pointed to few cases happened before, and after all 4 plugins were disabled, the errors still reoccur.

[Paul Sokolovsky (pfalcon)]

Fathi confirms this still happens on the new ci.*:

https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/31/

[Paul Sokolovsky (pfalcon)]

<fabo> pfalcon: we have done a *manual* to trigger all the builds and it didn't happened
<fabo> +action
<pfalcon> fabo: builds didn't start you mean? or exceptions didn't happen?
<fabo> pfalcon: when we triggered manually, it didn't happened
<fabo> pfalcon: so far it happened when it's triggered by the cron job

[Deepti B. Kalakeri (deeptik)]

Some similar bugs reported by others recently

https://issues.jenkins-ci.org/browse/JENKINS-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Other jobs apart from the Ubuntu builds are also getting affected.

Thanks!!
Deepti.

[Deepti B. Kalakeri (deeptik)]

The job Project precise-armhf-lt-panda was failing previously with the Java exception error.
As mentioned in the link above, I tried to restart the slave and ran the same job again and it passed.
This confirms that the problem is with the slaves and need to tackle with this.

Thanks!!
Deepti.

[Paul Sokolovsky (pfalcon)]

It turns out that https://ci.linaro.org/jenkins/view/Ubuntu%20Build%20Service/job/precise-armhf-lt-panda/ was failing *in row* from build #31 to build #39 (spread over 5 days and of course different slaves). Now, that's worrying.

[Deepti B. Kalakeri (deeptik)]

The problem reported in this bug is still an open and unresolved bug in the upstream jenkins, so not aware exactly how to solve this. For now, changed the number of executors to 1 to see if that works and helps the problem to be fixed.

Thanks!!!
Deepti.

[Deepti B. Kalakeri (deeptik)]

It appears that changing the executors to 1 helped to avoid the problem seen in the bug.
Until we get a fix from the upstream, we plan to follow this workaround.

Thanks!!!
Deepti.

[Deepti B. Kalakeri (deeptik)]

upstream bug https://issues.jenkins-ci.org/browse/JENKINS-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel has been updated with the findings.

[Joey Stanford (joey)]

Dumb question, did you clean out the temp folder as per the upstream bug?

[Deepti B. Kalakeri (deeptik)]

On Wed, Apr 4, 2012 at 10:19 PM, Joey Stanford <email address hidden> wrote:

> Dumb question, did you clean out the temp folder as per the upstream
> bug?
>

We cannot clean the /tmp/ directory because

1) When the jobs start building they mainly use /tmp/ to write some temp
scripts which includes the job instructions so if this gets removed then
job might fail.
2) If 2 builds are running in parallel on the same slave then they might
interfere each other if one tries to remove the things of the others.

>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/959945
>
> Title:
> build failure caused by java.io.IOException/java.lang.LinkageError
>
> Status in Linaro Continuous Integration:
> Fix Committed
>
> Bug description:
> Build failure:
> Caused by: java.io.IOException
> Caused by: java.lang.LinkageError
>
> Build affected:
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/5/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/8/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/14/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/15/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/21/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/22/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/24/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/25/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/28/
> https://ci.linaro.org/jenkins/job/precise-armhf-alip/7/
> https://ci.linaro.org/jenkins/job/precise-armhf-developer/10/
> https://ci.linaro.org/jenkins/job/precise-armhf-nano/29/
> https://ci.linaro.org/jenkins/job/precise-armhf-ubuntu-desktop/41/
> https://ci.linaro.org/jenkins/job/sid-armhf-debian/25/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-ci/+bug/959945/+subscriptions
>

--
Thanks and Regards,
Deepti

[Stevan Radaković (stevanr)]

The proposed fix is in the Jenkins' remoting library, the code is available here https://code.launchpad.net/~linaro-infrastructure/jenkins-remoting/linaro-trunk

For our current version of Jenkins, 1.456 the library version remoting-2.12 is used.
This fix was tested on the clone ci.* instance, https://ec2-184-72-153-99.compute-1.amazonaws.com/jenkins/ and since it didn't break anything we will deploy it to the production and closely observe whether it fixes the problem or not (since this bug could not be reproduced on our test instance under any circumstances).

[Deepti B. Kalakeri (deeptik)]

On Thu, Jun 7, 2012 at 8:39 PM, Stevan Radaković <
<email address hidden>> wrote:

> The proposed fix is in the Jenkins' remoting library, the code is
> available here https://code.launchpad.net/~linaro-infrastructure
> /jenkins-remoting/linaro-trunk
>
> For our current version of Jenkins, 1.456 the library version
> remoting-2.12 is used.
> This fix was tested on the clone ci.* instance,
> https://ec2-184-72-153-99.compute-1.amazonaws.com/jenkins/ and since it
> didn't break anything we will deploy it to the production and closely
> observe whether it fixes the problem or not (since this bug could not be
> reproduced on our test instance under any circumstances).
>

Ah! nice. Could you point me to the job which was used to test this fix ?

>
> --
> You received this bug notification because you are subscribed to Linaro
> CI.
> https://bugs.launchpad.net/bugs/959945
>
> Title:
> build failure caused by java.io.IOException/java.lang.LinkageError
>
> Status in Linaro Continuous Integration:
> Fix Released
> Status in Linaro Jenkins:
> In Progress
>
> Bug description:
> Build failure:
> Caused by: java.io.IOException
> Caused by: java.lang.LinkageError
>
> Build affected:
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/5/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/8/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/14/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/15/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/21/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/22/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/24/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/25/
> https://ci.linaro.org/jenkins/job/precise-armhf-lt-panda/28/
> https://ci.linaro.org/jenkins/job/precise-armhf-alip/7/
> https://ci.linaro.org/jenkins/job/precise-armhf-developer/10/
> https://ci.linaro.org/jenkins/job/precise-armhf-nano/29/
> https://ci.linaro.org/jenkins/job/precise-armhf-ubuntu-desktop/41/
> https://ci.linaro.org/jenkins/job/sid-armhf-debian/25/
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/linaro-ci/+bug/959945/+subscriptions
>

--
Thanks and Regards,
Deepti