hudson.Launcher exception causing build failures

[Davanum Srinivas (DIMS)]

Example:
http://logs.openstack.org/09/60209/2/gate/gate-tempest-dsvm-neutron/74597c9/console.html

Logstash query:
"at hudson.Launcher\$RemoteLauncher.launch\(Launcher.java\:864\)" AND build_status:"FAILURE"

Logstash url (226 hits in last 48 hours):
http://logstash.openstack.org/#eyJzZWFyY2giOiJcImF0IGh1ZHNvbi5MYXVuY2hlclxcJFJlbW90ZUxhdW5jaGVyLmxhdW5jaFxcKExhdW5jaGVyLmphdmFcXDo4NjRcXClcIiBBTkQgYnVpbGRfc3RhdHVzOlwiRkFJTFVSRVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg2ODUyNzQyMDMyLCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

Stack trace:
2013-12-12 07:48:49.029 | FATAL: command execution failed
2013-12-12 07:48:49.030 | java.io.InterruptedIOException
2013-12-12 07:48:49.030 | at hudson.Launcher$RemoteLauncher.launch(Launcher.java:864)
2013-12-12 07:48:49.030 | at hudson.Launcher$ProcStarter.start(Launcher.java:353)
2013-12-12 07:48:49.030 | at hudson.Launcher$ProcStarter.join(Launcher.java:360)
2013-12-12 07:48:49.030 | at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:91)
2013-12-12 07:48:49.031 | at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:60)
2013-12-12 07:48:49.031 | at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
2013-12-12 07:48:49.031 | at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
2013-12-12 07:48:49.031 | at hudson.model.Build$BuildExecution.build(Build.java:199)
2013-12-12 07:48:49.031 | at hudson.model.Build$BuildExecution.doRun(Build.java:160)
2013-12-12 07:48:49.032 | at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:586)
2013-12-12 07:48:49.032 | at hudson.model.Run.execute(Run.java:1593)
2013-12-12 07:48:49.032 | at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
2013-12-12 07:48:49.032 | at hudson.model.ResourceController.execute(ResourceController.java:88)
2013-12-12 07:48:49.032 | at hudson.model.Executor.run(Executor.java:247)
2013-12-12 07:48:49.033 | Caused by: java.lang.InterruptedException
2013-12-12 07:48:49.033 | at java.lang.Object.wait(Native Method)
2013-12-12 07:48:49.033 | at hudson.remoting.Request.call(Request.java:146)
2013-12-12 07:48:49.033 | at hudson.remoting.Channel.call(Channel.java:713)
2013-12-12 07:48:49.034 | at hudson.Launcher$RemoteLauncher.launch(Launcher.java:862)
2013-12-12 07:48:49.034 | ... 13 more
2013-12-12 07:48:55.511 | Build step 'Execute shell' marked build as failure

[Jeremy Stanley (fungi)]

Hard to diagnose, but it's probably a case of the vm going offline after being registered as a Jenkins slave but prior to a job getting assigned.

[Jeremy Stanley (fungi)]

Setting to low unless anyone has any ideas for how to fix this, besides the obvious "stop using Jenkins."

[Davanum Srinivas (DIMS) (dims-v)]

@fungi, could we move to 1.532.1 (2013/11/25) their long term support release (http://jenkins-ci.org/changelog-stable) and see if that helps?

(731 hits in last 48 hours)

[Davanum Srinivas (DIMS) (dims-v)]

@fungi, I was looking at another stack trace
http://logs.openstack.org/21/66221/1/gate/gate-python-heatclient-python26/23cdc0c/console.html

2014-01-17 03:18:13.094 | at hudson.model.Executor.run(Executor.java:247)
2014-01-17 03:18:13.094 | Caused by: java.lang.NoClassDefFoundError: Could not initialize class jenkins.model.Jenkins$MasterComputer
2014-01-17 03:18:13.094 | at hudson.Launcher$LocalLauncher.<init>(Launcher.java:755)
2014-01-17 03:18:13.094 | at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:991)
2014-01-17 03:18:13.094 | at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:965)
2014-01-17 03:18:13.094 | at hudson.remoting.UserRequest.perform(UserRequest.java:118)
2014-01-17 03:18:13.095 | at hudson.remoting.UserRequest.perform(UserRequest.java:48)
2014-01-17 03:18:13.095 | at hudson.remoting.Request$2.run(Request.java:326)
2014-01-17 03:18:13.095 | at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
2014-01-17 03:18:13.095 | at java.util.concurrent.FutureTask.run(FutureTask.java:262)
2014-01-17 03:18:13.095 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2014-01-17 03:18:13.096 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
2014-01-17 03:18:13.096 | at java.lang.Thread.run(Thread.java:744)

Which is clearly fixed already in Jenkins specifically in version 1.540 (2013/11/17)
https://issues.jenkins-ci.org/browse/JENKINS-19453

and the "long term support release" 1.532.1 (2013/11/25) does not have it backported yet.

So i change my recommendation to move to latest release 1.547 (2014/01/12) - http://jenkins-ci.org/changelog

[Jeremy Stanley (fungi)]

We're tracking that other stack trace pattern from comment #4 in bug 1267364, and are seeing it even on 1.543 (where that issue is supposedly fixed), so either we have a different bug on that one or they didn't fix it/regressed. Anyway, it seems to be unrelated to the stack trace in the bug description so please let's keep these issues separate (they likely have no more in common with one another than two random Python tracebacks would).

[James E. Blair (corvus)]

The original report for this bug linked to a build that was not reported to Gerrit. It was likely an intermediate build that Zuul aborted. That's not a bug as it does not represent a final result. If you see Zuul report a failed build to Gerrit with this, that is a bug.