gate error: Looks like the node went offline during the build. Check the slave log for the details.FATAL: /var/lib/jenkins/logs/slaves/*/slave.log (No such file or directory)

Bug #1284371 reported by Joe Gordon
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack-Gate
Fix Released
High
James E. Blair

Bug Description

Seen in the gate

Looks like the node went offline during the build. Check the slave log for the details.FATAL: /var/lib/jenkins/logs/slaves/bare-precise-hpcloud-az2-1572924/slave.log (No such file or directory)
java.io.FileNotFoundException: /var/lib/jenkins/logs/slaves/bare-precise-hpcloud-az2-1572924/slave.log (No such file or directory)
  at java.io.RandomAccessFile.open(Native Method)
  at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
  at org.kohsuke.stapler.framework.io.LargeText$FileSession.<init>(LargeText.java:397)
  at org.kohsuke.stapler.framework.io.LargeText$2.open(LargeText.java:120)
  at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:210)
  at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:159)
  at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:579)
  at hudson.model.Run.execute(Run.java:1665)
  at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
  at hudson.model.ResourceController.execute(ResourceController.java:88)
  at hudson.model.Executor.run(Executor.java:246)

This failure is making it all the way to gerrit (https://review.openstack.org/#/c/75918/)

http://logs.openstack.org/18/75918/1/check/gate-nova-python27/9c26936/console.html

message:" Looks like the node went offline during the build" AND message:"slave.log \(No such file or directory\)" AND filename:"console.html"

Tags: gate-failure
James E. Blair (corvus)
tags: added: gate-failure
Revision history for this message
Joe Gordon (jogo) wrote :

23:30 < jeblair> jog0, zaro: the incidences of this in the jenkins log strongly
                 correlate with the gearman plugin de-registering jobs
23:30 < jeblair> jog0, zaro: it's possible it's more complicated than that, but it's
                 worth a look.
23:31 < jeblair> jog0, zaro: http://paste.openstack.org/show/69188/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/76043

Revision history for this message
Daneyon Hansen (danehans) wrote :

I have just been affected by this bug. Here is a snippet from the jenkins logs:

2014-02-25 19:55:28.656 | Finished in 14.72 seconds
2014-02-25 19:55:28.656 | 240 examples, 0 failures
2014-02-25 19:55:28.907 | Looks like the node went offline during the build. Check the slave log for the details.FATAL: /var/lib/jenkins/logs/slaves/bare-precise-hpcloud-az2-1649626/slave.log (No such file or directory)
2014-02-25 19:55:28.907 | java.io.FileNotFoundException: /var/lib/jenkins/logs/slaves/bare-precise-hpcloud-az2-1649626/slave.log (No such file or directory)
2014-02-25 19:55:28.907 | at java.io.RandomAccessFile.open(Native Method)
2014-02-25 19:55:28.907 | at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
2014-02-25 19:55:28.907 | at org.kohsuke.stapler.framework.io.LargeText$FileSession.<init>(LargeText.java:397)
2014-02-25 19:55:28.908 | at org.kohsuke.stapler.framework.io.LargeText$2.open(LargeText.java:120)
2014-02-25 19:55:28.908 | at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:210)
2014-02-25 19:55:28.908 | at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:159)
2014-02-25 19:55:28.908 | at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:579)
2014-02-25 19:55:28.908 | at hudson.model.Run.execute(Run.java:1665)
2014-02-25 19:55:28.908 | at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
2014-02-25 19:55:28.908 | at hudson.model.ResourceController.execute(ResourceController.java:88)
2014-02-25 19:55:28.908 | at hudson.model.Executor.run(Executor.java:246)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to elastic-recheck (master)

Reviewed: https://review.openstack.org/76043
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=0abf712fb4dc693ed318f051978549a3142ef9b7
Submitter: Jenkins
Branch: master

commit 0abf712fb4dc693ed318f051978549a3142ef9b7
Author: Joe Gordon <email address hidden>
Date: Mon Feb 24 15:37:04 2014 -0800

    Add fingerprint for bug 1284371

    bug 1284371 is an infra bug.

    Change-Id: Id000cb37192c67616e208ba81f0c7e1fc39e534e
    Related-Bug: #1284371

Revision history for this message
davidlenwell (dlenwell) wrote :

I am having this same issue on .. https://review.openstack.org/#/c/78683/

 Looks like the node went offline during the build. Check the slave log for the details.FATAL: /var/lib/jenkins/logs/slaves/bare-precise-hpcloud-az2-2246223/slave.log (No such file or directory)
2014-03-06 18:06:59.227 | java.io.FileNotFoundException: /var/lib/jenkins/logs/slaves/bare-precise-hpcloud-az2-2246223/slave.log (No such file or directory)
2014-03-06 18:06:59.227 | at java.io.RandomAccessFile.open(Native Method)
2014-03-06 18:06:59.227 | at java.io.RandomAccessFile.<init>(RandomAccessFile.java:241)
2014-03-06 18:06:59.227 | at org.kohsuke.stapler.framework.io.LargeText$FileSession.<init>(LargeText.java:397)
2014-03-06 18:06:59.227 | at org.kohsuke.stapler.framework.io.LargeText$2.open(LargeText.java:120)
2014-03-06 18:06:59.227 | at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:210)
2014-03-06 18:06:59.227 | at hudson.console.AnnotatedLargeText.writeHtmlTo(AnnotatedLargeText.java:159)
2014-03-06 18:06:59.227 | at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:579)
2014-03-06 18:06:59.227 | at hudson.model.Run.execute(Run.java:1665)
2014-03-06 18:06:59.227 | at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
2014-03-06 18:06:59.228 | at hudson.model.ResourceController.execute(ResourceController.java:88)
2014-03-06 18:06:59.228 | at hudson.model.Executor.run(Executor.java:246)

James E. Blair (corvus)
Changed in openstack-ci:
status: New → Triaged
importance: Undecided → High
milestone: none → icehouse
Revision history for this message
Tom Fifield (fifieldt) wrote :
Jeremy Stanley (fungi)
Changed in openstack-ci:
milestone: icehouse → kilo
Jeremy Stanley (fungi)
Changed in openstack-gate:
status: New → Triaged
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to elastic-recheck (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/309724

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to elastic-recheck (master)

Reviewed: https://review.openstack.org/309724
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=15eb9df23024e7012a988b84c219d5fba33df770
Submitter: Jenkins
Branch: master

commit 15eb9df23024e7012a988b84c219d5fba33df770
Author: Matt Riedemann <email address hidden>
Date: Sat Apr 23 17:51:44 2016 -0400

    Update fingerprint for bug 1284371

    There is a new message for this bug so add that to the
    existing fingerprint. This started spiking on 4/20.

    81 hits in voting jobs, check and gate, all branches.

    Change-Id: Ic7915078064bd1bad50e7f27721f3f202dab9691
    Related-Bug: #1284371

Revision history for this message
Clark Boylan (cboylan) wrote :

This was fixed by stopping our use of Jenkins.

Changed in openstack-ci:
assignee: nobody → James E. Blair (corvus)
Changed in openstack-gate:
assignee: nobody → James E. Blair (corvus)
status: Triaged → Fix Released
James E. Blair (corvus)
no longer affects: openstack-ci
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.