Retry file fetches from build slave

Bug #1508433 reported by Colin Watson
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Expired
High
Unassigned

Bug Description

https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test/+build/8146556 finished its first (long) build successfully, but failed to fetch one of the files, and so gave up. The buildd-manager logs look like this (in part):

2015-10-21 11:26:17+0000 [QueryProtocol,client] Processing finished job PACKAGEBUILD-8146556 (amd64 build of gcc-5 5.2.1-22ubuntu3 in ubuntu wily RELEASE) from builder lgw01-08: OK
2015-10-21 11:26:17+0000 [QueryProtocol,client] Starting factory <HTTPDownloader: http://lgw01-08.lgw01.scalingstack.ppa:8221/filecache/buildlog>
2015-10-21 11:26:38+0000 [HTTPPageDownloader,client] Starting factory <HTTPDownloader: http://lgw01-08.lgw01.scalingstack.ppa:8221/filecache/ec59fc8f5c9d513b9085a8f10a03bafff8620d94>
2015-10-21 11:26:38+0000 [HTTPPageDownloader,client] Starting factory <HTTPDownloader: http://lgw01-08.lgw01.scalingstack.ppa:8221/filecache/3bb4f03001411c9baa5b946e725d2e63c6be9d45>
[lots more of the same]
2015-10-21 11:26:39+0000 [HTTPPageDownloader,client] Stopping factory <HTTPDownloader: http://lgw01-08.lgw01.scalingstack.ppa:8221/filecache/buildlog>
2015-10-21 11:26:56+0000 [HTTPPageDownloader,client] Stopping factory <HTTPDownloader: http://lgw01-08.lgw01.scalingstack.ppa:8221/filecache/856260abfec2057543c972b64ef339f3f6e9cbb1>
2015-10-21 11:26:56+0000 [HTTPPageDownloader,client] Stopping factory <HTTPDownloader: http://lgw01-08.lgw01.scalingstack.ppa:8221/filecache/33805f15f6e46828ef920975be356a038edb966d>
[lots more of the same]
2015-10-21 11:26:58+0000 [HTTPPageDownloader,client] Scanning lgw01-08 failed with: FirstError[#62, [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
2015-10-21 11:26:58+0000 [HTTPPageDownloader,client] ]]
2015-10-21 11:26:58+0000 [HTTPPageDownloader,client] Traceback (most recent call last):
2015-10-21 11:26:58+0000 [HTTPPageDownloader,client] Failure: twisted.internet.defer.FirstError: FirstError[#62, [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion.
2015-10-21 11:26:58+0000 [HTTPPageDownloader,client] ]]
2015-10-21 11:26:58+0000 [HTTPPageDownloader,client]
2015-10-21 11:26:58+0000 [HTTPPageDownloader,client] Judged builder lgw01-08 (3 failures) with job PACKAGEBUILD-8146556 (3 failures): None, True
2015-10-21 11:26:58+0000 [HTTPPageDownloader,client] Requeueing job PACKAGEBUILD-8146556.

This looks like a temporary glitch that might well have been fixed if only we'd tried a couple more times. I think BuilderSlave.getFile could do with some retry logic.

Tags: soyuz-build
Colin Watson (cjwatson)
Changed in launchpad:
importance: Undecided → High
status: New → Triaged
tags: added: soyuz-build
Revision history for this message
William Grant (wgrant) wrote :

We retry the scan three times, it just happened that in this case there were two widespread network issues earlier in the build that caused the failure here to be fatal. We could implement extra retry logic for each download, but I'm not sure it's worth it.

2015-10-21 03:31:16+0000 [-] Scanning lgw01-08 failed with:
2015-10-21 03:31:49+0000 [Uninitialized] Scanning lgw01-08 failed with: An error occurred while connecting: 113: No route to host.

Changed in launchpad:
status: Triaged → Incomplete
michelle (mitchy-72)
Changed in launchpad:
status: Incomplete → New
Colin Watson (cjwatson)
Changed in launchpad:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for Launchpad itself because there has been no activity for 60 days.]

Changed in launchpad:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.