Activity log for bug #646635

Date Who What changed Old value New value Message
2010-09-24 07:43:50 Michael Nelson bug added bug
2010-09-24 07:47:38 Michael Nelson description A number of times over the past few days, people have been complaining that builds are being bounced from one builder to another. maxb reported this morning that issues with a build of bzr. Checking the logs shows the builder dispatching fine, and then: 2010-09-24 07:00:49+0000 [-] <americium:http://americium.ppa:8221/> communication failed (User timeout caused connection failure.) Checking the history for this builder shows that, although it's still taking on new builds, the last build to finish was over 3 hrs ago. Looking for similar errors in the logs results in quite a few buildds: 2010-09-24 07:31:13+0000 [-] <actinium:http://actinium.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:13+0000 [-] <actinium:http://actinium.ppa:8221/> failure (None) 2010-09-24 07:31:13+0000 [-] <einsteinium:http://einsteinium.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:13+0000 [-] <einsteinium:http://einsteinium.ppa:8221/> failure (None) 2010-09-24 07:31:53+0000 [-] <cushaw:http://cushaw.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:53+0000 [-] <cushaw:http://cushaw.buildd:8221/> failure (None) 2010-09-24 07:31:53+0000 [-] <hawthorn:http://hawthorn.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:53+0000 [-] <hawthorn:http://hawthorn.buildd:8221/> failure (None) 2010-09-24 07:31:54+0000 [-] <allspice:http://allspice.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:54+0000 [-] <allspice:http://allspice.buildd:8221/> failure (None) 2010-09-24 07:31:54+0000 [-] <adare:http://adare.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:54+0000 [-] <adare:http://adare.buildd:8221/> failure (None) 2010-09-24 07:31:54+0000 [-] <gourd:http://gourd.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:54+0000 [-] <gourd:http://gourd.buildd:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <palmer:http://palmer.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:57+0000 [-] <palmer:http://palmer.buildd:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <genip:http://genip.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:57+0000 [-] <genip:http://genip.buildd:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <crested:http://crested.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:57+0000 [-] <crested:http://crested.buildd:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <plutonium:http://plutonium.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:57+0000 [-] <plutonium:http://plutonium.ppa:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <nannyberry:http://nannyberry.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:57+0000 [-] <nannyberry:http://nannyberry.ppa:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <mercury:http://mercury.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:58+0000 [-] <mercury:http://mercury.ppa:8221/> failure (None) 2010-09-24 07:31:58+0000 [-] <lakoocha:http://lakoocha.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:58+0000 [-] <lakoocha:http://lakoocha.ppa:8221/> failure (None) etc. A number of times over the past few days, people have been complaining that builds are being bounced from one builder to another. maxb reported this morning that issues with a build of bzr. Checking the logs shows the builder dispatching fine, and then: 2010-09-24 07:00:49+0000 [-] <americium:http://americium.ppa:8221/> communication failed (User timeout caused connection failure.) Checking the history for this builder shows that, although it's still taking on new builds, the last build to finish was over 3 hrs ago. Looking for similar errors in the logs results in quite a few buildds, shown below. I checked a few of these buildd's histories, and they all have a large gap of 3-4 hours where they didn't finish anything - as if none of them could communicate the results back. 2010-09-24 07:31:13+0000 [-] <actinium:http://actinium.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:13+0000 [-] <actinium:http://actinium.ppa:8221/> failure (None) 2010-09-24 07:31:13+0000 [-] <einsteinium:http://einsteinium.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:13+0000 [-] <einsteinium:http://einsteinium.ppa:8221/> failure (None) 2010-09-24 07:31:53+0000 [-] <cushaw:http://cushaw.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:53+0000 [-] <cushaw:http://cushaw.buildd:8221/> failure (None) 2010-09-24 07:31:53+0000 [-] <hawthorn:http://hawthorn.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:53+0000 [-] <hawthorn:http://hawthorn.buildd:8221/> failure (None) 2010-09-24 07:31:54+0000 [-] <allspice:http://allspice.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:54+0000 [-] <allspice:http://allspice.buildd:8221/> failure (None) 2010-09-24 07:31:54+0000 [-] <adare:http://adare.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:54+0000 [-] <adare:http://adare.buildd:8221/> failure (None) 2010-09-24 07:31:54+0000 [-] <gourd:http://gourd.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:54+0000 [-] <gourd:http://gourd.buildd:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <palmer:http://palmer.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:57+0000 [-] <palmer:http://palmer.buildd:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <genip:http://genip.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:57+0000 [-] <genip:http://genip.buildd:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <crested:http://crested.buildd:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:57+0000 [-] <crested:http://crested.buildd:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <plutonium:http://plutonium.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:57+0000 [-] <plutonium:http://plutonium.ppa:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <nannyberry:http://nannyberry.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:57+0000 [-] <nannyberry:http://nannyberry.ppa:8221/> failure (None) 2010-09-24 07:31:57+0000 [-] <mercury:http://mercury.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:58+0000 [-] <mercury:http://mercury.ppa:8221/> failure (None) 2010-09-24 07:31:58+0000 [-] <lakoocha:http://lakoocha.ppa:8221/> communication failed (User timeout caused connection failure.) 2010-09-24 07:31:58+0000 [-] <lakoocha:http://lakoocha.ppa:8221/> failure (None) etc.
2010-09-24 10:23:33 Julian Edwards affects launchpad-buildd soyuz
2010-09-24 10:23:53 Julian Edwards soyuz: status New Triaged
2010-09-24 10:23:56 Julian Edwards soyuz: importance Undecided High
2010-09-24 10:23:58 Julian Edwards soyuz: assignee Julian Edwards (julian-edwards)
2010-09-24 10:24:05 Julian Edwards tags buildd-manager
2010-11-16 17:54:12 Julian Edwards branch linked lp:~julian-edwards/launchpad/builderslave-resume
2010-11-16 17:54:30 Julian Edwards soyuz: status Triaged Fix Released
2010-11-16 17:55:15 Julian Edwards soyuz: milestone 10.11