Launchpad Buildd doesn't clean up old respective build dir

[Celso Providelo]

Some odd occurs in launchpad-buildd when it leaves a broken build directory (it's already a bug , it should be removed anyway). But when requested to build the same package it crashes as:

2006/02/21 04:31 GMT [HTTPChannel,452,82.211.81.167] Traceback (most recent call last):
          File "/usr/lib/python2.4/site-packages/twisted/protocols/http.py", line 557, in requestReceived
            self.process()
          File "/usr/lib/python2.4/site-packages/twisted/web/server.py", line 165, in process
            self.render(resrc)
          File "/usr/lib/python2.4/site-packages/twisted/web/server.py", line 172, in render
            body = resrc.render(self)
          File "/usr/lib/python2.4/site-packages/twisted/web/xmlrpc.py", line 130, in render
            defer.maybeDeferred(function, *args).addErrback(
        --- <exception caught here> ---
          File "/usr/lib/python2.4/site-packages/twisted/internet/defer.py", line 129, in maybeDeferred
            result = f(*args, **kw)
          File "/usr/share/launchpad-buildd/canonical/buildd/slave.py", line 484, in xmlrpc_build
            self.slave.manager.initiate(filemap, chrootsum, args)
          File "/usr/share/launchpad-buildd/canonical/buildd/debian.py", line 70, in initiate
            BuildManager.initiate(self, files, chroot, extra_args)
          File "/usr/share/launchpad-buildd/canonical/buildd/slave.py", line 115, in initiate
            os.mkdir("%s/build-%s" % (os.environ["HOME"], self._buildid))
        exceptions.OSError: [Errno 17] File exists: '/home/buildd/build-170168-4809'

[Celso Providelo (cprov)]

It blocks the bulder in a very unusual way, people take a lot of time to figure out it's broken.

[Adam Conrad (adconrad)]

I'll be adding a cleanup method, and tying it into startup and the abort methods to make sure this doesn't happen anymore.

[Celso Providelo (cprov)]

Great, thanks.

Don't forget to send me the patch, since it's still maintained as RF code.

[Celso Providelo (cprov)]

I believe this issue is already fixed in production.

[Adam Conrad (adconrad)]

This is definitely not fixed in production.

[Robert Collins (lifeless)]

Does this happen often?

[Julian Edwards (julian-edwards)]

I have never seen this happen.

[William Grant (wgrant)]

lamont complains about it often enough, so it does happen.

[Julian Edwards (julian-edwards)]

Doesn't the slave.clean() method clear up the crap?

[William Grant (wgrant)]

Yes, when it's called. But it isn't called when the builder dies or the master gets really confused.

[Julian Edwards (julian-edwards)]

On Friday 19 November 2010 11:31:22 you wrote:
> Yes, when it's called. But it isn't called when the builder dies or the
> master gets really confused.

Ok, we need to get rid of all that logic in the master and make the slave do
it when we start a new build.

This will need a bit of a protocol change as we currently do:

 * send file
 * send file
 ....
 * startbuild

Alternatively we can remove the slave clean call from the behavioral methods
and just do it once at the start of the build, equivalent to resuming a
virtual host.

[Robert Collins (lifeless)]

Doing it just in time before a build makes a lot of sense to me

On 20/11/2010 1:11 AM, "Julian Edwards" <email address hidden> wrote:

On Friday 19 November 2010 11:31:22 you wrote:
> Yes, when it's called. But it isn't called when the...
Ok, we need to get rid of all that logic in the master and make the slave do
it when we start a new build.

This will need a bit of a protocol change as we currently do:

 * send file
 * send file
 ....
 * startbuild

Alternatively we can remove the slave clean call from the behavioral methods
and just do it once at the start of the build, equivalent to resuming a
virtual host.

--
Launchpad Buildd doesn't clean up old respective build dir
https://bugs.launchpad.net/bugs/3223...

[LaMont Jones (lamont)]

The most common occurances are:

1) launchpad-buildd gets upgraded to a new version, in the middle of a build.
2) the machine crashes mid-build (most notably armel)

#1 is a function of the development cycle on launchpad-buildd, #2 is just a function of builds overloading the poor hardware.

[Robert Collins (lifeless)]

We upgrade mid-builds? Cool. Scary. Cool.