recipe builds can use too much memory

[Aaron Bentley]

It is believed that excess memory use was a factor in https://wiki.canonical.com/IncidentReports/2010-10-28-LP-build-manager-not-dispatching and https://wiki.canonical.com/IncidentReports/2010-11-17-buildd-manager-disabling-builders

We believe that excessive memory use drove the builders so far into swap that they became unresponsive.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r11943 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/11943>.

[Dimitri John Ledkov (xnox)]

Is "bzr: out of memory" during recipe build is a new feature introduced by this fix?

E.g. https://code.launchpad.net/~mingw-w64/+recipe/gcc4.4+trunk/+build/8623

[Aaron Bentley (abentley)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/24/2010 01:27 AM, Dmitrijs Ledkovs wrote:
> Is "bzr: out of memory" during recipe build is a new feature introduced
> by this fix?

Yes, more or less. We are in the unfortunate position that bzr uses
excessive memory with large branches and the build farm (not just a
single builder) appears to become unstable when excessive memory is used.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzstDwACgkQ0F+nu1YWqI0uOQCdELFMtio4W8S1a6w3tZ0t2889
lTYAnRSYU8vHoXcDgvQGV3FU8R1VXB8P
=rpGW
-----END PGP SIGNATURE-----

[Martin Pool (mbp)]

On 24 November 2010 17:44, Aaron Bentley <email address hidden> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 11/24/2010 01:27 AM, Dmitrijs Ledkovs wrote:
>> Is "bzr: out of memory" during recipe build is a new feature introduced
>> by this fix?

We're working on reducing bzr memory usage through other bugs,
separate from this. One way to help is to attach a traceback showing
where bzr stopped when it ran out of memory.

@aaron maybe bzr should be run with -Derror in the builders?

--
Martin

[Aaron Bentley (abentley)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/24/2010 01:52 AM, Martin Pool wrote:
> @aaron maybe bzr should be run with -Derror in the builders?

We're currently running an old version of bzr, so the results wouldn't
necessarily be relevant. But also, these memory issues with large
branches are very easy to reproduce locally just by branching, so I
don't think bzr needs help finding them. My gut says we should stick to
the normal behaviour that people are used to, for now.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkztHJoACgkQ0F+nu1YWqI2JhQCfa0pyvEupl5JFGQ1jkhZZOOSW
XvsAnidbQtg/3RWgnAWllhSP1z6E21WQ
=Eq52
-----END PGP SIGNATURE-----

[era (era)]

In the meantime see also bug #109114

[Rohan Garg (rohangarg)]

Not quite fixed yet : https://code.launchpad.net/~neon/+recipe/project-neon-calligra