recipe builds can use too much memory

Reported by Aaron Bentley on 2010-11-17
28
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Aaron Bentley

Bug Description

It is believed that excess memory use was a factor in https://wiki.canonical.com/IncidentReports/2010-10-28-LP-build-manager-not-dispatching and https://wiki.canonical.com/IncidentReports/2010-11-17-buildd-manager-disabling-builders

We believe that excessive memory use drove the builders so far into swap that they became unresponsive.

Related branches

Tim Penhey (thumper) on 2010-11-18
Changed in launchpad-code:
status: Triaged → In Progress
assignee: nobody → Aaron Bentley (abentley)
Changed in launchpad-code:
milestone: none → 10.12
tags: added: qa-needstesting
Changed in launchpad-code:
status: In Progress → Fix Committed
Aaron Bentley (abentley) on 2010-11-22
tags: added: qa-ok
removed: qa-needstesting
Dimitri John Ledkov (xnox) wrote :

Is "bzr: out of memory" during recipe build is a new feature introduced by this fix?

E.g. https://code.launchpad.net/~mingw-w64/+recipe/gcc4.4+trunk/+build/8623

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/24/2010 01:27 AM, Dmitrijs Ledkovs wrote:
> Is "bzr: out of memory" during recipe build is a new feature introduced
> by this fix?

Yes, more or less. We are in the unfortunate position that bzr uses
excessive memory with large branches and the build farm (not just a
single builder) appears to become unstable when excessive memory is used.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkzstDwACgkQ0F+nu1YWqI0uOQCdELFMtio4W8S1a6w3tZ0t2889
lTYAnRSYU8vHoXcDgvQGV3FU8R1VXB8P
=rpGW
-----END PGP SIGNATURE-----

Martin Pool (mbp) wrote :

On 24 November 2010 17:44, Aaron Bentley <email address hidden> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 11/24/2010 01:27 AM, Dmitrijs Ledkovs wrote:
>> Is "bzr: out of memory" during recipe build is a new feature introduced
>> by this fix?

We're working on reducing bzr memory usage through other bugs,
separate from this. One way to help is to attach a traceback showing
where bzr stopped when it ran out of memory.

@aaron maybe bzr should be run with -Derror in the builders?

--
Martin

Aaron Bentley (abentley) wrote :

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11/24/2010 01:52 AM, Martin Pool wrote:
> @aaron maybe bzr should be run with -Derror in the builders?

We're currently running an old version of bzr, so the results wouldn't
necessarily be relevant. But also, these memory issues with large
branches are very easy to reproduce locally just by branching, so I
don't think bzr needs help finding them. My gut says we should stick to
the normal behaviour that people are used to, for now.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkztHJoACgkQ0F+nu1YWqI2JhQCfa0pyvEupl5JFGQ1jkhZZOOSW
XvsAnidbQtg/3RWgnAWllhSP1z6E21WQ
=Eq52
-----END PGP SIGNATURE-----

Curtis Hovey (sinzui) on 2010-12-08
Changed in launchpad-code:
status: Fix Committed → Fix Released
era (era) wrote :

In the meantime see also bug #109114

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers