launchpad-buildd processes stderr/stdout very suboptimally

[Adam Conrad]

Builds that produce a large amount of log output take significantly longer (like, several orders of magnitude) on launchpad-buildd than they do using raw sbuild from the lp-buildd source. Fairly trivially reproduced by taking a simple package (say "hello") and tossing a for loop in debian/rules that throws a few thousand (or million, if testing on a fast machine) lines to stdout. sbuild or dpkg-buildpackage teeing a log to disk will have no real issues with this, lp-buildd will swap like crazy and flirt with bringing your machine down.

Julian and I talked about this at the Platform Rally, and he said he'd look at it and maybe cargo-cult some code from buildd-manager or some such.

[Adam Conrad (adconrad)]

(And as discussed previously, this is affecting us real-world right now, where I have a ludicrously long test timeout for a telepathy-glib test that executes in about 12s on a manual build on the same hardware, while taking several minutes on lp-buildd, obviously I can't/won't push patches upstream to extend test timeouts for the sake of our slightly nutty buildds)

[Julian Edwards (julian-edwards)]

Adam, if you want this marked as Critical, and thus get fixed within 6 months, you need to escalate it via the Stakeholders process.

[Julian Edwards (julian-edwards)]

Having said that I'll take a look and see if there's any obviously wrong code.

[Julian Edwards (julian-edwards)]

Huh, I wonder if it's the log sanitizer that removes passwords from private archive builds ... there's a regex that scans for URLs with passwords in them.

[Adam Conrad (adconrad)]

I'm not too fussed about it being critical. It's a real bug and a real problem, but not the end of the world to work around for a while.

[Robert Collins (lifeless)]

I suspect this is pretty straight forward once someone starts eyeballing it. Marking easy (as I think its < 8 hours work)

[Colin Watson (cjwatson)]

I think the log sanitiser is a red herring here, since it isn't used for public archive builds; Adam mentioned telepathy-glib and I'd be surprised if that were private.

That said, I'm no closer to spotting a cause.

[Adam Conrad (adconrad)]

I've not looked at this one in years. I wonder if it might be the implicit-pointer thingee? I've not at all tried to reproduce this since 2011, and I wonder if maybe we've just masked over it by having faster hardware now (I was probably seeing this on beagles or something).

[Colin Watson (cjwatson)]

As far as I can see the implicit-pointer thing would only be a problem if there were a particularly large number of errors. ICBW.

[Colin Watson (cjwatson)]

The implicit-pointer filter takes 2m40 on a 2GB file here and roughly constant memory; it's not optimal but I think we can rule it out as the cause here.