Support 'external' launchpad-buildds

[Fabio Alessandro Locati]

I think that Soyuz need a compiler network due to be able to compile big amounts of programs in a short time, and without graving on the Canonical's servers. OpenSuse ane Fedora have already this kind of thing and it seems working very good.
In this way we will be able to keep the ubuntu repositories more updated without the worry of charge too much work on the Canonical's servers.

[Celso Providelo (cprov)]

Hi Fabio, I'm not exactly sure about what you mean by 'compiler network'. Could you, please, point me to some document about this topic ?

[Fabio Alessandro Locati (f4l3)]

Something like icecream (http://en.opensuse.org/Icecream) using as server the ppa server, and as client all the computer whose owner want. Is like BOINC as idea or as all the others grid of computers.

[Harald Sitter (apachelogger)]

Awesome suggestion.

I think some kind of distributed Soyuz buildd would be the way to go (i.e. a distribution network for packages)
The Idea is that everyone can install the buildd and build the packages on his local machine, or project server or whatever... and then publishes the resulting debs on Launchpad.

This can of course can be enhanced to a compiler network, like Fabio suggests, a distributed compile cluster or the less fine gridded version where the packages get distributed to various servers which register as buildd at Launchpad.

With either approach the target should be to increase build power and therefore speed up builds.

[Celso Providelo (cprov)]

Right, there is a long going effort to allow the current, already distributed, infrastructure to support 'external' buildds. Read 'external' as machines/client outside out trusted network.

Technically speaking, it would be quite easy to support that, including the its natural implications as dedicated per project/ppa buldds. However it incurs in some possible security issues that can't be easily sorted.

For instance, as an user of PPA-hosted packages, the most important assumption that I can make is that "the binaries being installed in my computer *were* indeed built from the corresponding sources hosted in the same PPA and nothing else". That, IMHO, is what encourages users to actually *prefer* packages built in the PPA system than other ones built in somewhere else, who knows where.

You know, better than me, that poisoned binaries aren't any difficult to build or distribute, but they are very hard to be identified, due to the very trusted nature of our package system. When we allow 'external' buildds, we will be mixing trusted (at least, easier to trust) packages with some other that must be very carefully audited before being trusted.

That said, thanks for you report, that's indeed a challenge that we want to face and thus improve the throughput of the launchpad buildfarm with the help of our community. Let's keep this bug report open to track progress on this area.

[Fabio Alessandro Locati (f4l3)]

maybe giving the same sources to more than one computer and comparing the results?

[Fabio Alessandro Locati (f4l3)]

I have seen that is 'confirmed'... anything new is going on about this?

[Julian Edwards (julian-edwards)]

Fabio

This is low priority for us right now (see "wishlist" in the status) as we have a lot of other stuff that needs doing first. We're still thinking about it, however, and will take another look in the future.

[Fabio Alessandro Locati (f4l3)]

I see :)
I hope it will become reality soon :)

Oh, btw, I think this could be connected to https://bugs.launchpad.net/soyuz/+bug/332958

[Richard Hansen (rhansen)]

I agree with the security concerns.

Comparing the results of two different builds may not work. Many projects embed data about the build environment inside the compiled results (e.g., hostname of the machine it was built on, date it was built, etc.). These bits of information would be different on different machines, resulting in a high probability that two functionally identical builds are flagged as different.

Also, a malicious user that is able to control multiple build machines might be able to fool the network into thinking that a malicious build of benign source code is OK.