swapping, stalling: memory requirements for launchpad builds

[Nico Schlömer]

One of the projects I regularly build on launchpad is written mostly in C++ and partly uses templating quite heavily. This results in rather high memory requirements for the build, which effectively leads to swapping of the build process. After 150 minutes of stalling, the build process is then aborted by launchpad, and an error is thrown. This happens virtually every time, e.g., <https://launchpadlibrarian.net/172994769/buildlog_ubuntu-trusty-amd64.trilinos_11.9~20140417-trusty2_FAILEDTOBUILD.txt.gz>.

What could be done to avoid this?

[William Grant (wgrant)]

All the PPA builders now have 4GiB of RAM. Does this still happen?

[Nico Schlömer (nschloe)]

Yes, cf. [1], although the error message is now clearer:
```
virtual memory exhausted: Cannot allocate memory
```
What I would need is the possibility for specifying memory requirements. I wouldn't even mind waiting a week until a higher-memory build bot becomes available. Right now, I can hardly build at all.

[1] https://launchpadlibrarian.net/181060236/buildlog_ubuntu-trusty-i386.trilinos_11.11~20140730-trusty2_FAILEDTOBUILD.txt.gz

[William Grant (wgrant)]

Are you sure that builds at all on i386, even with a tonne of RAM? The builder VMs have 4GiB of RAM each, which is the maximum a 32-bit process can address. I suspect that the way you're compiling things simply can't complete successfully on a 32-bit architecture, and you'll need to restructure it to do less in one process. This suspicion is reinforced by the fact that the amd64 builds seem to be working fine, on VMs with exactly the same specs and kernel.

[Nico Schlömer (nschloe)]

Hm, good point!

I'll check back with the authors of the package and see what we could do about it.

On launchpad, is it possible to submit amd64-only builds?

[Nico Schlömer (nschloe)]

Curiously, the i386 for utopic [1] (largely) succeeded.

[1] https://launchpadlibrarian.net/181060826/buildlog_ubuntu-utopic-i386.trilinos_11.11~20140730-utopic2_FAILEDTOBUILD.txt.gz

[William Grant (wgrant)]

On 30/07/14 23:48, Nico Schlömer wrote:
> Curiously, the i386 for utopic [1] (largely) succeeded.
>
> [1] https://launchpadlibrarian.net/181060826/buildlog_ubuntu-
> utopic-i386.trilinos_11.11~20140730-utopic2_FAILEDTOBUILD.txt.gz

That might mean that the toolchain in utopic is more memory-efficient
than trusty's.

[Nico Schlömer (nschloe)]

The amd64 builds also fail [1], but at a different point.
This time, the error message is the good old
```
Build killed with signal 15 after 150 minutes of inactivity
```
pointing towards swapping and stalling.

[1] https://launchpadlibrarian.net/181066413/buildlog_ubuntu-trusty-amd64.trilinos_11.11~20140730-trusty2_FAILEDTOBUILD.txt.gz

[Colin Watson (cjwatson)]

You can certainly write "Architecture: amd64" in all the binary stanzas of your control file to cause builds to be amd64-only.

[Nico Schlömer (nschloe)]

This is still an issue. Many of my builds fail with
```
virtual memory exhausted: Cannot allocate memory
```
cf. [1].

[1] https://launchpadlibrarian.net/183514847/buildlog_ubuntu-precise-amd64.trilinos_11.11~20140829-precise11_FAILEDTOBUILD.txt.gz

[Nico Schlömer (nschloe)]

Note that this happens for utopic as well [1].
May this be related to the fact that builds are (now?) performed with `make -j8`? How is it possible to change this to `-j1`?

[1] https://launchpadlibrarian.net/183515360/buildlog_ubuntu-utopic-amd64.trilinos_11.11~20140829-utopic11_FAILEDTOBUILD.txt.gz

[William Grant (wgrant)]

A misconfiguration on our build cloud led to the VMs having 1/4 of the usual RAM for the last 12 hours. That's fixed now, so you shouldn't see the problem any more.

[Nico Schlömer (nschloe)]

I restarted the builds and indeed they run fine now. Thanks!