libshogun-dev upgrade impossible - shogun-octave missing due to 4GB out-of-memory compilation error

[B Bobo]

After upgrading from Ubuntu 11.10 to 12.04, all the upgrades happened except for libshogun-dev. It was not upgraded from version 0.10.0-2.1ubuntu1 to 1.1.0-4ubuntu2.

Both Ubuntu Software Center and Synaptic are unable to upgrade this package automatically.

In synaptic, the package is listed as needing an upgrade, having a "!" symbol next to it, but clicking Mark All Upgrades fails to select libshogun-dev for automatic upgrading.

If I manually select libshogun-dev in synaptic, and try to upgrade, the additional required changes are

 To be removed:
     libshogunui5
     libshogunui6
     shogun-cmdline
     shogun-octave
     shogun-r

 To be installed:
     libshogun11

However, I use shogun-r, shogun-octave and shogun-cmdline. I don't want them uninstalled.

Is there a packaging issue in libshogun-dev?

[Julian Taylor (jtaylor)]

shogun-r and shogun-cmdline are still available as shogun-r-static and and shogun-cmdline-static
shogun-octave has been dropped, apperently due to issues with compiling octave with clang
you have to speak with the debian maintainer about this, see http://packages.qa.debian.org/s/shogun.html

- Since octave is failing to compile with clang - temporarily disable
    shogun's octave interface.

[Julian Taylor (jtaylor)]

yu can pin the package to keep the old version:
http://jaqque.sbih.org/kplug/apt-pinning.html

[B Bobo (yout-bobo123)]

Julian, thank you very much for taking the time to explain that. I'll certainly raise this issue with the maintainer.

[B Bobo (yout-bobo123)]

According to the maintainer, the problem is when using the buildd service to compile shogun-octave there is an out-of-memory error. The buildd service is apparently limited to 4GB memory. In this day and age of huge computers, a 4GB limit is too small. Can somebody please remove the 4GB limit if it is hardcoded or else find a workaround to enable shogun-octave to be built?

[Robert Collins (lifeless)]

My suggestion is to fix your compiler :) 4GB is still 2-4 times the
physical memory of most consumer machines.

[B Bobo (yout-bobo123)]

First, isn't compilation is limited by the amount of virtual memory?

Turning the question around, what is the point of buildd arbitrarily imposing a 4GB virtual memory limit (if it is indeed hardcoded)? If it isn't hardcoded, trying to fix scrambled spaghetti compiler code is sadly not a very practical suggestion. The issue according to the maintainer is really about why servers running the buildd service fail because of a 4GB out-of-memory error on the server. The 4GB limit means he can't build the shogun-octave package on the buildd server, so he cannot create the package for the distro. It can, however, be built with 8GB virtual memory. That's really not much memory for a complex scientific application compilation. Servers typically have much more physical (and virtual) memory than that; the average server sold last year had 18GB physical memory.

Could the 4GB limit be increased to 8GB?

[William Grant (wgrant)]

It's not an arbitrary limit. We don't upgrade builder hardware every year, and in many cases there are multiple virtual builders on each physical machine, so they have limited amounts of RAM. Other vast packages in Ubuntu have no problem with this.

Requiring more than 4GiB of virtual memory to link will prevent you from building in a 32-bit environment anyway, so it's entirely unreasonable to expect that much during a build. Even WebKit -- one of the worst offenders in terms of link-time memory usage -- manages to fit within 4GiB nowadays. There are also architectures where it's simply not practical to have 4GiB of RAM, eg. on ARM devices.

It also looks like the package was dropped in Debian, not Ubuntu, so it may be more a problem with the Debian builders's, not Launchpad's Ubuntu builders.

[B Bobo (yout-bobo123)]

I hear what you say - it's certainly not normal for a piece of software to require that much memory to compile - but in the end this reduces to the fact that it's a complex scientific application. It's literally orders of magnitude different in various features from "normal" pieces of software including WebKit. There is really no comparison with ordinary desktop software. For example, it would be typical to handle 100GB - 10TB datasets. Basically it's a different beast altogether. Because of that, nobody would want to use this application in a 32-bit environment or on ARM devices, so it really doesn't matter if it can't compile for those cases because they're just not especially relevant.

As I understand it from the maintainer, the 4GB memory limit is generic to buildd, whether running on Debian or Ubuntu. So, this issue is going to keep biting until the limit is increased. This seems as good a place as any to address the issue.

What's the real issue with making a moderate increase from 4GB to 8GB? I'm still assuming it's actually hardcoded somewhere in buildd, which it might not be of course. The builder hardware has limited memory and multiple virtual builders, but in practice very few applications except perhaps only this one are going to use up that much memory for a compilation, so the change should hardly impact the builder servers.

[Robert Collins (lifeless)]

It's not hardcoded. Its a limit in different places.

On the 32-bit buildds, no one process can address that much memory anyway.

On the 64-bit buildds, many of them don't have (or didn't when I left
:P - the situation may have changed) enough RAM that having a large
swap partition makes sense - if a build thrashes too much, it will
timeout and be killed. (So the rule is 'builds must complete without
causing the build machine to thrash').

Now, for *some* of the builder machines we could run larger VMs with
8GB of RAM, but we have no way in the current software of restricting
the scheduling builds onto them, so shogun would simply end up failing
much of the time when it scheduled onto smaller VMs.

Could we discard all the smaller machines and just run a smaller set
of such larger VMs? Sure. But that then ends up wasting physical ram
for most of the VMs most of the time.

I accept that there is a definite issue for your sofware. However,
there isn't a code bug here AFAICT. I suggest filing a support ticket
if you want to take it further in the operational direction.

That said, even for a scientific operation, various things we've
learnt about software maintenance - such as having a single clear
focus for a module - will still apply, won't they ? Building a mismash
of other projects and doing - I presume - a static link [why else
would you include webkit in your build?] - is going to pose
maintenance and and robustness issues. It is of course your perogative
to accept your build footprint as a given rather than something that
can be addressed.

[B Bobo (yout-bobo123)]

To clarify how WebKit came into the discussion: WebKit was mentioned by William Grant above merely as an example of a more-or-less worst-case compilation that still manages to fit inside 4GB memory; I pointed out that WebKit like other ordinary desktop software is not a good reference for build requirements because complex scientific software is vastly different, and also that 32-bit build environments can be ignored for this package.

It's actually not true the module is a mishmash of other projects; its aim is simply to provide an octave interface for shogun, which is extremely useful for scientific computations. The very large build requirements are actually not unusual for complex scientific software. Ubuntu and Debian communities have no other examples of this kind of very large-scale complex scientific software, so experience of how to manage the compilation requirements is limited, and hearing the requirements sometimes causes consternation at first. Experience from similar commercial software such as Matl@b, however, is that very large build requirements are real and normal for such necessarily complex software. It would be a pity if the first and only example of very large-scale complex scientific software to appear in the Debian and Ubuntu communities were excluded forever because of a trivial issue like a memory limit (and anti-thrashing check).

As you explained, the buildd servers manage memory allocation in a standard way that works for most compilations, but there are exceptional cases such as this package where it always fails. You said there is no way in the current software of restricting the builds onto larger VMs. It would be great if buildd could do that. Where is the best place to make a support request?

[B Bobo (yout-bobo123)]

Clarifying further that WebKit is not included in the build.

[B Bobo (yout-bobo123)]

I don't understand how or where to make a support request as recommended in comment #9. What do I need to do?