ld.exe: out of memory allocating N bytes for large projects using win32 toolchain

[Jacek Ślimok]

While linking large projects (project I'm working with is just over 1.7k object files), linker reports "ld.exe: out of memory allocating N bytes".

Currently the only available toolchain version for Windows is 32-bit one. This version hasn't been linked with "large address aware" link option and therefore has access to only 2GB RAM. This is a problem for large projects as it's effectively impossible to link such projects under Windows.

It would be best if 64-bit version would be released. Additionally, adding the "--large_address_aware" to 32-bit version wouldn't be bad either.

[Tejas Belagod (belagod-tejas)]

Hi Jacek,

We could try enabling --large-address-aware for our binutils builds for Win32. But the real question is can ld.exe handle links that take >2GB RAM - can it do links with I/O reads from disk etc. We will explore this option and let you know.

Thanks,
Tejas.

[Jacek Ślimok (jack445)]

Hello Tejas,

From our experience it does handle it quite well, given the size of the project of course. Up to this point we haven't had any issues linking the project and it can be safely assumed that it's been built hundreds of times already.

The main factor is disk read speed with CPU speed being less important. From the experiments we've conducted the linking times of this project for HDD may vary from single minutes (system+compiler+IDE+project on same HDD) to under a minute (project on HDD, everything else on SSD). Putting such project on an unused or faster drive (not used for OS etc. or just using an SSD) helps to improve the time. In my case with the project stored on an SSD the linking takes 15 seconds (tested on AMD Ryzen 5 1600).

As a sidenote - when testing you may stumble upon another Windows limitation, namely the command line character limit (32k in case of newer version of Windows). This can be easily solved with the "@response" file.

As for the fix itself - adding the already mentioned "--large-address-aware" should do the trick and so should releasing the 64-bit version, although not sure how much hassle that would be.

Jacek

[Thomas Preud'homme (thomas-preudhomme)]

Hi Jacek,

There is still a number of users of our toolchain on Windows 32bit systems and supporting both a 32bit and 64bit version increase the testing/validation effort at each release. We are thus looking into --large-address-aware at the moment.

Should the validation become less of a problem (eg. more resources or more automation) or the number of 32bit Windows users become small enough, a Windows 64bit would then be considered.

Best regards.