btool hitting command line limit on XP/Mingw/Msys

[David Mathog]

It seems that current releases of trunk will not build on WIndows XP under mingw because they are hitting command line length limits.

History:

In trunk at r12815 btool running in cmd.exe failed with:

##### link objects and library to create executable
--- link / rc
--- link / link
Make error line 437: LINK problem: mingw32-g++: error: CreateProcess: No
such file or directory

To diagnose this I hacked up the local copy of buildtool.cpp to dump the command it was choking on. This turned out to be a very long lne of 30152 characters (longlink.txt). Tried to run it from within Msys via:

. longlink.txt

but it didn't like the '\' characters. So swapped those all to '/' and tried:

. longlink2.txt

which gave the same error.

Then edited longlink2.txt into longlink3.txt, which is one clause per line. Tried

mingw32-g++ @longlink3.txt

and still the same error. Then did

mingw32-g++ -v @longlink3.txt

and it turned out that g++ was reassembling the file in lines and passing it to collect2, so it could still be a long line issue.
Noticed that there were MANY "-Lc:/progs/devlibs50/lib" copies in longlink3.txt, so reduced it to 1. After doing so

mingw32-g++ -v @longlink3.txt

worked. Conclusion: btool is trying to use a command line that is too long for one of the underlying tools, most likely Mingw or Msys, but possibly some library in Windows XP.

Versions:
mingw32-g++ (GCC) 4.6.2
XP SP3
mingw32 3.20 (from /mingw/include/_mingw.h)

Related issue:

build.xml constructs the link line using:

  ${pcl.packageName}

for each library. However, since they share various dependencies, this replicates a very large number of linker command line options. That is no problem on an OS with an unlimited command line, but here, it pushes us over the limit. Having btool sort that list and eliminate the duplicates would save some characters on the command line. However, that would be at best a stop gap measure, since most of the characters are in all of the "build/(something)/somefile.o" parameters.

Suggested fix - have btool construct a static library for each subdirectory of src, then have it generate a final command line
that links those static libraries together. So long as no subdirectory contains more than 30k characters worth of object files, that should work going forward for the foreseeable future.

It is possible that btool might work in some other run environment. There may be a cmd.exe replacement that somehow lifts the 30k character limit. (I do not know specifically where that comes from, or how Mingw is affected by it.)

Reference: another project with similar issues:

  http://dev.omnetpp.org/bugs/view.php?id=155

[David Mathog (mathog)]

Run like:

. longlink2.txt

[David Mathog (mathog)]

This is longlink2.txt edited to one clause per line, then all the redundant "-Lc:/progs/devlibs50/lib" copies removed. This reduced its size to 28665 bytes, which is below whatever the critical limit is, so that

g++ @longlink3.txt

will run.

[David Mathog (mathog)]

Note: The version of mingw used was from sourceforge.

http://sourceforge.net/projects/mingw/

Unsure when it was downloaded exactly, but it looks like it was about 16 months ago.

[David Mathog (mathog)]

Changes to build.xml when resolving bug #1227956, at r12559, resulted in part in the current bug.

Before r12559 build.xml put almost everything in an inkscape library. That command line was very long too, but under the limit.

How many of the modules have the C++ global initialization issue? Perhaps we can exclude the ones that do, and stuff everything else into a static library?

[David Mathog (mathog)]

Found a link that claims the program tcc.exe has a higher command line limit.

  http://mingw.5.n7.nabble.com/MSYS-bash-maximum-length-of-command-line-arguments-td22786.html

However, in my hands the limit is still too low for the current issue:

download and install software from JPSoft.com

In an Msys session:

export COMSPEC=C:/Program\ Files/JPSoft/TCCLE13/tcc.exe
export DEVLIBS_PATH='C:/progs/devlibs50'
export MINGW_PATH=C:/progs/mingw
export MINGW_BIN=C:/progs/mingw/bin
export PKG_CONFIG_PATH=C:/progs/devlibs50/lib/pkgconfig
export GTKMM_BASEPATH=C:/progs/devlibs50
export PATH='C:/progs/devlibs50/bin;C:/progs/devlibs50/python:.:/usr/local/bin:/mingw/bin:/bin:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/Program Files/Common Files/GTK/2.0/bin'

copy libstdc++-6.dll and libgc_s_dw2-1.dll into the inkscape directory

./btool

##### Target : link
##### link objects and library to create executable
--- link / rc
--- link / link
Make error line 0: executeCommand : could not create process : The system cannot find the file specified.
Make error line 0: error executing 'bzr revno':
Make error line 0: executeCommand : could not create process : The system cannot find the file specified.
Make error line 0: error executing 'bzr revno':
Make error line 437: executeCommand : could not create process : The system cannot find the file specified.
Make error line 437: LINK problem:

bzr wasn't in any of the paths, so that is OK, but the final error is the same as when running in cmd.exe.

[David Mathog (mathog)]

The attached patch for btool sends only the unique parameters from the "libs" field during the link stage to the command. This saves enough characters so that btool will complete the build.

It is not a long term solution, but will allow development to continue on Windows for a while longer, until the underlying problems can be addressed.

[jazzynico (jazzynico)]

Confirmed on Windows XP, when testing devlibs-gtk3. I hit the limit when the devlib's path is z:\dev\devlibs-gtk3, but not with z:\dl3 (workaround!).

[Bryce Harrington (bryce)]

I don't have windows to test the patch on, but I applied and checked it on Linux and things built and ran fine, so I've gone ahead and committed this.

I concur the fix is more of a workaround but hey, with the release coming up we want to remove any impediments that might prevent people from fixing bugs, especially on Windows. Plus having repeated linker line elements is messy, so uniq'ing the line makes sense from a cleanliness standpoint.