Ninja installation fails: problems with wget

[David Englert]

Dear MadGraph Developers,

It seems that the wget version that is installed on our cluster is not really compatible with what is tested by MG.
Again package installation fails, now with while trying to install 'ninja'.

### This is what I get ###

MG5_aMC>install ninja
   You are installing 'ninja', please cite ref(s): arXiv:1203.0291, arXiv:1403.1229, arXiv:1604.01363.
Downloading the HEPToolInstaller at:
   http://madgraph.phys.ucl.ac.be/Downloads/HEPToolsInstaller/HEPToolsInstaller_V72.tar.gz
Now installing ninja. Be patient...
--2016-11-08 15:44:20-- https://bitbucket.org/peraro/ninja/downloads/ninja-latest.tar.gz
Resolving bitbucket.org... 104.192.143.2, 104.192.143.3, 104.192.143.1, ...
Connecting to bitbucket.org|104.192.143.2|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://bbuseruploads.s3.amazonaws.com/21678e19-7e54-4a85-867a-6501efba753d/downloads/2dc01dd9-c592-474b-9485-fa86415f0da4/ninja-latest.tar.gz?Signature=L6FYTpPZQCCOhKCx0tiWAAAeGLo%3D&Expires=1478621335&AWSAccessKeyId=AKIAIVFPT2YJYYZY3H4A&versionId=phynf6NIvZbMzuOY8T.wQgHBNNk6WTQy&response-content-disposition=attachment%3B%20filename%3D%22ninja-latest.tar.gz%22 [following]
--2016-11-08 15:44:20-- https://bbuseruploads.s3.amazonaws.com/21678e19-7e54-4a85-867a-6501efba753d/downloads/2dc01dd9-c592-474b-9485-fa86415f0da4/ninja-latest.tar.gz?Signature=L6FYTpPZQCCOhKCx0tiWAAAeGLo%3D&Expires=1478621335&AWSAccessKeyId=AKIAIVFPT2YJYYZY3H4A&versionId=phynf6NIvZbMzuOY8T.wQgHBNNk6WTQy&response-content-disposition=attachment%3B%20filename%3D%22ninja-latest.tar.gz%22
Resolving bbuseruploads.s3.amazonaws.com... 52.216.80.80
Connecting to bbuseruploads.s3.amazonaws.com|52.216.80.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 581455 (568K) [application/x-tar]
Saving to: `ninja-latest.tar.gz?Signature=L6FYTpPZQCCOhKCx0tiWAAAeGLo=&Expires=1478621335&AWSAccessKeyId=AKIAIVFPT2YJYYZY3H4A&versionId=phynf6NIvZbMzuOY8T.wQgHBNNk6WTQy&response-content-disposition=attachment; filename="ninja-latest.tar.gz"'

100%[================================================================================================================>] 581,455 729K/s in 0.8s

2016-11-08 15:44:21 (729 KB/s) - `ninja-latest.tar.gz?Signature=L6FYTpPZQCCOhKCx0tiWAAAeGLo=&Expires=1478621335&AWSAccessKeyId=AKIAIVFPT2YJYYZY3H4A&versionId=phynf6NIvZbMzuOY8T.wQgHBNNk6WTQy&response-content-disposition=attachment; filename="ninja-latest.tar.gz"' saved [581455/581455]

Could not download data at '/tmp/tmpq2dGcR/ninja-latest.tar.gz'.
Command "install ninja" interrupted with error:
InvalidCmd : Installation of ninja failed.

##################################################################

So it seems that the filename gets appended by some sort of HTML code: 'ninja-latest.tar.gz?Signature=L6FYTpPZQCCOhKCx0tiWAAAeGLo=&Expires=1478621335&AWSAccessKeyId=AKIAIVFPT2YJYYZY3H4A&versionId=phynf6NIvZbMzuOY8T.wQgHBNNk6WTQy&response-content-disposition=attachment'

The way I went around to get it installed is the following: I downloaded the ninja package and uploaded onto a private webserver and modified the 'tarball' variable in HEPToolInstaller.py:
'tarball': ['online','https://bitbucket.org/peraro/ninja/downloads/ninja-latest.tar.gz'].

##################################################################
#### This is the wget version that'tarball': ['online','https://bitbucket.org/peraro/ninja/downloads/ninja-latest.tar.gz'] is installed on the cluster ###
##################################################################

wget -V
GNU Wget 1.12 built on linux-gnu.

+digest +ipv6 +nls +ntlm +opie +md5/openssl +https -gnutls +openssl
-iri

Wgetrc:
    /etc/wgetrc (system)
Locale: /usr/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
    -DLOCALEDIR="/usr/share/locale" -I. -I../lib -O2 -g -pipe -Wall
    -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
    --param=ssp-buffer-size=4 -m64 -mtune=generic
Link: gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
    -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
    -L/usr/lib6464 -lssl -lcrypto -ldl -lz /usr/lib64/libssl.so
    /usr/lib64/libcrypto.so -ldl -lrt ftp-opie.o openssl.o http-ntlm.o
    gen-md5.o ../lib/libgnu.a

Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <email address hidden>.
Currently maintained by Micah Cowan <email address hidden>.
Please send bug reports and questions to <email address hidden>.

##################################################################

Best,
David

[Olivier Mattelaer (olivier-mattelaer)]

Hi,

This one I have no idea what the problem is.

On my side, I have

[omatt@ingrid-ui1 madgraph5]$ wget -V
GNU Wget 1.12 built on linux-gnu.

+digest +ipv6 +nls +ntlm +opie +md5/openssl +https -gnutls +openssl
-iri

Wgetrc:
    /etc/wgetrc (system)
Locale: /usr/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
    -DLOCALEDIR="/usr/share/locale" -I. -I../lib -O2 -g -pipe -Wall
    -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
    --param=ssp-buffer-size=4 -m64 -mtune=generic -fno-strict-aliasing
Link: gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
    -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic
    -fno-strict-aliasing -Wl,-z,relro -lssl -lcrypto
    /usr/lib64/libssl.so /usr/lib64/libcrypto.so -ldl -lrt ftp-opie.o
    openssl.o http-ntlm.o gen-md5.o ../lib/libgnu.a

So it looks like very similar to yours.

On my case, I have:
Removing existing installation of tool 'ninja' in '/nfs/scratch/fynu/omatt/madgraph5/HEPTools'.
--2016-11-08 20:28:54-- https://bitbucket.org/peraro/ninja/downloads/ninja-latest.tar.gz
Resolving bitbucket.org... 104.192.143.2, 104.192.143.1, 104.192.143.3, ...
Connecting to bitbucket.org|104.192.143.2|:443... connected.
HTTP request sent, awaiting response... 302 FOUND
Location: https://bbuseruploads.s3.amazonaws.com/21678e19-7e54-4a85-867a-6501efba753d/downloads/2dc01dd9-c592-474b-9485-fa86415f0da4/ninja-latest.tar.gz?Signature=ZsCzfjMr6enIRMEPZEAkAd%2B1PHQ%3D&Expires=1478635135&AWSAccessKeyId=AKIAIVFPT2YJYYZY3H4A&versionId=phynf6NIvZbMzuOY8T.wQgHBNNk6WTQy&response-content-disposition=attachment%3B%20filename%3D%22ninja-latest.tar.gz%22 [following]
--2016-11-08 20:28:55-- https://bbuseruploads.s3.amazonaws.com/21678e19-7e54-4a85-867a-6501efba753d/downloads/2dc01dd9-c592-474b-9485-fa86415f0da4/ninja-latest.tar.gz?Signature=ZsCzfjMr6enIRMEPZEAkAd%2B1PHQ%3D&Expires=1478635135&AWSAccessKeyId=AKIAIVFPT2YJYYZY3H4A&versionId=phynf6NIvZbMzuOY8T.wQgHBNNk6WTQy&response-content-disposition=attachment%3B%20filename%3D%22ninja-latest.tar.gz%22
Resolving bbuseruploads.s3.amazonaws.com... 52.216.16.72
Connecting to bbuseruploads.s3.amazonaws.com|52.216.16.72|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 581455 (568K) [application/x-tar]
Saving to: “ninja-latest.tar.gz”

So the name on the filesystem is ok.
One solution might be to force the -O ninja-latest.tar.gz
but Valentin does not think that this is the problem.

Cheers,

Olivier

> On Nov 8, 2016, at 16:56, David Englert <email address hidden> wrote:
>
> Public bug reported:
>
> Dear MadGraph Developers,
>
> It seems that the wget version that is installed on our cluster is not really compatible with what is tested by MG.
> Again package installation fails, now with while trying to install 'ninja'.
>
> ### This is what I get ###
>
> MG5_aMC>install ninja
> You are installing 'ninja', please cite ref(s): arXiv:1203.0291, arXiv:1403.1229, arXiv:1604.01363.
> Downloading the HEPToolInstaller at:
> http://madgraph.phys.ucl.ac.be/Downloads/HEPToolsInstaller/HEPToolsInstaller_V72.tar.gz
> Now installing nin...