install_rally.sh: ERROR: Failed to install the latest version of Python 'setuptools'

[Antonio Ojea]

The environment is ubuntu 16.04 and installing rally with -v switch into a virtualenv. We have this setup running nigthly without any issue, but it started to fail on Dec 4th (don't know if it's a coincidence but the python setuptools released a new version that date https://pypi.python.org/pypi/setuptools)

Below you can see the output of the installer script with the error (I tried to pin to older rally versions without success)

15:40:15 + ./install_rally.sh -v -y --branch 0.4.0

15:40:16 Installing Rally virtualenv in directory '/home/jenkins/rally' ...
15:40:16 --2016-12-05 14:40:15-- https://raw.github.com/pypa/virtualenv/1.9.1/virtualenv.py
15:40:16 Resolving raw.github.com (raw.github.com)... 151.101.100.133
15:40:16 Connecting to raw.github.com (raw.github.com)|151.101.100.133|:443... connected.

15:40:16 HTTP request sent, awaiting response... 301 Moved Permanently
15:40:16 Location: https://raw.githubusercontent.com/pypa/virtualenv/1.9.1/virtualenv.py [following]
15:40:16 --2016-12-05 14:40:16-- https://raw.githubusercontent.com/pypa/virtualenv/1.9.1/virtualenv.py
15:40:17 Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.100.133
15:40:17 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.100.133|:443... connected.
15:40:17 HTTP request sent, awaiting response... 200 OK
15:40:17 Length: 115013 (112K) [text/plain]
15:40:17 Saving to: ‘/home/jenkins/rally/virtualenv-191.py’
15:40:17
15:40:17 0K .......... .......... .......... .......... .......... 44% 938K 0s
15:40:17 50K .......... .......... .......... .......... .......... 89% 1.84M 0s
15:40:17 100K .......... .. 100% 32.3M=0.08s
15:40:17
15:40:17 2016-12-05 14:40:17 (1.37 MB/s) - ‘/home/jenkins/rally/virtualenv-191.py’ saved [115013/115013]
15:40:17
15:40:17 Already using interpreter /usr/bin/python
15:40:17 Creating /home/jenkins/rally/lib/python2.7
15:40:17 Symlinking Python bootstrap modules
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/lib-dynload
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/os.py
15:40:17 Ignoring built-in bootstrap module: posix
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/posixpath.py
15:40:17 Cannot import bootstrap module: nt
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/ntpath.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/genericpath.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/fnmatch.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/locale.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/encodings
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/codecs.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/stat.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/UserDict.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/copy_reg.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/types.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/re.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/sre.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/sre_parse.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/sre_constants.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/sre_compile.py
15:40:17 Ignoring built-in bootstrap module: zlib
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/warnings.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/linecache.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/_abcoll.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/abc.py
15:40:17 Symlinking /home/jenkins/rally/lib/python2.7/_weakrefset.py
15:40:17 Creating /home/jenkins/rally/lib/python2.7/site-packages
15:40:17 Writing /home/jenkins/rally/lib/python2.7/site.py
15:40:17 Writing /home/jenkins/rally/lib/python2.7/orig-prefix.txt
15:40:17 Writing /home/jenkins/rally/lib/python2.7/no-global-site-packages.txt
15:40:17 Creating parent directories for /home/jenkins/rally/include
15:40:17 Symlinking /home/jenkins/rally/include/python2.7
15:40:17 Creating /home/jenkins/rally/bin
15:40:17 New python executable in /home/jenkins/rally/bin/python
15:40:17 Changed mode of /home/jenkins/rally/bin/python to 0775
15:40:17 Testing executable with /home/jenkins/rally/bin/python -c "import sys;out=sys.stdout;getattr(out, "buffer", out).write(sys.prefix.encode("utf-8"))"
15:40:17 Got sys.prefix result: u'/home/jenkins/rally'
15:40:17 Creating /home/jenkins/rally/lib/python2.7/distutils
15:40:17 Writing /home/jenkins/rally/lib/python2.7/distutils/__init__.py
15:40:17 Writing /home/jenkins/rally/lib/python2.7/distutils/distutils.cfg
15:40:17 No setuptools egg found; downloading
15:40:17 Installing setuptools...

15:40:22 Downloading http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
15:40:22 Searching for setuptools
15:40:22 Downloading https://pypi.python.org/packages/48/c0/f01379bc7b1b7684bdf4cc744b6124bc0a089349978ec576687ac8f809a7/setuptools-30.2.0.zip#md5=19d6d461f2d85720837e4b33ac56f65d
15:40:22 Running setuptools-30.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-f1yKY1/setuptools-30.2.0/egg-dist-tmp-a9U9bE
15:40:22 /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'src_root'
15:40:22 warnings.warn(msg)
15:40:22 Traceback (most recent call last):
15:40:22 File "<string>", line 278, in <module>
15:40:22 File "<string>", line 213, in main
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1712, in main
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1700, in with_ei_usage
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1716, in <lambda>
15:40:22 File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
15:40:22 dist.run_commands()
15:40:22 File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
15:40:22 self.run_command(cmd)
15:40:22 File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
15:40:22 cmd_obj.run()
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 211, in run
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 446, in easy_install
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 476, in install_item
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 655, in install_eggs
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 930, in build_and_install
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 919, in run_setup
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/sandbox.py", line 62, in run_setup
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/sandbox.py", line 105, in run
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/sandbox.py", line 64, in <lambda>
15:40:22 File "setup.py", line 181, in <module>
15:40:22 File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
15:40:22 _setup_distribution = dist = klass(attrs)
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/dist.py", line 264, in __init__
15:40:22 File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
15:40:22 self.finalize_options()
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/dist.py", line 297, in finalize_options
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1959, in load
15:40:22 ImportError: <module 'setuptools.dist' from '/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/dist.py'> has no 'check_specifier' attribute
15:40:22 Complete output from command /home/jenkins/rally/bin/python -c "#!python
15:40:22 \"\"\"Bootstra...sys.argv[1:])
15:40:22
15:40:22
15:40:22
15:40:22
15:40:22
15:40:22
15:40:22 " --always-copy -U setuptools:
15:40:22 Downloading http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
15:40:22 Searching for setuptools
15:40:22 Reading http://pypi.python.org/simple/setuptools/
15:40:22 Best match: setuptools 30.2.0
15:40:22 Downloading https://pypi.python.org/packages/48/c0/f01379bc7b1b7684bdf4cc744b6124bc0a089349978ec576687ac8f809a7/setuptools-30.2.0.zip#md5=19d6d461f2d85720837e4b33ac56f65d
15:40:22 Processing setuptools-30.2.0.zip
15:40:22 Running setuptools-30.2.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-f1yKY1/setuptools-30.2.0/egg-dist-tmp-a9U9bE
15:40:22 /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'src_root'
15:40:22 warnings.warn(msg)
15:40:22 Traceback (most recent call last):
15:40:22 File "<string>", line 278, in <module>
15:40:22 File "<string>", line 213, in main
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1712, in main
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1700, in with_ei_usage
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1716, in <lambda>
15:40:22 File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
15:40:22 dist.run_commands()
15:40:22 File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
15:40:22 self.run_command(cmd)
15:40:22 File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
15:40:22 cmd_obj.run()
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 211, in run
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 446, in easy_install
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 476, in install_item
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 655, in install_eggs
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 930, in build_and_install
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 919, in run_setup
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/sandbox.py", line 62, in run_setup
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/sandbox.py", line 105, in run
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/sandbox.py", line 64, in <lambda>
15:40:22 File "setup.py", line 181, in <module>
15:40:22 File "/usr/lib/python2.7/distutils/core.py", line 111, in setup
15:40:22 _setup_distribution = dist = klass(attrs)
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/dist.py", line 264, in __init__
15:40:22 File "/usr/lib/python2.7/distutils/dist.py", line 287, in __init__
15:40:22 self.finalize_options()
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/dist.py", line 297, in finalize_options
15:40:22 File "/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/pkg_resources.py", line 1959, in load
15:40:22 ImportError: <module 'setuptools.dist' from '/jenkins/workspace/software_acceptance/api_benchmark_master_mitaka/setuptools-0.6c11-py2.7.egg/setuptools/dist.py'> has no 'check_specifier' attribute
15:40:22 ----------------------------------------
15:40:22 ...Installing setuptools...done.
15:40:22 Traceback (most recent call last):
15:40:22 File "/home/jenkins/rally/virtualenv-191.py", line 2577, in <module>
15:40:22 main()
15:40:22 File "/home/jenkins/rally/virtualenv-191.py", line 979, in main
15:40:22 no_pip=options.no_pip)
15:40:22 File "/home/jenkins/rally/virtualenv-191.py", line 1091, in create_environment
15:40:22 search_dirs=search_dirs, never_download=never_download)
15:40:22 File "/home/jenkins/rally/virtualenv-191.py", line 611, in install_setuptools
15:40:22 search_dirs=search_dirs, never_download=never_download)
15:40:22 File "/home/jenkins/rally/virtualenv-191.py", line 583, in _install_req
15:40:22 cwd=cwd)
15:40:22 File "/home/jenkins/rally/virtualenv-191.py", line 1057, in call_subprocess
15:40:22 % (cmd_desc, proc.returncode))
15:40:22 OSError: Command /home/jenkins/rally/bin/python -c "#!python
15:40:22 \"\"\"Bootstra...sys.argv[1:])
15:40:22
15:40:22
15:40:22
15:40:22
15:40:22
15:40:22
15:40:22 " --always-copy -U setuptools failed with error code 1
15:40:22 ./install_rally.sh: line 425: bin/activate: No such file or directory
15:40:22 --2016-12-05 14:40:22-- https://bootstrap.pypa.io/ez_setup.py
15:40:22 Resolving bootstrap.pypa.io (bootstrap.pypa.io)... 151.101.76.175
15:40:22 Connecting to bootstrap.pypa.io (bootstrap.pypa.io)|151.101.76.175|:443... connected.
15:40:22 HTTP request sent, awaiting response... 200 OK
15:40:22 Length: 12746 (12K) [text/x-python]
15:40:22 Saving to: ‘STDOUT’
15:40:22
15:40:22 0K .......... .. 100% 1.30M=0.009s
15:40:22
15:40:22 2016-12-05 14:40:22 (1.30 MB/s) - written to stdout [12746/12746]
15:40:22

15:40:23 Downloading https://pypi.io/packages/source/s/setuptools/setuptools-30.2.0.zip

15:40:24 Extracting in /tmp/tmpZ5Sd_K
15:40:24 Now working in /tmp/tmpZ5Sd_K/setuptools-30.2.0
15:40:24 Installing Setuptools

15:40:26 running install

15:40:26 error: can't create or remove files in install directory
15:40:26
15:40:26 The following error occurred while trying to add or remove files in the
15:40:26 installation directory:
15:40:26
15:40:26 [Errno 13] Permission denied: '/usr/local/lib/python2.7/dist-packages/test-easy-install-22262.write-test'
15:40:26
15:40:26 The installation directory you specified (via --install-dir, --prefix, or
15:40:26 the distutils default setting) was:
15:40:26
15:40:26 /usr/local/lib/python2.7/dist-packages/
15:40:26
15:40:26 Perhaps your account does not have write access to this directory? If the
15:40:26 installation directory is a system-owned directory, you may need to sign in
15:40:26 as the administrator or "root" account. If you do not have administrative
15:40:26 access to this machine, you may wish to choose a different installation
15:40:26 directory, preferably one that is listed in your PYTHONPATH environment
15:40:26 variable.
15:40:26
15:40:26 For information on other options, you may wish to consult the
15:40:26 documentation at:
15:40:26
15:40:26 https://setuptools.readthedocs.io/en/latest/easy_install.html
15:40:26
15:40:26 Please make the appropriate changes for your system and try again.
15:40:26
15:40:26 Something went wrong during the installation.
15:40:26 See the error message above.
15:40:26 ==========================================================
15:40:26 install_rally.sh: ERROR: Failed to install the latest version of Python 'setuptools'
15:40:26 ==========================================================
15:40:26
15:40:26
15:40:26 The required Python package setuptools could not be installed.
15:40:26
15:40:26
15:40:26 If the above does not help you resolve the issue, please contact the
15:40:26 Rally team by sending an email to the OpenStack mailing list
15:40:26 <email address hidden>. Include the full output of this
15:40:26 script to help us identifying the problem.
15:40:26

[Abel Lopez (al592b)]

You see others complaining about setuptools 30.2.0.
https://github.com/pypa/pip/issues/4104

The workaround there is to simply use 30.1.0

[Abel Lopez (al592b)]

Looks like this was recently fixed with
https://github.com/openstack/rally/commit/648da41e12f63869d71a56d937a0c4a248345983