tempest-dsvm-neutron-src-python-heatclient-icehouse failed due to ERROR: smoke: could not install deps

[Naohiro Tamura]

https://review.openstack.org/#/c/143059/
The above patch set 3 and 5 are exactly the same.
However the patch set 3 passed gate-tempest-dsvm-neutron-src-python-heatclient-icehouse without any problem, the patch set 5 failed due to "ERROR: smoke: could not install deps".
Therefor this problem is not caused by the code change made in the patch set 3 and 5.
Please investigate the gate devstack environment.

The patch set 3, OK log:
http://logs.openstack.org/59/143059/3/check/gate-tempest-dsvm-neutron-src-python-heatclient-icehouse/4d5a592/

The patch set 5, NG log:
http://logs.openstack.org/59/143059/5/check/gate-tempest-dsvm-neutron-src-python-heatclient-icehouse/6c87292/

"ERROR: smoke: could not install deps" can be found the following log. Please investigate it.

2014-12-25 07:43:01.890 | self.cmdclass[command] = cmdclass = ep.load()
2014-12-25 07:43:01.890 | File "/opt/stack/new/tempest/.tox/smoke/local/lib/python2.7/site-packages/pkg_resources.py", line 2254, in load
2014-12-25 07:43:01.890 | ['__name__'])
2014-12-25 07:43:01.890 | ImportError: No module named command.install
2014-12-25 07:43:01.890 |
2014-12-25 07:43:01.890 |
2014-12-25 07:43:01.890 | ERROR: could not install deps [-r/opt/stack/new/tempest/requirements.txt]
2014-12-25 07:43:01.891 | ___________________________________ summary ____________________________________
2014-12-25 07:43:01.891 | ERROR: smoke: could not install deps [-r/opt/stack/new/tempest/requirements.txt]
2014-12-25 07:43:01.891 | ERROR: the main setup script run by this job failed - exit code: 1
2014-12-25 07:43:01.891 | please look at the relevant log files to determine the root cause
2014-12-25 07:43:01.891 | Cleaning up host
2014-12-25 07:43:01.892 | ... this takes 3 - 4 minutes (logs at logs/devstack-gate-cleanup-host.txt.gz)

P.S. this problem happened after having been fixed the following bug.
https://bugs.launchpad.net/devstack/+bug/1405318

[Jeremy Stanley (fungi)]

Somehow setuptools is getting upgraded from 8.2.1 to 8.3 while it's being used by other pip calls (we've seen a similar race condition in the past too). The trick is figuring out where 8.2.1 is coming from since it's not the latest version.

[Jeremy Stanley (fungi)]

Aha, 8.3 was released yesterday, so maybe we somehow managed to preinstall 8.2.1 and then tempest is attempting to upgrade.

[Jeremy Stanley (fungi)]

Upon closer observation, this looks like tempest's tox run is using pip install -U on its requirements list and has a transitive dependency on setuptools, causing the 8.2.1 which was preinstalled by virtualenv to get upgraded to a later version and triggering what is probably a regressed race condition in newer pip.

[Jeremy Stanley (fungi)]

Attempting to work around this in tempest via https://review.openstack.org/144347

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144347
Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=0ae5d52a31735c35310401993a32b4f92b420dfd
Submitter: Jenkins
Branch: master

commit 0ae5d52a31735c35310401993a32b4f92b420dfd
Author: Jeremy Stanley <email address hidden>
Date: Mon Dec 29 19:21:42 2014 +0000

    Preupgrade setuptools in tox tempestenv

    * tox.ini: If setuptools is declared as a dependency, even
    transitively, it can trigger a race in some versions of pip when
    called with -U wherein pip removes setuptools in preparation for
    upgrading it, and then proceeds to install something else using
    setuptools. Work around this behavior by upgrading setuptools within
    the virtualenv in advance.

    Change-Id: I6d9bf61c1ce41e46ff6f621bfa88d511f6c16120
    Related-Bug: #1405579