unable to run tests due to missing deps in the virtual env

[Matt Fischer]

On both my Ubuntu box and my Mac, I've been unable to run the glance tests since this evening due to a missing dependency, specifically a version of psutil between 0.6 and 1.0. The archive only has 1.1 and up. Here are the logs:

Downloading/unpacking psutil>=0.6.1,<1.0 (from -r /Users/mfischer/code/glance/test-requirements.txt (line 19))
  http://tarballs.openstack.org/oslo.messaging/oslo.messaging-1.2.0a11.tar.gz#egg=oslo.messaging-1.2.0a11 uses an insecure transport scheme (http). Consider using https if tarballs.openstack.org has it available
  Could not find a version that satisfies the requirement psutil>=0.6.1,<1.0 (from -r /Users/mfischer/code/glance/test-requirements.txt (line 19)) (from versions: 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.2.0, 1.2.1)
  Some externally hosted files were ignored (use --allow-external to allow).
Cleaning up...
No distributions matching the version for psutil>=0.6.1,<1.0 (from -r /Users/mfischer/code/glance/test-requirements.txt (line 19))
Storing debug log for failure in /var/folders/d2/qr0r7fc10j35_lwkz9wwmtxc0000gp/T/tmpBIMPmg

[Zhi Yan Liu (lzy-dev)]

This is partial duplicated with bug# 1251105 .

[Zhi Yan Liu (lzy-dev)]

@Matt Fischer, I can run test cases successful with "tox" instead of "run_tests.sh", tox will setup psutil v0.7.1 automatically.

Furthermore, from current glance codebase, there is only one psutil calls at glance/tests/functional/v1/test_multiprocessing.py line #46, and I can run test cases via "run_tests.sh" successful with uncapped psutil requirement (v1.2.1 been installed by venv).

So if you just like to allow "run_tests.sh" work, this is worth thing to do I agree since it's a useful script to trigger local testing easily, can we just unlock psutil version cap to "psutil>=0.6.1" for Glance only? Does Jenkins gate allow it if we don't change global-requirement.txt? Of cause I believe it will be better if we do it in global-requirement to make consistent globally, but we probably need to change/verify other project code to make them workable with psutil>=1.0 first.

[Matt Fischer (mfisch)]

Once you get past the first error, pysendfile is also a problem:

Downloading/unpacking pysendfile==2.0.0 (from -r /home/mfisch/openstack/glance/test-requirements.txt (line 24))
  http://tarballs.openstack.org/oslo.messaging/oslo.messaging-1.2.0a11.tar.gz#egg=oslo.messaging-1.2.0a11 uses an insecure transport scheme (http). Consider using https if tarballs.openstack.org has it available
  Could not find any downloads that satisfy the requirement pysendfile==2.0.0 (from -r /home/mfisch/openstack/glance/test-requirements.txt (line 24))
  Some externally hosted files were ignored (use --allow-external pysendfile to allow).
Cleaning up...
No distributions at all found for pysendfile==2.0.0 (from -r /home/mfisch/openstack/glance/test-requirements.txt (line 24))

It looks like the repo isn't hosting any copies of pysendfile at all, so I'm not sure how to fix this one.

[Matt Fischer (mfisch)]

Also I think that this should be higher than a Wishlist. run_tests not working is a broken feature, not a Wishlist item. If it's truly wishlist then lets just get rid of run_tests completely rather than have it broken and confusing for contributors, which is the current state.

[David Peraza (dperaza)]

This is affecting neutron as well:

Downloading/unpacking psutil>=0.6.1,<1.0 (from -r /home/openstack/workspace/neutron/requirements.txt (line 19))
  Could not find a version that satisfies the requirement psutil>=0.6.1,<1.0 (from -r /home/openstack/workspace/neutron/requirements.txt (line 19)) (from versions: 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.2.0, 1.2.1)
  Some externally hosted files were ignored (use --allow-external to allow).
Cleaning up...
No distributions matching the version for psutil>=0.6.1,<1.0 (from -r /home/openstack/workspace/neutron/requirements.txt (line 19))

[David Peraza (dperaza)]

trying to figure out why psutil 1.0 was upper bound to begin with

[David Peraza (dperaza)]

Found this is being fixed here for Neutron: https://review.openstack.org/#/c/68993/

[Zhi Yan Liu (lzy-dev)]

https://review.openstack.org/#/c/59699/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/59699
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=d7f1221684ea3c958f0a382764074adb5ee6cbcf
Submitter: Jenkins
Branch: master

commit d7f1221684ea3c958f0a382764074adb5ee6cbcf
Author: Zhi Yan Liu <email address hidden>
Date: Tue Dec 3 23:32:07 2013 +1100

    Switch to testrepository for running tests

    OpenStack as a whole is moving towards using testrepository and
    testtools for running tests. To that end, bring Glance into line by
    switching it to use testrepository to run tests.

    This copies run_tests.sh and tools/colorizer.py from Nova.

    This change also has some minor changes to make run_test.sh work well.

    Partial fixes bug: 1179009
    Fixes bug: 1271806

    Change-Id: Ic265bc0d2f1528358f6e8ee5b4139f991923fc72
    Signed-off-by: Steve Kowalik <email address hidden>
    Signed-off-by: Zhi Yan Liu <email address hidden>