Unable to install software dependencies due to: "Cannot fetch index base URL"

[Peter Portante]

Some gate jobs fail because they were not able to install software dependencies due to the following error: "Cannot fetch index base URL"

E.g. see: http://logs.openstack.org/04/65604/3/gate/gate-swift-python26/025c813/console.html

2014-01-18 16:20:52.838 | + tox -epy26
2014-01-18 16:20:52.980 | py26 create: /home/jenkins/workspace/gate-swift-python26/.tox/py26
2014-01-18 16:20:54.888 | py26 installdeps: -r/home/jenkins/workspace/gate-swift-python26/requirements.txt, -r/home/jenkins/workspace/gate-swift-python26/test-requirements.txt
2014-01-18 16:21:15.146 | ERROR: invocation failed, logfile: /home/jenkins/workspace/gate-swift-python26/.tox/py26/log/py26-1.log
2014-01-18 16:21:15.146 | ERROR: actionid=py26
2014-01-18 16:21:15.146 | msg=getenv
2014-01-18 16:21:15.147 | cmdargs=[local('/home/jenkins/workspace/gate-swift-python26/.tox/py26/bin/pip'), 'install', '--allow-external', 'netifaces', '--allow-insecure', 'netifaces', '-U', '-r/home/jenkins/workspace/gate-swift-python26/requirements.txt', '-r/home/jenkins/workspace/gate-swift-python26/test-requirements.txt']
2014-01-18 16:21:15.147 | env={'JENKINS_HOME': '/var/lib/jenkins', 'PYTHONIOENCODING': 'utf_8', 'BUILD_CAUSE': 'USERIDCAUSE', 'HUDSON_URL': 'https://jenkins02.openstack.org/', 'BUILD_URL': 'https://jenkins02.openstack.org/job/gate-swift-python26/3305/', 'BUILD_TAG': 'jenkins-gate-swift-python26-3305', 'SSH_CLIENT': '166.78.48.99 46771 22', 'JENKINS_URL': 'https://jenkins02.openstack.org/', 'LOGNAME': 'jenkins', 'USER': 'jenkins', 'WORKSPACE': '/home/jenkins/workspace/gate-swift-python26', 'HOME': '/home/jenkins', 'PATH': '/home/jenkins/workspace/gate-swift-python26/.tox/py26/bin:/usr/local/bin:/bin:/usr/bin', 'ZUUL_CHANGES': 'openstack/nova:master:refs/changes/35/62835/6^openstack/python-swiftclient:master:refs/changes/06/67106/1^openstack/python-swiftclient:master:refs/changes/50/62850/5^openstack/python-swiftclient:master:refs/changes/38/63838/2^openstack/heat:master:refs/changes/85/59685/10^openstack-dev/grenade:master:refs/changes/34/66934/1^openstack-dev/grenade:master:refs/changes/00/67300/1^openstack/python-neutronclient:master:refs/changes/28/63328/3^openstack/keystone:master:refs/changes/75/64575/15^openstack/keystone:master:refs/changes/04/67204/1^openstack/horizon:master:refs/changes/07/66807/1^openstack/heat:master:refs/changes/14/63914/4^openstack/nova:master:refs/changes/33/64933/2^openstack/keystone:master:refs/changes/80/66680/1^openstack/keystone:master:refs/changes/81/66681/1^openstack/keystone:master:refs/changes/82/66682/1^openstack/python-glanceclient:master:refs/changes/49/67149/1^openstack/glance:master:refs/changes/26/66626/2^openstack/keystone:master:refs/changes/27/64427/13^openstack/keystone:master:refs/changes/11/57811/30^openstack/heat:master:refs/changes/42/64142/6^openstack/neutron:master:refs/changes/09/66609/1^openstack/keystone:master:refs/changes/08/66208/3^openstack/keystone:master:refs/changes/09/66209/2^openstack/swift:master:refs/changes/20/67220/1^openstack/python-novaclient:master:refs/changes/95/60495/2^openstack/python-novaclient:master:refs/changes/80/65680/1^openstack/heat:master:refs/changes/88/65888/2^openstack/cinder:master:refs/changes/00/65900/5^openstack/horizon:master:refs/changes/27/67027/3^openstack/horizon:master:refs/changes/90/65490/5^openstack/nova:master:refs/changes/38/55038/6^openstack/requirements:master:refs/changes/31/64831/2^openstack/glance:master:refs/changes/99/66799/1^openstack/python-ceilometerclient:master:refs/changes/36/67136/1^openstack/python-cinderclient:master:refs/changes/34/67134/1^openstack-dev/devstack:master:refs/changes/25/67325/3^openstack/swift:master:refs/changes/98/65398/2^openstack/nova:master:refs/changes/20/66520/2^openstack/nova:master:refs/changes/96/54096/24^openstack/nova:master:refs/changes/19/55419/4^openstack/cinder:master:refs/changes/67/67367/1^openstack/swift:master:refs/changes/15/66615/3^openstack/swift:master:refs/changes/83/66883/5^openstack/heat:master:refs/changes/30/67030/2^openstack/heat:master:refs/changes/46/66646/2^openstack/heat:master:refs/changes/47/66647/2^openstack/heat:master:refs/changes/48/66648/2^openstack/nova:stable/havana:refs/changes/36/57836/3^openstack/heat:master:refs/changes/32/63932/5^openstack/python-heatclient:master:refs/changes/22/66222/1^openstack/python-heatclient:master:refs/changes/23/66223/1^openstack/python-heatclient:master:refs/changes/24/66224/2^openstack/python-heatclient:master:refs/changes/25/66225/2^openstack/tempest:master:refs/changes/93/66293/7^openstack/glance:master:refs/changes/64/67364/2^openstack/ceilometer:master:refs/changes/57/66657/3^openstack/tempest:master:refs/changes/71/66871/11^openstack/tempest:master:refs/changes/72/67172/4^openstack/tempest:master:refs/changes/60/65260/4^openstack/tempest:master:refs/changes/67/67167/1^openstack/tempest:master:refs/changes/51/65751/1^openstack/swift:master:refs/changes/09/66909/1^openstack/swift:master:refs/changes/61/67561/1^openstack/tempest:master:refs/changes/18/67218/2^openstack/neutron:master:refs/changes/75/67475/2^openstack/tempest:master:refs/changes/70/66970/7^openstack/keystone:master:refs/changes/87/64587/10^openstack/keystone:master:refs/changes/49/64749/10^openstack/keystone:master:refs/changes/89/64589/14^openstack/keystone:master:refs/changes/58/64758/11^openstack/tempest:master:refs/changes/71/67371/1^openstack/neutron:master:refs/changes/58/67558/2^openstack/tempest:master:refs/changes/27/63627/4^openstack/tempest:master:refs/changes/60/64660/5^openstack/tempest:master:refs/changes/73/62573/5^openstack/tempest:master:refs/changes/95/66795/2^openstack/swift:master:refs/changes/99/65399/3^openstack/neutron:master:refs/changes/36/66736/1^openstack/swift:master:refs/changes/16/66916/1^openstack/neutron:master:refs/changes/37/67537/1^openstack-dev/devstack:stable/grizzly:refs/changes/25/67425/2^openstack/neutron:stable/havana:refs/changes/70/60570/4^openstack/neutron:stable/havana:refs/changes/71/60571/3^openstack/neutron:stable/havana:refs/changes/72/60572/3^openstack/cinder:master:refs/changes/62/65662/2^openstack/horizon:master:refs/changes/54/65254/1^openstack/horizon:master:refs/changes/55/65255/1^openstack/swift:master:refs/changes/04/65604/3', 'DEBFULLNAME': 'OpenStack Hudson', 'JOB_NAME': 'gate-swift-python26', 'ZUUL_PROJECT': 'openstack/swift', 'NOSE_WITH_COVERAGE': '1', 'VIRTUAL_ENV': '/home/jenkins/workspace/gate-swift-python26/.tox/py26', 'SHELL': '/bin/bash', 'SHLVL': '3', 'ZUUL_CHANGE': '65604', 'NOSE_OPENSTACK_RED': '0.05', 'ZUUL_UUID': '025c81399bae485aafcb539df00507a8', 'HUDSON_HOME': '/var/lib/jenkins', 'ZUUL_CHANGE_IDS': '62835,6 67106,1 62850,5 63838,2 59685,10 66934,1 67300,1 63328,3 64575,15 67204,1 66807,1 63914,4 64933,2 66680,1 66681,1 66682,1 67149,1 66626,2 64427,13 57811,30 64142,6 66609,1 66208,3 66209,2 67220,1 60495,2 65680,1 65888,2 65900,5 67027,3 65490,5 55038,6 64831,2 66799,1 67136,1 67134,1 67325,3 65398,2 66520,2 54096,24 55419,4 67367,1 66615,3 66883,5 67030,2 66646,2 66647,2 66648,2 57836,3 63932,5 66222,1 66223,1 66224,2 66225,2 66293,7 67364,2 66657,3 66871,11 67172,4 65260,4 67167,1 65751,1 66909,1 67561,1 67218,2 67475,2 66970,7 64587,10 64749,10 64589,14 64758,11 67371,1 67558,2 63627,4 64660,5 62573,5 66795,2 65399,3 66736,1 66916,1 67537,1 67425,2 60570,4 60571,3 60572,3 65662,2 65254,1 65255,1 65604,3', 'NODE_LABELS': 'centos6 centos6-4', 'NOSE_OPENSTACK_YELLOW': '0.025', 'TMPDIR': '/tmp/tmp.1w7zGsoEOr', 'DEBEMAIL': '<email address hidden>', 'TOX_INDEX_URL': 'http://pypi.openstack.org/openstack', 'BUILD_ID': '2014-01-18_16-20-49', 'BUILD_NUMBER': '3305', 'NOSE_WITH_OPENSTACK': '1', 'HUDSON_COOKIE': '1912cb0a-cce3-4d3a-80db-e40ad155ff57', 'JOB_URL': 'https://jenkins02.openstack.org/job/gate-swift-python26/', 'NOSE_WITH_HTML_OUTPUT': '1', 'ZUUL_PATCHSET': '3', 'BASE_LOG_PATH': '04/65604/3/gate', 'ZUUL_BRANCH': 'master', 'BUILD_CAUSE_USERIDCAUSE': 'true', 'ZUUL_COMMIT': 'ba0b4a31cabd2b0d543f8f392d8aa946c49f17eb', 'ZUUL_URL': 'http://zuul.openstack.org/p', 'NLSPATH': '/usr/dt/lib/nls/msg/%L/%N.cat', 'NOSE_WITH_XUNIT': '1', '_': '/usr/bin/tox', 'NOSE_COVER_BRANCHES': '1', 'NOSE_OPENSTACK_COLOR': '1', 'ZUUL_PIPELINE': 'gate', 'NOSE_OPENSTACK_SHOW_ELAPSED': '1', 'HUDSON_SERVER_COOKIE': 'a8c4be63eae9909c', 'EXECUTOR_NUMBER': '0', 'NODE_NAME': 'centos6-4', 'LOG_PATH': '04/65604/3/gate/gate-swift-python26/025c813', 'PWD': '/home/jenkins/workspace/gate-swift-python26', 'NOSE_OPENSTACK_STDOUT': '1', 'JENKINS_SERVER_COOKIE': 'a8c4be63eae9909c', 'BUILD_DISPLAY_NAME': '#3305', 'ZUUL_REF': 'refs/zuul/master/Za5c39f961aae414ca5a0268937bc65a9', 'XFILESEARCHPATH': '/usr/dt/app-defaults/%L/Dt', 'MAIL': '/var/mail/jenkins', 'SSH_CONNECTION': '166.78.48.99 46771 166.78.7.197 22', 'NOSE_HTML_OUT_FILE': 'nose_results.html'}
2014-01-18 16:21:15.149 | Downloading/unpacking dnspython>=1.9.4 (from -r /home/jenkins/workspace/gate-swift-python26/requirements.txt (line 1))
2014-01-18 16:21:15.150 | Running setup.py egg_info for package dnspython
2014-01-18 16:21:15.152 | Downloading/unpacking eventlet>=0.9.15 (from -r /home/jenkins/workspace/gate-swift-python26/requirements.txt (line 2))
2014-01-18 16:21:15.153 | Running setup.py egg_info for package eventlet
2014-01-18 16:21:15.154 | Downloading/unpacking greenlet>=0.3.1 (from -r /home/jenkins/workspace/gate-swift-python26/requirements.txt (line 3))
2014-01-18 16:21:15.154 | Running setup.py egg_info for package greenlet
2014-01-18 16:21:15.155 | Downloading/unpacking netifaces>=0.5 (from -r /home/jenkins/workspace/gate-swift-python26/requirements.txt (line 4))
2014-01-18 16:21:15.155 | Downloading netifaces-0.8.tar.gz
2014-01-18 16:21:15.155 | Running setup.py egg_info for package netifaces
2014-01-18 16:21:15.156 | Downloading/unpacking pastedeploy>=1.3.3 (from -r /home/jenkins/workspace/gate-swift-python26/requirements.txt (line 5))
2014-01-18 16:21:15.156 | Cannot fetch index base URL http://pypi.openstack.org/openstack/
2014-01-18 16:21:15.156 | Could not find any downloads that satisfy the requirement pastedeploy>=1.3.3 (from -r /home/jenkins/workspace/gate-swift-python26/requirements.txt (line 5))
2014-01-18 16:21:15.157 | Cleaning up...
2014-01-18 16:21:15.157 | No distributions at all found for pastedeploy>=1.3.3 (from -r /home/jenkins/workspace/gate-swift-python26/requirements.txt (line 5))
2014-01-18 16:21:15.157 | Storing complete log in /home/jenkins/.pip/pip.log
2014-01-18 16:21:15.158 |
2014-01-18 16:21:15.158 | ERROR: could not install deps [-r/home/jenkins/workspace/gate-swift-python26/requirements.txt, -r/home/jenkins/workspace/gate-swift-python26/test-requirements.txt]

Perhaps we should also be saving the pip log files?

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/68965

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/68965
Committed: https://git.openstack.org/cgit/openstack-infra/elastic-recheck/commit/?id=8ba68be313e6ad25e9cf96a838eab0e32775cf60
Submitter: Jenkins
Branch: master

commit 8ba68be313e6ad25e9cf96a838eab0e32775cf60
Author: Peter Portante <email address hidden>
Date: Fri Jan 24 11:55:18 2014 -0500

    Add fingerprint for bug 1272417

    Related-Bug: 1272417

    Change-Id: I0faae94ee3832d4f522fd3f503a1f1eae7d6e653

[Anita Kuno (anteaya)]

More logs: http://logs.openstack.org/85/61985/5/gate/gate-tempest-dsvm-large-ops/115e4bc/logs/devstacklog.txt.gz

[Derek Higgins (derekh)]

I've been debugging a similar issue in tripleo-ci jobs
https://bugs.launchpad.net/tripleo/+bug/1292141

From what I can see the error occurs when pip requests the index and doesn't receive the response before a 15 second timeout

pip has two ways it tries to get the path to the package
GET /openstack/packagename/
GET /openstack/ (the returned listing is then searched with a non case sensitive search)

If we assume a small number of responses are delayed passed the 15 second timeout then in most cases the problem doesn't manifest itself because one of the two requests gets a hit

Most occurrences of this (if not all) are happening in cases where the subdirectory name and package name don't have matching cases, in this scenario then the first HTTP GET always gets a 404 and we are left with only the get to the main index page (effectively there is no fall-back), so these packages are more likely to see an error due to the 15 second timeout.

How about I submit an Increase to the timeout to see if it improves matters?
export PIP_DEFAULT_TIMEOUT=30

[Clark Boylan (cboylan)]

I think that is definitely worth a shot. And we should be able to have elastic recheck let us know if it helps at all.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/81664

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/81665

[Clark Boylan (cboylan)]

Marked triage as we have a potential fix in progress and don't want this bug to go away.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/81664
Committed: https://git.openstack.org/cgit/openstack-infra/config/commit/?id=30721474fc204821e52c0d08d791f91f36ac11ca
Submitter: Jenkins
Branch: master

commit 30721474fc204821e52c0d08d791f91f36ac11ca
Author: Derek Higgins <email address hidden>
Date: Wed Mar 19 23:05:21 2014 +0000

    Increase the default pip socket timeout

    pip's default timeout of 15 seconds doesn't seem to be high enough
    causing us to intermittently fail getting the pypi index. Increasing
    the timeout to 60 in an attempt to make it less likely to hit the
    problem.

    Change-Id: Ifd4689283e53d6afb7a76b480e004cdde3b302f6
    Related-Bug: #1272417

[Sean Dague (sdague)]

The last fix really didn't impact this at all.

[Derek Higgins (derekh)]

For some reason I can't currently see log results beyond a week, but iirc when I checked the fix did impact the problem (although possibly didn't get rid of it completely), so the fix just wasn't enough or the problem is getting worse.

[Matt Riedemann (mriedem)]

Doesn't seem to be an issue anymore.