Nova unit tests fail on CentOS 6 when python-jinja2 package is installed

[Jeremy Stanley]

The requests library released 1.2.1 today which brings in sphinx as a new dependency. One of sphinx's transitive dependencies is on a newer version of jinja than is provided in the python-jinja2 package on CentOS 6, and since nova's unit tests allow site-packages in the venv for libvirt support this cap is needed so that gate-nova-python26 will continue to work.

2013-05-20 22:58:19.611 | Downloading/unpacking requests>=0.8 (from python-cinderclient>=1.0.1->-r /home/jenkins/workspace/gate-nova-python26/tools/pip-requires (line 25))
2013-05-20 22:58:19.611 | Running setup.py egg_info for package requests
2013-05-20 22:58:19.611 | Couldn't find index page for 'sphinx' (maybe misspelled?)
2013-05-20 22:58:19.611 | warning: no files found matching 'README'
2013-05-20 22:58:19.611 | no previously-included directories found matching 'doc/_build'
2013-05-20 22:58:19.611 |
2013-05-20 22:58:19.611 | Installed /home/jenkins/workspace/gate-nova-python26/.tox/py26/build/requests/Sphinx-1.2b1-py2.6.egg
2013-05-20 22:58:19.611 | Searching for docutils>=0.7
2013-05-20 22:58:19.611 | Reading http://pypi.openstack.org/openstack/docutils/
2013-05-20 22:58:19.611 | Reading http://pypi.openstack.org/openstack/docutils/?C=S;O=A
2013-05-20 22:58:19.612 | Reading http://pypi.openstack.org/openstack/docutils/?C=N;O=D
2013-05-20 22:58:19.612 | Reading http://pypi.openstack.org/openstack/docutils/?C=D;O=A
2013-05-20 22:58:19.612 | Reading http://pypi.openstack.org/openstack/docutils/?C=M;O=A
2013-05-20 22:58:19.612 | Best match: docutils 0.10
2013-05-20 22:58:19.612 | Downloading http://pypi.openstack.org/openstack/docutils/docutils-0.10.tar.gz
2013-05-20 22:58:19.612 | Processing docutils-0.10.tar.gz
2013-05-20 22:58:19.612 | Writing /tmp/tmp.xs1DIWN3pg/easy_install-ma2nIo/docutils-0.10/setup.cfg
2013-05-20 22:58:19.612 | Running docutils-0.10/setup.py -q bdist_egg --dist-dir /tmp/tmp.xs1DIWN3pg/easy_install-ma2nIo/docutils-0.10/egg-dist-tmp-DJs1s0
2013-05-20 22:58:19.612 | warning: no files found matching 'MANIFEST'
2013-05-20 22:58:19.612 | warning: no files found matching '*' under directory 'extras'
2013-05-20 22:58:19.612 | warning: no previously-included files matching '.cvsignore' found under directory '*'
2013-05-20 22:58:19.613 | warning: no previously-included files matching '*.pyc' found under directory '*'
2013-05-20 22:58:19.613 | warning: no previously-included files matching '*~' found under directory '*'
2013-05-20 22:58:19.613 | warning: no previously-included files matching '.DS_Store' found under directory '*'
2013-05-20 22:58:19.613 | zip_safe flag not set; analyzing archive contents...
2013-05-20 22:58:19.613 | docutils.parsers.rst.directives.misc: module references __file__
2013-05-20 22:58:19.613 | docutils.writers.docutils_xml: module references __path__
2013-05-20 22:58:19.613 | docutils.writers.latex2e.__init__: module references __file__
2013-05-20 22:58:19.613 | docutils.writers.odf_odt.__init__: module references __file__
2013-05-20 22:58:19.613 | docutils.writers.s5_html.__init__: module references __file__
2013-05-20 22:58:19.613 | docutils.writers.html4css1.__init__: module references __file__
2013-05-20 22:58:19.613 | docutils.writers.pep_html.__init__: module references __file__
2013-05-20 22:58:19.613 |
2013-05-20 22:58:19.613 | Installed /home/jenkins/workspace/gate-nova-python26/.tox/py26/build/requests/docutils-0.10-py2.6.egg
2013-05-20 22:58:19.613 | Traceback (most recent call last):
2013-05-20 22:58:19.614 | File "<string>", line 16, in <module>
2013-05-20 22:58:19.614 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/build/requests/setup.py", line 55, in <module>
2013-05-20 22:58:19.614 | 'Programming Language :: Python :: 3.3',
2013-05-20 22:58:19.614 | File "/usr/lib64/python2.6/distutils/core.py", line 113, in setup
2013-05-20 22:58:19.614 | _setup_distribution = dist = klass(attrs)
2013-05-20 22:58:19.614 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/distribute-0.6.34-py2.6.egg/setuptools/dist.py", line 221, in __init__
2013-05-20 22:58:19.614 | self.fetch_build_eggs(attrs.pop('setup_requires'))
2013-05-20 22:58:19.614 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/distribute-0.6.34-py2.6.egg/setuptools/dist.py", line 245, in fetch_build_eggs
2013-05-20 22:58:19.614 | parse_requirements(requires), installer=self.fetch_build_egg
2013-05-20 22:58:19.615 | File "/home/jenkins/workspace/gate-nova-python26/.tox/py26/lib/python2.6/site-packages/distribute-0.6.34-py2.6.egg/pkg_resources.py", line 598, in resolve
2013-05-20 22:58:19.615 | raise VersionConflict(dist,req) # XXX put more info here
2013-05-20 22:58:19.615 | pkg_resources.VersionConflict: (Jinja2 2.2.1 (/usr/lib64/python2.6/site-packages), Requirement.parse('Jinja2>=2.3'))
2013-05-20 22:58:19.615 | Complete output from command python setup.py egg_info:
2013-05-20 22:58:19.615 | Couldn't find index page for 'sphinx' (maybe misspelled?)

[Jeremy Stanley (fungi)]

One wrinkle in this is that requests is actually a transitive dependency of nova via python-cinderclient, so we may need to cap it there and get them to tag a new release (and possibily update nova's dependency on python-cinderclient to point to a master branch tarball in the interim).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/29852

[Jeremy Stanley (fungi)]

This situation does not actually seem to be impacting stable/grizzly based on the most recent bitrot job... http://logs.openstack.org/periodic/periodic-nova-python26-stable-grizzly/41/console.html.gz

I don't think it's affecting stable/folsom either, but that job's no longer relevant since (gate|periodic)-nova-python26-stable-folsom never worked on CentOS anyway.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/29852
Committed: http://github.com/openstack/nova/commit/c39687edb9138c6eac675b8781a5d4bf5a51c968
Submitter: Jenkins
Branch: master

commit c39687edb9138c6eac675b8781a5d4bf5a51c968
Author: Jeremy Stanley <email address hidden>
Date: Tue May 21 00:28:24 2013 +0000

    Add requests requirement capped <1.2.1.

    Fixes bug 1182271.

    * tools/pip-requires: The requests library released 1.2.1 today
    which brings in sphinx as a new dependency. One of sphinx's
    transitive dependencies is on a newer version of jinja than is
    provided in the python-jinja2 package on CentOS 6, and since nova's
    unit tests allow site-packages in the venv for libvirt support this
    cap is needed so that gate-nova-python26 will continue to work.
    Because requests is actually a dependency of python-cinderclient but
    is only breaking nova's unit tests, it has to be added and come
    before python-cinderclient in the pip-requires list. A separate
    change has been proposed to openstack/requirements which will need
    to merge first. https://review.openstack.org/29850

    Change-Id: Ia198192d6231f71272d3aa4f5942828a45d86ceb

[Jeremy Stanley (fungi)]

Removed grizzly-backport-potential tag after confirming that this is not impacting stable/grizzly because sitepackages = True was not actually used until commit dbbbbf4 when it was added to the testenv section of tox.ini.

[Pádraig Brady (p-draigbrady)]

Issues with the package version pin introduced with the fix for this issue
are being discussed at https://review.openstack.org/#/c/37461

We're probably going to revert this change.

Would it be possible to install the centos version of sphinx that is compatible with its version of jinja2?

If that's not possible would it be possible to install a newer jinja2 package on the gate system?
For example if you ensured that python-jinja2-26 was installed first from EPEL or directly from http://kojipkgs.fedoraproject.org/packages/python-jinja2-26/, and then that the default python-jinja2
package was not installed, then this shouldn't be an issue right?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/38011

[Jeremy Stanley (fungi)]

Abandoned that change. Apparently it should now be possible to simply uncap requests since we've started doing a pip install -U to work around the previoys site packages related breakage. I'll propose that revert to nova instead and see how it fares.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/38012

[Jeremy Stanley (fungi)]

This merged to nova master on August 1, but failed to mark the bug fix-committed for some reason.