docs build fails for nova-specs

[Matt Riedemann]

I just cloned nova-specs and ran 'tox -e docs', I'm on Ubuntu Trusty 14.04 (no devstack), it fails with this:

mriedem@ubuntu:~/git/nova-specs$ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
mriedem@ubuntu:~/git/nova-specs$ tox -e docs
docs develop-inst-nodeps: /home/mriedem/git/nova-specs
docs runtests: PYTHONHASHSEED='2805332766'
docs runtests: commands[0] | python setup.py build_sphinx
running build_sphinx
Running Sphinx v1.1.3
loading pickled environment... not yet created
Using openstack theme from /home/mriedem/git/nova-specs/.tox/docs/local/lib/python2.7/site-packages/oslosphinx/theme
building [html]: all source files
updating environment: 7 added, 0 changed, 0 removed
reading sources... [100%] specs/kilo/template
scanning /home/mriedem/git/nova-specs/doc/source for redirects...
   found redirects at /home/mriedem/git/nova-specs/doc/source/specs/kilo/redirects
Traceback (most recent call last):
  File "setup.py", line 22, in <module>
    pbr=True)
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/mriedem/git/nova-specs/.tox/docs/local/lib/python2.7/site-packages/pbr/packaging.py", line 754, in run
    self._sphinx_run()
  File "/home/mriedem/git/nova-specs/.tox/docs/local/lib/python2.7/site-packages/pbr/packaging.py", line 715, in _sphinx_run
    app.build(force_all=self.all_files)
  File "/home/mriedem/git/nova-specs/.tox/docs/local/lib/python2.7/site-packages/sphinx/application.py", line 206, in build
    self.emit('build-finished', err)
  File "/home/mriedem/git/nova-specs/.tox/docs/local/lib/python2.7/site-packages/sphinx/application.py", line 314, in emit
    results.append(callback(self, *args))
  File "/home/mriedem/git/nova-specs/doc/source/redirect.py", line 49, in emit_redirects
    process_directory(app.builder.srcdir)
  File "/home/mriedem/git/nova-specs/doc/source/redirect.py", line 44, in process_directory
    process_directory(p)
  File "/home/mriedem/git/nova-specs/doc/source/redirect.py", line 44, in process_directory
    process_directory(p)
  File "/home/mriedem/git/nova-specs/doc/source/redirect.py", line 47, in process_directory
    process_redirect_file(app, path, ent)
  File "/home/mriedem/git/nova-specs/doc/source/redirect.py", line 23, in process_redirect_file
    from_path, to_path = line.rstrip().split(' ')
ValueError: need more than 1 value to unpack
ERROR: InvocationError: '/home/mriedem/git/nova-specs/.tox/docs/bin/python setup.py build_sphinx'
___________________________________________________________________________________________ summary ___________________________________________________________________________________________
ERROR: docs: commands failed

My virtualenv pip freeze output looks the same as what's in community runs:

(docs)mriedem@ubuntu:~/git/nova-specs$ pip freeze
Jinja2==2.7.3
MarkupSafe==0.23
Pygments==2.0.1
Sphinx==1.1.3
argparse==1.2.1
cssselect==0.9.1
docutils==0.12
extras==0.0.3
feedformatter==0.4
fixtures==1.0.0
lxml==3.4.1
-e git://git.openstack.org/openstack/nova-specs@ea08644e90efb680f1eeba5731d01315b12e8d6f#egg=nova_specs-bp/db2-database
oslosphinx==2.3.0
pbr==0.10.0
pyquery==1.2.9
python-mimeparse==0.1.4
python-subunit==1.0.0
six==1.8.0
testrepository==0.0.20
testtools==1.5.0
unittest2==0.8.0
wsgiref==0.1.2
yasfb==0.5.1
(docs)mriedem@ubuntu:~/git/nova-specs$

http://logs.openstack.org/87/136487/6/check/gate-nova-specs-python27/d617099/console.html#_2014-12-11_16_08_01_111

I'm assuming I'm missing some native site-package dependency, like sphinxcontrib or something, but I'd think that should be called out somewhere if it's needed to build docs.

[Sean Dague (sdague)]

My pip freeze:

Jinja2==2.7.3
MarkupSafe==0.23
Pygments==2.0.1
Sphinx==1.1.3
argparse==1.2.1
cssselect==0.9.1
docutils==0.9.1
extras==0.0.3
feedformatter==0.4
fixtures==1.0.0
lxml==3.4.1
-e git+http://github.com/openstack/nova-specs@ac85aa2af6de020dd8489b9b72eced143f5657f8#egg=nova_specs-master
oslosphinx==2.3.0
pbr==0.10.0
pyquery==1.2.9
python-mimeparse==0.1.4
python-subunit==1.0.0
six==1.8.0
testrepository==0.0.20
testtools==1.5.0
unittest2==0.8.0
wsgiref==0.1.2
yasfb==0.5.1

Works here.

[Sean Dague (sdague)]

did a rebuild, and got:

Jinja2==2.7.3
MarkupSafe==0.23
Pygments==2.0.1
Sphinx==1.1.3
argparse==1.2.1
cssselect==0.9.1
docutils==0.12
extras==0.0.3
feedformatter==0.4
fixtures==1.0.0
lxml==3.4.1
-e git+http://github.com/openstack/nova-specs@aebe9d38a9552d9cc059b70cb1f1d4cc7a375257#egg=nova_specs-master
oslosphinx==2.3.0
pbr==0.10.0
pyquery==1.2.9
python-mimeparse==0.1.4
python-subunit==1.0.0
six==1.8.0
testrepository==0.0.20
testtools==1.5.0
unittest2==0.8.0
wsgiref==0.1.2
yasfb==0.5.1

[Sean Dague (sdague)]

I honestly wonder if tox min ver of 1.4 is the problem. That broke some other folks recently.

[Matt Riedemann (mriedem)]

I have tox 1.8.1 and still hitting this.

[Matt Riedemann (mriedem)]

(docs)mriedem@ubuntu:~/git/nova-specs$ pip freeze
cssselect==0.9.1
docutils==0.12
extras==0.0.3
feedformatter==0.4
fixtures==1.0.0
Jinja2==2.7.3
linecache2==1.0.0
lxml==3.4.2
MarkupSafe==0.23
-e git://git.openstack.org/openstack/nova-specs@6c95de582a33e8f47c3adbd13bfb309784faf435#egg=nova_specs-review_sean_dague_liberty
oslosphinx==2.5.0
pbr==0.10.8
Pygments==2.0.2
pyquery==1.2.9
python-mimeparse==0.1.4
python-subunit==1.1.0
requests==2.6.0
six==1.9.0
Sphinx==1.1.3
testrepository==0.0.20
testtools==1.7.1
traceback2==1.4.0
unittest2==1.0.1
yasfb==0.5.1
(docs)mriedem@ubuntu:~/git/nova-specs$

[Matt Riedemann (mriedem)]

I upgrade tox to the latest, 1.9.2, and that didn't help. I had Sphinx 1.1.3 installed in site-packages, removed that and rebuilt tox but it still fails.

[Matt Riedemann (mriedem)]

Per Sean's suggestion, I also tried re-ordering requirements.txt so that pbr was first, sphinx was second, everything came after, removed everything under .tox, but those didn't help.

[Matt Riedemann (mriedem)]

BTW, this is my site-packages: http://paste.openstack.org/show/197612/

At one point I had Sphinx 1.2.3 installed in site-packages but I've since removed that. tox shouldn't be using site-packages anyway.

[Sean Dague (sdague)]

Do you have an old redirect.pyc file?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/169298

[Matt Riedemann (mriedem)]

Sean pointed out that a bunch of my files weren't symlinks like they should be. I'm using eclipse and when I clone the project outside eclipse sure enough the symlinks are there. The problem is when using eclipse with the pydev plugin, apparently it blows away symlinks, there is a ML thread on it here:

http://sourceforge.net/p/pydev/mailman/message/2199242/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/169298
Committed: https://git.openstack.org/cgit/openstack/nova-specs/commit/?id=2b2a322e1cc9a1515b9f8607a3a758ff9f6a4197
Submitter: Jenkins
Branch: master

commit 2b2a322e1cc9a1515b9f8607a3a758ff9f6a4197
Author: Sean Dague <email address hidden>
Date: Tue Mar 31 09:21:08 2015 -0400

    remove stale pyc before tests

    Stale pyc files can create massive confusion on failing
    tests. Extremely hard to debug. Remove all pyc files before we run
    tests.

    Change-Id: I67da587382af87389c2d04c23d84f13f341a1e46
    Related-Bug: #1401600