Setting autodoc_tree_index_modules makes documentation builds fail

[David Stanek]

The arguments originally being passed into sphinx.apidoc specified '.' as the path to index. Unfortunately this includes the setup.py module. Sphinx dies while trying to process the setup.rst likely because the setup.py module calls setuptools.setup() when imported causing some sort of recursion. The final result is something like:

  2013-12-08 21:08:12.088 | reading sources... [ 80%] api/setup
  2013-12-08 21:08:12.100 | /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'setup_requires'
  2013-12-08 21:08:12.101 | warnings.warn(msg)
  2013-12-08 21:08:12.102 | /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'pbr'
  2013-12-08 21:08:12.102 | warnings.warn(msg)
  2013-12-08 21:08:12.103 | usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
  2013-12-08 21:08:12.103 | or: setup.py --help [cmd1 cmd2 ...]
  2013-12-08 21:08:12.104 | or: setup.py --help-commands
  2013-12-08 21:08:12.104 | or: setup.py cmd --help
  2013-12-08 21:08:12.104 |
  2013-12-08 21:08:12.105 | error: invalid command 'build_sphinx'
  2013-12-08 21:08:12.622 | ERROR: InvocationError: '/home/jenkins/workspace/gate-keystone-docs/.tox/venv/bin/python setup.py build_sphinx'

[David Stanek (dstanek)]

Adding Keystone to make sure we update our code to use this when committed and remove the temporary extension.

[David Stanek (dstanek)]

I did submit a patch (https://review.openstack.org/#/c/61875) to fix this issue, but now that I am digging deeper into the magic I think I have found a simplier way. We just need to stop setup.py from being imported. Since when running 'python setup.py build_sphinx' the setup module is imported as __main__ we can just use that like:

    diff --git i/pbr/packaging.py w/pbr/packaging.py
    index a066d3b..37b41c6 100644
    --- i/pbr/packaging.py
    +++ w/pbr/packaging.py
    @@ -636,6 +636,9 @@ try:
         from sphinx import config
         from sphinx import setup_command

    + import sys
    + sys.modules['setup'] = sys.modules['__main__']
    +
         class LocalBuildDoc(setup_command.BuildDoc):

             command_name = 'build_sphinx'

This is sort of a hack and I'm not entirely sure where the best place to put those lines. Right now this is not on Keystone's critical path because I have created a workaround: http://git.openstack.org/cgit/openstack/keystone/commit/?id=3f27b309

[Benjamin Staffin (benley)]

An even simpler potential workaround is to add 'api/setup.rst' to the exclude_patterns list in Sphinx's conf.py. Pbr will still make it generate doc/source/api/setup.rst, but sphinx will then ignore it when building the docs, so it is rendered harmless.

[David Stanek (dstanek)]

@Benjamin, I don't think that will fix the problem. It's not just that I don't want the setup to be a part of the code documentation, it's that the process of creating the docs will import setup.py.

It may be possible to use your idea and wrap our setup function invocation in "if __name__ == '__main__'" to get around both issues.

[Doug Hellmann (doug-hellmann)]

We successfully use this feature in several projects now. Is this still a problem?

[David Stanek (dstanek)]

As far as I know it is still a problem. Other projects are using autodoc_tree_index_modules?

[Doug Hellmann (doug-hellmann)]

We have some patches up for oslo projects that use autodoc_index_modules. See https://review.openstack.org/#/c/119097/ for an example.

[David Stanek (dstanek)]

Doug, that's a different configuration option. It's autodoc_tree_index_modules that's broken. I have a more acceptable (I hope!) patch on the way.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/120216

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/121667

[David Stanek (dstanek)]

Here is the bug in action!

With an invalid exclude path set the documentation built correctly, but included the setup module: http://docs-draft.openstack.org/67/121667/1/check/gate-python-keystoneclient-docs/7f9b0f3/doc/build/html/api/modules.html

When I corrected the commit to use api/setup.rst as recommended I get an error because Keystone we turn warnings into errors. From https://jenkins04.openstack.org/job/gate-python-keystoneclient-docs/570/console :
    2014-09-23 22:59:46.925 | raise SphinxWarning(warntext)
    2014-09-23 22:59:46.926 | sphinx.errors.SphinxWarning: /home/jenkins/workspace/gate-python-keystoneclient-docs/doc/source/api/modules.rst:4: WARNING: toctree contains reference to nonexisting document u'api/setup'
    2014-09-23 22:59:46.926 |
    2014-09-23 22:59:47.066 | ERROR: InvocationError: '/home/jenkins/workspace/gate-python-keystoneclient-docs/.tox/venv/bin/python setup.py build_sphinx'

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/120216
Committed: https://git.openstack.org/cgit/openstack-dev/pbr/commit/?id=0d6bfaf2e3f839704f07b265ba81e224936a2b49
Submitter: Jenkins
Branch: master

commit 0d6bfaf2e3f839704f07b265ba81e224936a2b49
Author: David Stanek <email address hidden>
Date: Tue Sep 9 20:25:27 2014 +0000

    Adds option for excluding files from autodoc trees

    The arguments originally being passed into sphinx.apidoc specify '.'
    as the path to index. Unfortunately this includes the setup.py module.
    This causes Sphinx to complain or break depending on the configuration.

    This patch ignores setup.py by default and allows the project to
    override it in their setup.cfg.

    Change-Id: I7c164d42a096ba1a0daf314a689cc3252ca81c49
    Closes-Bug: #1260495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/121667
Committed: https://git.openstack.org/cgit/openstack/python-keystoneclient/commit/?id=045e47938f445741b6efedeb7243d3b413ea3229
Submitter: Jenkins
Branch: master

commit 045e47938f445741b6efedeb7243d3b413ea3229
Author: David Stanek <email address hidden>
Date: Mon Sep 15 18:38:25 2014 +0000

    Removes temporary fix for doc generation

    A temporary fix was added to get around a bug in how pbr handles
    its autodoc_tree_index_modules setting. Since this bug is fixed we no
    longer need the work around.

    Change-Id: Id8274ef5c244bf50a34702ed9b4e50d3b82d8028
    Closes-Bug: #1260495

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/182643

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/182643
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=6f98a9e2bd7eed7f618a453d18055c34f84cd1ca
Submitter: Jenkins
Branch: master

commit 6f98a9e2bd7eed7f618a453d18055c34f84cd1ca
Author: David Stanek <email address hidden>
Date: Wed May 13 12:01:16 2015 +0000

    Removes temporary fix for doc generation

    A temporary fix was added to get around a bug in how pbr handles
    its autodoc_tree_index_modules setting. Since this bug is fixed we no
    longer need the work around.

    Change-Id: I6af0fdd6d1efacb47692b89c329e45ac59fef7cb
    Closes-Bug: #1260495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/199609
Committed: https://git.openstack.org/cgit/openstack-dev/pbr/commit/?id=07292c2d8437ca3e10669038d1036f6903e54c80
Submitter: Jenkins
Branch: master

commit 07292c2d8437ca3e10669038d1036f6903e54c80
Author: Matt Riedemann <email address hidden>
Date: Wed Jul 8 07:35:20 2015 -0700

    Add more documentation around building docs

    After digging in pbr and sphinx source code for a day to figure out
    what I was doing wrong, let's update the pbr docs with respect to
    building docs using the autodoc features in pbr.

    Specifically, document the autodoc_tree_excludes and warnerrors
    options and point out that you will probably need to set the
    exclude_patterns options in your sphinx config when excluding
    modules from the docs build and warnerrors=True. Also provide
    some links to the Sphinx docs for more details.

    Related-Bug: #1260495
    Related-Bug: #1472276

    Change-Id: Ib43830d08a156f37a3763ad2807d5c1f4cdbe3fe