autodoc_exclude_modules doesn't help if warnerrors=True

[Matt Riedemann]

(docs)mriedem@ubuntu:~/git/nova$ pip show pbr
You are using pip version 7.0.3, however version 7.1.0 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
---
Metadata-Version: 2.0
Name: pbr
Version: 1.2.0
Summary: Python Build Reasonableness
Home-page: https://launchpad.net/pbr
Author: OpenStack
Author-email: <email address hidden>
License: UNKNOWN
Location: /home/mriedem/git/nova/.tox/docs/lib/python2.7/site-packages
Requires:
(docs)mriedem@ubuntu:~/git/nova$

--

I'm trying to fix nova bug 1471934 and setting autodoc_index_modules=True in setup.cfg for nova.

I'm hitting a problem with autodoc for this sub-module:

https://github.com/openstack/nova/tree/master/nova/wsgi

The problem is there is no __init__.py for nova.wsgi (at least I think that's the problem).

I keep hitting this with tox -e docs:

reading sources... [100%] vmstates
2015-07-07 07:03:53.640 22143 CRITICAL nova [req-6066bd01-5090-4743-81ce-457fc731d778 fake fake - - -] SphinxWarning: /home/mriedem/git/nova/doc/source/api/nova.wsgi.nova-api.rst:4: WARNING: invalid signature for automodule (u'nova.wsgi.nova-api')

2015-07-07 07:03:53.640 22143 ERROR nova Traceback (most recent call last):
2015-07-07 07:03:53.640 22143 ERROR nova File "setup.py", line 29, in <module>
2015-07-07 07:03:53.640 22143 ERROR nova pbr=True)
2015-07-07 07:03:53.640 22143 ERROR nova File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
2015-07-07 07:03:53.640 22143 ERROR nova dist.run_commands()
2015-07-07 07:03:53.640 22143 ERROR nova File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
2015-07-07 07:03:53.640 22143 ERROR nova self.run_command(cmd)
2015-07-07 07:03:53.640 22143 ERROR nova File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
2015-07-07 07:03:53.640 22143 ERROR nova cmd_obj.run()
2015-07-07 07:03:53.640 22143 ERROR nova File "/home/mriedem/git/nova/.tox/docs/local/lib/python2.7/site-packages/pbr/builddoc.py", line 189, in run
2015-07-07 07:03:53.640 22143 ERROR nova self._sphinx_run()
2015-07-07 07:03:53.640 22143 ERROR nova File "/home/mriedem/git/nova/.tox/docs/local/lib/python2.7/site-packages/pbr/builddoc.py", line 143, in _sphinx_run
2015-07-07 07:03:53.640 22143 ERROR nova app.build(force_all=self.all_files)
2015-07-07 07:03:53.640 22143 ERROR nova File "/home/mriedem/git/nova/.tox/docs/local/lib/python2.7/site-packages/sphinx/application.py", line 216, in build
2015-07-07 07:03:53.640 22143 ERROR nova self.builder.build_all()
2015-07-07 07:03:53.640 22143 ERROR nova File "/home/mriedem/git/nova/.tox/docs/local/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 177, in build_all
2015-07-07 07:03:53.640 22143 ERROR nova self.build(None, summary='all source files', method='all')
2015-07-07 07:03:53.640 22143 ERROR nova File "/home/mriedem/git/nova/.tox/docs/local/lib/python2.7/site-packages/sphinx/builders/__init__.py", line 239, in build
2015-07-07 07:03:53.640 22143 ERROR nova self.warn(*warning)
2015-07-07 07:03:53.640 22143 ERROR nova File "/home/mriedem/git/nova/.tox/docs/local/lib/python2.7/site-packages/sphinx/application.py", line 268, in warn
2015-07-07 07:03:53.640 22143 ERROR nova raise SphinxWarning(warntext)
2015-07-07 07:03:53.640 22143 ERROR nova SphinxWarning: /home/mriedem/git/nova/doc/source/api/nova.wsgi.nova-api.rst:4: WARNING: invalid signature for automodule (u'nova.wsgi.nova-api')
2015-07-07 07:03:53.640 22143 ERROR nova
2015-07-07 07:03:53.640 22143 ERROR nova

So I added autodoc_exclude_modules = nova.wsgi.* to setup.cfg:

[pbr]
autodoc_index_modules = True
autodoc_exclude_modules = nova.wsgi.*
warnerrors = true

But it still fails. The fnmatch should be fine though:

Python 2.7.6 (default, Jun 22 2015, 18:00:18)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import fnmatch
>>>
>>> test = 'nova.wsgi.nova-api'
>>> test
'nova.wsgi.nova-api'
>>> pattern = 'nova.wsgi.*'
>>> fnmatch.fnmatch(test, pattern)
True

[Matt Riedemann (mriedem)]

I added some debug code here:

https://github.com/openstack-dev/pbr/blob/master/pbr/builddoc.py#L80

And it is parsing the option correctly:

running build_sphinx
[pbr] Writing ChangeLog
[pbr] Generating ChangeLog
[pbr] Generating AUTHORS
[pbr] Autodocumenting from /home/mriedem/git/nova
Traceback (most recent call last):
  File "setup.py", line 29, in <module>
    pbr=True)
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/mriedem/git/nova/.tox/docs/local/lib/python2.7/site-packages/pbr/builddoc.py", line 181, in run
    [None, ""])[1].split()))
  File "/home/mriedem/git/nova/.tox/docs/local/lib/python2.7/site-packages/pbr/builddoc.py", line 84, in generate_autoindex
    excluded_modules)
Exception: [pbr] Excluded modules: set(['nova.wsgi.*'])

[Matt Riedemann (mriedem)]

Ah I see the problem, autodoc_exclude_modules works fine, the nova.wsgi.* stuff is skipped here:

[pbr] Generating doc/source/api/nova.weights.rst
[pbr] Generating doc/source/api/nova.wsgi.rst
Running Sphinx v1.2.3

The problem is when running the sphinx html builder due to having warnerrors=True set in setup.cfg, there is no exclusion list passed to the sphinx html builder so it processes all files.

[Matt Riedemann (mriedem)]

It looks like we could possibly set the exclude_patterns key in confoverrides here:

https://github.com/openstack-dev/pbr/blob/master/pbr/builddoc.py#L125

Before running sphinx. The problem would be translating the fnmatch autodoc excludes that pbr supports vs the regex patterns that sphinx supports with exclude_patterns:

http://sphinx-doc.org/config.html

https://github.com/sphinx-doc/sphinx/blob/1.2.3/sphinx/config.py#L53

fnmatch is looking for module matches but exclude_patterns is looking for filename/path patterns, so we'd have to translate that all somehow.

[Matt Riedemann (mriedem)]

Probably the easiest solution, albeit redundant, would be to have a new pbr config option that is the sphinx exclude_patterns value and we just pass that in, and people only need to use it if using warnerrors=True.

[Matt Riedemann (mriedem)]

I eventually got exclude_patterns to work with hard-coding something like this:

confoverrides['exclude_patterns'] = ['**/nova.wsgi.nova-*']

The ** is to handle the source_dir, which was /home/mriedem/git/nova/doc/source in my case.

In my setup.cfg, I have:

autodoc_exclude_modules = nova.wsgi.nova-*

So it might be possible to do some translation of that in this code using a **/ prefix, or make them specify the path after source_dir, which would be 'api/nova.wsgi.nova-*' since source_dir is /home/mriedem/git/nova/doc/source.

[Matt Riedemann (mriedem)]

The solution for the sphinx stuff is to configure sphinx using the conf.py in the doc source dir, like in nova that means updating this file:

https://github.com/openstack/nova/blob/master/doc/source/conf.py#L85

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/199609
Committed: https://git.openstack.org/cgit/openstack-dev/pbr/commit/?id=07292c2d8437ca3e10669038d1036f6903e54c80
Submitter: Jenkins
Branch: master

commit 07292c2d8437ca3e10669038d1036f6903e54c80
Author: Matt Riedemann <email address hidden>
Date: Wed Jul 8 07:35:20 2015 -0700

    Add more documentation around building docs

    After digging in pbr and sphinx source code for a day to figure out
    what I was doing wrong, let's update the pbr docs with respect to
    building docs using the autodoc features in pbr.

    Specifically, document the autodoc_tree_excludes and warnerrors
    options and point out that you will probably need to set the
    exclude_patterns options in your sphinx config when excluding
    modules from the docs build and warnerrors=True. Also provide
    some links to the Sphinx docs for more details.

    Related-Bug: #1260495
    Related-Bug: #1472276

    Change-Id: Ib43830d08a156f37a3763ad2807d5c1f4cdbe3fe