28 warnings when building docs

[aeva black]

When building documentation with "python setup.py build_sphinx", the output shows:

  done
  writing...
  build succeeded, 28 warnings.

The warnings are for such things as
  WARNING: document isn't included in any toctree
  WARNING: Definition list ends without a blank line; unexpected unindent.
  WARNING: toctree contains reference to nonexisting document
.. and so on.

We should clean up all these warnings, and then enable a jenkins job check to enforce that new patches do not introduce doc build warnings.

[Éric Araujo (merwok)]

I’d like to work on this.

[Éric Araujo (merwok)]

I fixed some of the easy warnings and will continue. I also ran rstlint from the CPython repository to find other style issues.

I also had a look at the errors emitted during the doc build:

The autodoc warnings (for ironic.db.sqlalchemy.alembic and ironic.nova.compute.manager) seem related to the use of packaged found in duplicate (installed as proper dependencies and vendored into ironic).

There are many “DuplicateOptError: duplicate option: use-syslog” from oslo.config; maybe they’re caused by both ironic and nova configuring the logging system.

[Éric Araujo (merwok)]

Question: many warnings are caused by incorrect whitespace in docstrings. For those vendored modules under ironic.common, should I fix it there or submit patches to the upstream neutron/nova/etc projects?

If there is documentation explaining how the copy-paste code sharing process works, I’d like to read it :)

[Éric Araujo (merwok)]

One of the warnings in autoindex.rst come from a bug in pbr 0.8.0 (there is duplicate code in ironic.openstack.common.setup that does not have the bug, but it is not used by build_sphinx). I guess the right process to have it fixed there and update ironic’s dep on pbr?

I really should have found time to go to the PyCon sprint to figure all that out :)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/125273

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/125273
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=56fed1198fd42953f827521748c6be475d9f4fa4
Submitter: Jenkins
Branch: master

commit 56fed1198fd42953f827521748c6be475d9f4fa4
Author: Mark Atwood <email address hidden>
Date: Wed Oct 1 06:58:10 2014 +0000

    Cleans up some Sphinx rST warnings in Ironic

    The names of parameters in docstrings should not have leading astrisk
    characters, because it confuses the rST parser.

    Change-Id: I8c3574b03903a23c10ef2afcb793c6766a486007
    Partial-Bug: 1277282

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/127034

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/127034
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=91fde155be16500c3c2044abe5f44bbfae2c4857
Submitter: Jenkins
Branch: master

commit 91fde155be16500c3c2044abe5f44bbfae2c4857
Author: Vladyslav Drok <email address hidden>
Date: Thu Oct 9 00:45:22 2014 +0300

    Fix markup-related issues in documentation

    This change fixes all markup mistakes in docstrings that lead to
    warnings and errors during docs building.

    Partial-bug: #1277282
    Change-Id: I092a684b7587b802427d57f4292398bfd21deeb1

[Dmitry Tantsur (divius)]

Is someone working on this? Could you update status accordingly?

[Dmitry Tantsur (divius)]

I still see:

python setup.py build_sphinx 2>&1 | grep -i warning
/home/dtantsur/Projects/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.env.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.env'; the following exception was raised:
/home/dtantsur/Projects/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.versions.21b331f883ef_add_provision_updated_at.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.versions.21b331f883ef_add_provision_updated_at'; the following exception was raised:
/home/dtantsur/Projects/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.versions.2581ebaf0cb2_initial_migration.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.versions.2581ebaf0cb2_initial_migration'; the following exception was raised:
/home/dtantsur/Projects/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.versions.31baaf680d2b_add_node_instance_info.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.versions.31baaf680d2b_add_node_instance_info'; the following exception was raised:
/home/dtantsur/Projects/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.versions.3bea56f25597_add_unique_constraint_to_instance_uuid.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.versions.3bea56f25597_add_unique_constraint_to_instance_uuid'; the following exception was raised:
/home/dtantsur/Projects/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.versions.3cb628139ea4_nodes_add_console_enabled.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.versions.3cb628139ea4_nodes_add_console_enabled'; the following exception was raised:
/home/dtantsur/Projects/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.versions.487deb87cc9d_add_conductor_affinity_and_online.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.versions.487deb87cc9d_add_conductor_affinity_and_online'; the following exception was raised:
/home/dtantsur/Projects/ironic/doc/source/api/ironic.nova.compute.manager.rst:4: WARNING: autodoc: failed to import module u'ironic.nova.compute.manager'; the following exception was raised:
/home/dtantsur/Projects/ironic/doc/source/api/ironic.nova.scheduler.ironic_host_manager.rst:4: WARNING: autodoc: failed to import module u'ironic.nova.scheduler.ironic_host_manager'; the following exception was raised:
/home/dtantsur/Projects/ironic/doc/source/api/ironic.nova.virt.ironic.driver.rst:4: WARNING: autodoc: failed to import module u'ironic.nova.virt.ironic.driver'; the following exception was raised:
/home/dtantsur/Projects/ironic/doc/source/dev/conductor.rst:7: WARNING: toctree contains reference to nonexisting document u'api/ironic.conductor.resource_manager'
checking consistency... /home/dtantsur/Projects/ironic/doc/source/api/autoindex.rst:: WARNING: document isn't included in any toctree
/home/dtantsur/Projects/ironic/doc/source/dev/api.rst:: WARNING: document isn't included in any toctree
/home/dtantsur/Projects/ironic/doc/source/dev/conductor.rst:: WARNING: document isn't included in any toctree
build succeeded, 14 warnings.
/home/dtantsur/Projects/ironic/d...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/139965

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/139965
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=9efc942e51c24b62d3377e5243fbfc37c189941d
Submitter: Jenkins
Branch: master

commit 9efc942e51c24b62d3377e5243fbfc37c189941d
Author: Satoru Moriya <email address hidden>
Date: Mon Dec 8 15:40:08 2014 +0900

    Use Literal Blocks to write code sample in docstring

    In python docstring, one should use Literal Blocks to write sample
    code otherwise one gets build warnings and badly formatted documents.

    Also, because sphinx treats the line which starts ".." as a comment,
    this patch moves "..." to the end of the previous line.

    Change-Id: I0bbd61d43daaffe9de579b1a6b50ea6826bdfd50
    Partial-bug: 1277282

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/145699

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/145699
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=afde5bf1f8f589138c0186e420d15687b1e67132
Submitter: Jenkins
Branch: master

commit afde5bf1f8f589138c0186e420d15687b1e67132
Author: Satoru Moriya <email address hidden>
Date: Thu Jan 8 14:01:54 2015 +0900

    Delete unnecessary document files

    This patch deletes unnecessary files and references to them
    because they are just TOC files and also cause some warnings
    in doc building.

    Change-Id: I3bd230cc3b38a6d7a8d9d9b8c230eb34b90dd936
    Partial-bug: 1277282

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/147043

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/147043
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=43355e180f164ab7917bf1a05ce60d099f2b45aa
Submitter: Jenkins
Branch: master

commit 43355e180f164ab7917bf1a05ce60d099f2b45aa
Author: Satoru Moriya <email address hidden>
Date: Tue Jan 13 19:03:59 2015 +0900

    Remove links autogenerated from module names

    Having links autogenerated from module names in the developer
    documentation is prone to bit rot as modules get renamed or moved
    around.

    One should use module index to get the same kind of information.

    Change-Id: I499343ba669893736324116b44df01c69391fff1
    Partial-bug: 1277282

[John Stafford (john-stafford)]

Is this bug still being worked on?

[Mario Villaplana (mario-villaplana-j)]

There are still 16 warnings. I'll take a look.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/180339

[Mario Villaplana (mario-villaplana-j)]

I got rid of a lot of the "nova.compute" import warnings through using a feature in a new version of Sphinx that allows you to mock dependencies. Also fixed a few random warnings that were introduced. Those fixes are in the change in the last comment.

It looks like the ironic.db.sqlalchemy.alembic.env imports are still throwing some ImportErrors. This is probably due to dependencies that aren't in the system path at doc build runtime. I'm able to mock out the "alembic.<something>" imports, but I don't want to mock out all the versioned DB migration files because that list would grow long, and we'd have to change it with each DB migration.

Two potential solutions would be mocking based on wildcards, which I'm not sure sphinx supports, or adding the imports to the system path. I'll try these solutions tomorrow.

Below is the current warning output with the above change, for reference:

Running Sphinx v1.3.1
Initializing sphinxcontrib.pecanwsme.rest
loading pickled environment... done
Using openstack theme from /opt/stack/ironic/.tox/venv/local/lib/python2.7/site-packages/oslosphinx/theme
building [mo]: all of 0 po files
building [man]: all source files
updating environment: 0 added, 14 changed, 0 removed
reading sources... [100%] api/ironic.db.sqlalchemy.alembic.versions.bb59b63f55a_add_node_driver_internal_info
/opt/stack/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.env.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.env'; the following exception was raised:
Traceback (most recent call last):
  File "/opt/stack/ironic/.tox/venv/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 385, in import_object
    __import__(self.modname)
ImportError: No module named alembic.env
/opt/stack/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.versions.1e1d5ace7dc6_add_inspection_started_at_and_.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.versions.1e1d5ace7dc6_add_inspection_started_at_and_'; the following exception was raised:
Traceback (most recent call last):
  File "/opt/stack/ironic/.tox/venv/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 385, in import_object
    __import__(self.modname)
ImportError: No module named alembic.versions.1e1d5ace7dc6_add_inspection_started_at_and_
/opt/stack/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.versions.21b331f883ef_add_provision_updated_at.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.versions.21b331f883ef_add_provision_updated_at'; the following exception was raised:
Traceback (most recent call last):
  File "/opt/stack/ironic/.tox/venv/local/lib/python2.7/site-packages/sphinx/ext/autodoc.py", line 385, in import_object
    __import__(self.modname)
ImportError: No module named alembic.versions.21b331f883ef_add_provision_updated_at
/opt/stack/ironic/doc/source/api/ironic.db.sqlalchemy.alembic.versions.242cc6a923b3_add_node_maintenance_reason.rst:4: WARNING: autodoc: failed to import module u'ironic.db.sqlalchemy.alembic.versions.2...

[Mario Villaplana (mario-villaplana-j)]

I ended up taking out the mocking because the new sphinx lib was breaking man pages for other products, so the global requirement restrictions wouldn't let us upgrade it to anything >= 1.3. I think the mock strategy was avoiding the underlying issue, anyway. The change I made now just fixes some easy whitespace and indentation issues.

I noticed that both ImportErrors (alembic... and nova...) occur in files where the import name begins with the name of the directory the file that's doing the importing is in. For example, the alembic error occurs in the directory "alembic" and the nova error occurs in the directory "nova".

These imports probably refer to absolute imports, so I tried to force Sphinx to read these as absolute imports by adding "from __future__ import absolute_import" to the top of each file. The imports were still reported as missing, however.

To be honest, I'm not sure how nova/manager.py even imports nova at runtime, since I don't see it in requirements.txt. I was surprised that the __future__ absolute import didn't fix the alembic import, since alembic is in requirements.txt.

Passing this off to someone else for now. I hope these comments are helpful.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/180339
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=0818ffd3b09373427195a7cfa85b8a903c978033
Submitter: Jenkins
Branch: master

commit 0818ffd3b09373427195a7cfa85b8a903c978033
Author: Mario Villaplana <email address hidden>
Date: Tue May 5 21:57:08 2015 +0000

    Fixes some docstring warnings

    Some warnings related to whitespace, improper indentation,
    and improper use of literals are fixed.

    Change-Id: I86b245e44a48518ca077b4b84a09e7d72d372702
    Partial-Bug: 1277282

[Mario Villaplana (mario-villaplana-j)]

Returning this since I haven't worked on it in a while.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/210575

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/210575
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=1d82c93652b2db512a1e0fe70156e6be8c6b0d7a
Submitter: Jenkins
Branch: master

commit 1d82c93652b2db512a1e0fe70156e6be8c6b0d7a
Author: stephane <email address hidden>
Date: Fri Aug 7 10:27:49 2015 -0700

    Fix warnings on doc builds

    In the sphinx config, mock out the nova imports we do for the
    compute manager so the docs will build with fewer errors.

    Exclude the alembic migration environment from autodoc.

    Fix the remaining markup errors, which are now easier to spot
    given the other fixes.

    Note that this leaves one remaining warning, which is that the
    generated API documentation isn't currently included in a
    toctree. This is easy to fix but it would be good to discuss
    whether/where to include it in the documentation hierarchy.

    Closes-Bug: 1277282
    Change-Id: Iee5fc3eeb95a09198d5234d0ea05e150ab26f16b