Provide structured API documentation generated from docstrings

[PyDeq]

Running Sphinx [1] in the nitpicky [2] mode with the intersphinx [3] extension allows one to check whether references in their documentation is correct. E.g. it checks whether the class named `bs4.BeautifulSoup` in this example::

    .. function:: process(document)

       Process an HTML document.

       :param document: the HTML document
       :type document: bs4.BeautifulSoup

really exists. It works by pointing [4] the extension to an `objects.inv` file of the given library. I don't know the format very well but I guess that it contains at least the list of all the documented identifiers/names of the given library. The file is automatically generated by Sphinx [3].

Since BeautifulSoup's documentation doesn't use directives like `.. class::` to document its classes, its `objects.inv` file does not contain references to its classes etc. Would it be possible to use these directives to automatically populate the `objects.inv` file so that other developers can use it for the reference check as described above?

Here is a minimal reproducer:

1. Create a file `test/contents.rst`::

    .. function:: process(document)

       Process an HTML document.

       :param document: the HTML document
       :type document: bs4.BeautifulSoup

2. Create a file `test/conf.py`::

    extensions = ["sphinx.ext.intersphinx"]
    intersphinx_mapping = {"bs4": ("http://www.crummy.com/software/BeautifulSoup/bs4/doc/", None)}

3. Run `sphinx-build -n test test`

So far, you can see this warning in the output: "contents.rst:6: WARNING: py:obj reference target not found: bs4.BeautifulSoup"

[1] http://sphinx-doc.org/
[2] http://sphinx-doc.org/config.html#confval-nitpicky
[3] http://sphinx-doc.org/ext/intersphinx.html
[4] http://sphinx-doc.org/ext/intersphinx.html?highlight=intersphinx#confval-intersphinx_mapping

[Adrián Chaves (gallaecio)]

objects.inv exists, what does not exist is the API reference documentation.

[Leonard Richardson (leonardr)]

I took some time to look into this, and Adrián has summarized the problem well. The use of intersphinx to check references implies the existence of targets for any reference you might make--basically, API reference documentation--and that's not how the Beautiful Soup documentation is written. Sphinx syntax does not allow me to just drop in targets where appropriate. These targets become section headings rendered with special formatting, and adding them requires changes to the text and structure of the documentation.

I was able to rewrite the documentation such that it contains intersphinx targets for the classes most likely to be mentioned in other projects' documentation. It reads a little bit awkward, but it might be worth using in the short term. In the long term, the documentation needs to be rearchitected with the two parts that are now traditional for open source software projects: a narrative guide and a structured set of API documentation automatically generated from docstrings.

[Leonard Richardson (leonardr)]

After putting out the 4.12.0 release I updated the documentation to include the intersphinx targets. I tested it out with a test document that looks like this:

Intersphinx test documentation for Beautiful Soup
=================================================

* A class: :py:class:`bs4.BeautifulSoup`
* Another class: :py:class:`bs4.Tag`
* An attribute: :py:attr:`bs4.Tag.name`

In conf.py for the test project, my intersphinx_mapping looks like this:

intersphinx_mapping = {
    "beautifulsoup4": (
        "https://www.crummy.com/software/BeautifulSoup/bs4/doc/", None
    )
}

[Leonard Richardson (leonardr)]

Leaving this issue open pending a better long-term solution.