Add bash-completion action to support documentation

[Stephen Gordon]

The "traditional" CLIs support a bash-completion argument which is used to build bash-completion support for the CLI and is also used to generate the Command Line Interface reference on docs.openstack.org.

The unified client should provide such an action to support these activities.

[Terry Howe (thowe-g)]

I'm very confused here, do you mean the unified CLI should support an action to generate bash-complete (which it does already) and generate reference docs? Generating reference docs from the CLI seems a bit odd to me.

[Stephen Gordon (sgordon)]

The docs team has a utility that uses the bash-completion to generate the CLI documentation - no extra changes are required in the CLI *if* it exposes a bash-completion action. The script in question is here:

http://git.openstack.org/cgit/openstack/openstack-doc-tools/tree/os_doc_tools/commands.py

I'm in the process of changing it to accept/generate docs for the unified CLI, but it does not appear to have the required action:

$ python commands.py openstack
OpenStack Auto Documenting of Commands (using openstack-doc-tools version 0.8.2)

Documenting 'openstack'
Documenting 'openstack help'
Documenting 'openstack' subcommands...
ERROR: cliff.app Unknown command ['bash-completion']
Traceback (most recent call last):
  File "commands.py", line 528, in <module>
    sys.exit(main())
  File "commands.py", line 524, in main
    document_single_project(prog_args.client)
  File "commands.py", line 491, in document_single_project
    subcommands)
  File "commands.py", line 409, in generate_subcommands
    "bash-completion"]).strip().split()
  File "commands.py", line 40, in check_output
    raise subprocess.CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['openstack', 'bash-completion']' returned non-zero exit status 2
$ openstack bash-completion
ERROR: cliff.app Unknown command ['bash-completion']

If I run one of the traditional clients with the bash-completion argument (i.e. nova) then it spits out the expected output listing arguments etc.

Installed python-openstackclient from pip is:

$ pip show python-openstackclient
---
Name: python-openstackclient
Version: 0.3.0
Location: /usr/lib/python2.7/site-packages
Requires: pbr, cliff, keyring, pycrypto, python-glanceclient, python-keystoneclient, python-novaclient, python-cinderclient, requests, six

[Stephen Gordon (sgordon)]

Interestingly I see that the unified client does list bash-complete actions, e.g. from openstack compute --help:

complete print bash completion command

But they don't actually appear to be implemented?:

$ openstack compute complete
ERROR: cliff.app Unknown command ['compute', 'complete']

[Stephen Gordon (sgordon)]

Ok, looks like it's meant to be just "openstack complete". So the two things I notice are:

1) Unlike the existing clients the action is "complete", not "bash-complete".

2) Unlike the existing clients you need a "real" environment even to get the bash completion output:

$ openstack complete
ERROR: cliff.app You must provide a username via either --os-username or env[OS_USERNAME]

[Stephen Gordon (sgordon)]

s/bash-complete/bash-completion/ in comment # 4, also adding #3 that the output I get is the actual script:

$ openstack complete
INFO: urllib3.connectionpool Starting new HTTP connection (1): 10.15.24.141
_openstack()
{
  local cur prev words
  COMPREPLY=()
  _get_comp_words_by_ref -n : cur prev words

  # Command data:
...

Whereas from the traditional clients I get:

$ nova bash-completion
restore --boot-volume --port help secgroup-delete-rule --priority --ipv6 --file --image-with --instance-name suspend --action...

[Terry Howe (thowe-g)]

The complete command is part of cliff actually and there is a "none" format that produces output kind of like:

$ os complete --shell none
  cmds_volume_type_create='-h --help -f --format -c --column --variable --prefix --property'
  cmds_volume_type_delete='-h --help'
  cmds_volume_type_list='-h --help -f --format -c --column --quote --long'
  cmds_volume_type_set='-h --help --property'

Maybe some RE substitutions on that?

[Steve Martinelli (stevemar)]

A valid point is raised about the current requirement of a real environment. We should look into seeing if it's possible (likely is) to bypass that (we do so with the help command).

[Terry Howe (thowe-g)]

I had it as non-authorize at first, but I think it may have been Dean that argued that you can't really tell what commands you have until you authorize. Either an option not to authorize or add auth_optional to go along with auth_required flag there is now.

[Doug Hellmann (doug-hellmann)]

All of the commands and their arguments are discoverable programmatically now, so we could even write something to access the data directly instead of parsing the output of the completion command.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/78758

[Terry Howe (thowe-g)]

It would definitely be possible for the docs to extract a list of commands from the complete command, using the "--shell none" option would definitely help. It could also be done from the help:

    os --help | sed -e 's/ //' | sed -e 's/ .*//' |
    while read LINE;
    do
      if [ "${LINE}" == "Commands:" ]; then START=true; continue; fi;
      if $START; then echo "${LINE}"; fi;
    done

Using bash complete command a little easier:

    os complete --shell none | sed -e 's/=.*//' -e 's/ cmds_//' -e 's/_/ /g' | grep -v ' cmds'

That would work for the neutron client as well as it uses cliff.

[Steve Martinelli (stevemar)]

One issue I noticed with the output generated from your suggestions Terry, the end result has an extra single word command which won't work:

i.e. `user` in the following output:

 user
 user create
 user delete
 user list
 user set
 user show

[Terry Howe (thowe-g)]

Good point Steve. I find this whole thing odd where there is some program to generate documentation that generates HTML rather than some sort of markdown like restructured text. Why isn't something like sphinx-argparse being used?

[Andreas Jaeger (jaegerandi)]

The tool generates docbook that we then use to generate PDF and HTML. There's no support for docbook in sphinx-argparse.

[Stephen Gordon (sgordon)]

The script generates DocBook XML, this is the primary input format for the tooling that builds docs.openstack.org, this approach works when generating CLI documentation for all of the existing clients. Sphinx does not support DocBook XML as an output format.

[Doug Hellmann (doug-hellmann)]

I wonder if it would make more sense to write something that used cliff's discovery capabilities more directly, instead of generating text in one format and then converting it to docbook.

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/78758
Committed: https://git.openstack.org/cgit/openstack/python-openstackclient/commit/?id=211cd31d7ac2c7768cc871ac7c9228f7713b8dfc
Submitter: Jenkins
Branch: master

commit 211cd31d7ac2c7768cc871ac7c9228f7713b8dfc
Author: Terry Howe <email address hidden>
Date: Thu Mar 6 13:56:10 2014 -0700

    Make bash comple command best effort to authorize

    If authorization fails for the complete command, generate the bash complete anyway.
    * Added best_effort flag to command
    * Attempts to authorize for bash complete, but if it fails, it tries anyway

    Change-Id: I796258f8044f42abc6a51164d920a26f73397962
    Partial-Bug: #1283550