[RFE] Transition neutron CLI from python-neutronclient to python-openstackclient

[Richard Theis]

[Existing problem]
The OpenStackClient CLI (python-openstackclient project) has limited neutron support (network, security group, security group rules, floating IP address and fixed IP address CRUD).

[Proposal]
Per OpenStack spec https://review.openstack.org/#/c/243348/, the OpenStackClient CLI should be the recommended CLI for OpenStack projects. Therefore, support provided by the existing neutron CLI should be moved from the python-neutronclient project to the python-openstackclient project.

[Benefits]
A common CLI for OpenStack projects provides users and operators a consistent CLI experience when working with OpenStack cloud environments.

[What is the enhancement?]
This enhancement would result in the neutron CLI moving from the python-neutronclient project to the python-openstackclient project. In addition, users of the python-neutronclient extensibility support would need to transition to the python-openstackclient plugin system. The state of the python-neutronclient project serving as a Python library is TBD given that the python-openstacksdk would be used to implement the neutron CLI support in the python-openstackclient project. Eventually, use of the python-neutronclient project would be deprecated and possibly removed.

I've recorded additional details in the following etherpad: https://etherpad.openstack.org/p/transition_neutronclient_to_openstackclient

[Related information]
- OpenStack spec: Deprecate individual CLIs in favour of OSC ( https://review.openstack.org/#/c/243348/ )
- python-openstackclient spec: https://blueprints.launchpad.net/python-openstackclient/+spec/neutron-client
- Related etherpad: https://etherpad.openstack.org/p/transition_neutronclient_to_openstackclient

[Henry Gessau (gessau)]

We have talked about this for a few cycles now so I acknowledge it is on the wishlist. It is a big undertaking and has so far lacked human resources. There is not much new for the drivers team to discuss about this. If the will and the resources present themselves then some discussions can be opened.

[Armando Migliaccio (armando-migliaccio)]

I am not sure that the openstack client is meant to deprecate the native client, so I would not talk about moving, but rather re-using. The steps involved for making this happens makes me feel this needs a spec. Having said that, if we could able to staff this initiative, it would be a sensible thing to do, at least to try and grasp the bare minimum, like providing CRUD for network, security group, security group rules, and floating IP. That makes me wonder, what's the story as far as the novaclient is concerned? That client does provide those crud operations...

[Richard Theis (rtheis)]

Hi Henry and Armando,

I am willing to work this and it appears that others on the OpenStackClient side are interested too. Some of the foundation work is underway now as noted by recent updates to https://blueprints.launchpad.net/python-openstackclient/+spec/neutron-client.

As for re-using python-neutronclient, folks on the OpenStackClient team told me their goal was to use python-openstacksdk to implement the neutron support in the OpenStackClient instead of using python-neutronclient like is done today for the network CRUD support. I'll ask them to comment here.

As far as python-novaclient, it is used to implement the fixed IP, floating IP, security group and security group rules CRUD within OpenStackClient for both nova network and neutron. However, the transition proposes only using python-novaclient for these when nova network is enabled and switch to python-openstacksdk when neutron is enabled.

I am willing to write a neutron spec for this to lay out the basic transition steps.

Thanks,
Richard

[Akihiro Motoki (amotoki)]

Thanks for moving this forward, Richard.

Supporting neutron API by openstackclient does not mean the deprecation of neutronclient directly.

In addition to the basic transition step, I think the following thing needs to be covered:

* subcommand naming conventions (openstackclient approach easily causes subcommand naming conflict. we need a guideline. do we need some reserved prefix or something?)
* option convention: the current neutronclient supports various types of parameters. we need to assess the current openstackclient convention supports all or we need to expand the convention.
  I recently proposed a patch to neutronclient which tries to describe neutronclient command line option conventions. I don't want to see this kind of discussions in openstackclient implementation. we would like to check osc conventions first.
* what's the future of extra args of neutronclient? Honestly speaking, I don't like it because it is a chaos area and brings a lot of confusions. It is sometimes useful for extensibility but it is too flexible....

I will check the contents you captured in the etherpad page.

[Richard Theis (rtheis)]

Hi Akihiro,

You bring up good points and I think that the OpenStackClient may already address some of them. I'll take a closer look. Maybe a spec could capture the basics of this transition and the future role of python-neutronclient.

Thanks,
Richard

[Richard Theis (rtheis)]

Follow-up on my previous comment...

http://docs.openstack.org/developer/python-openstackclient/commands.html lists the OpenStackClient command objects and actions which provide the current naming convention. This and related documentation serve as a guide for future naming and option convention (e.g. http://docs.openstack.org/developer/python-openstackclient/humaninterfaceguide.html).

While old and now deprecated, https://wiki.openstack.org/wiki/OpenStackClient/Commands#Network provides a starting point for the design for neutron support in OpenStackClient.

Hopefully, these address the general concerns raised by Akihiro. We could possibly address specific concerns via patch sets to these OpenStackClient documents.

[Tang Chen (tangchen)]

Hi guys,

Glad to see this moving on. I'm now working on developing network commands in osc using sdk. And as you can see in here:

- https://blueprints.launchpad.net/python-openstackclient/+spec/neutron-client

I 'd like to do it step by step.

1. The current unit tests is not good enough. I have improved part of compute unit test in osc. And the first step I want to do to move this on is to improve the network unit test framework.

Please refer to:
- https://blueprints.launchpad.net/python-openstackclient/+spec/osc-network-unit-test-refactor
- https://blueprints.launchpad.net/python-openstackclient/+spec/osc-unit-test-framework-improvement

Please do help to review the patches. :)

2. Then the basic commands, create, list, ...... and as you said, using the sdk. Will send some of the WIP patches out for you guys to review.

3. The future plan I think I need to discuss more with you.

Thanks. :)

[Akihiro Motoki (amotoki)]

@Richard,
Thanks for sharing various useful sources. I will check them.
openstackclient has good documentation on various kind of conventions.
It will reduces a chance of mess we have in neutronclient.

@Tang
Good to know that. I am aware you are working on the initial effort of neutron support in OSC.
You and RIchard can collaborate. I am glad to help you all.

If we can implement neutron support in openstackclient as an experimental phase, we can modify inappropriate options or something in fast iteration without technical debts.

[Doug Wiegley (dougwig)]

I need to read the above links, which may answer my question, but is there an extension mechanism for code to live in other repos that extends the client?

[Richard Theis (rtheis)]

@Doug, please refer to the following link for the extension mechanism: http://docs.openstack.org/developer/python-openstackclient/plugins.html

[Armando Migliaccio (armando-migliaccio)]

Richard: thanks for taking the initiative here. Based on the discussion happened during the latest drivers meeting [1], I think it's beneficial to make progress on this front. It'd be good to capture a 'roadmap' that outlines the relationship between the two clients, and what is the plan for superseding the native client with OSC. More details on the meeting's log.

[1] http://eavesdrop.openstack.org/meetings/neutron_drivers/2015/neutron_drivers.2015-12-08-15.03.log.html

[Steve Martinelli (stevemar)]

For keystone, once we had full parity for our v2 commands in OSC, and OSC had a 1.0.0 release, we marked our `keystone` client as deprecated. We have had it deprecated for 2 cycles now, and will be removing it in the N release.

[Richard Theis (rtheis)]

WIP patch set has been created for python-neutronclient devref: https://review.openstack.org/254993

[Richard Theis (rtheis)]

I've removed the WIP tag to open up the patch set for general review.

[Richard Theis (rtheis)]

The python-neutronclient devref has merged: https://review.openstack.org/254993

[Richard Theis (rtheis)]

[Status Update]
See http://docs.openstack.org/developer/python-neutronclient/devref/transition_to_osc.html for a general overview of the transition plan and high-level status. See below for detailed status. This work won't be completed in Mitaka.

NeutronClient version 3.1.1
Total commands: 220
Commands to transition: 195 (excludes LBaaS v1 commands)

OpenStackClient (OSC) version 2.1.0 (to be released soon)
Network commands: 30
Network commands using neutron via Python OpenStack SDK (SDK): 18

Note: Network commands not using the SDK are using python-novaclient.

Significant progress was made on the foundation work necessary (namely OSC using SDK) for adding network commands to OSC. With a significant portion of that work complete, we can continue the transition of the remaining OSC network commands to use the SDK along with adding new commands.

[Richard Theis (rtheis)]

Detailed status is now being tracked at https://etherpad.openstack.org/p/osc-neutron-support

[Richard Theis (rtheis)]

Progress continues with the latest status update available at https://etherpad.openstack.org/p/ocata-neutron-client

[Ankur (ankur-gupta-f)]

https://etherpad.openstack.org/p/osc-transition-priorities

For remaining priorities for Ocata release.

Many are close to being done.

It seems as though many of the Plugins that are going to merged into neutronclient (or other) as OSC plugins do need some work since they do not yet align to OSC standards.

Refactoring of Network SDK for resource2 still maintained at
https://etherpad.openstack.org/p/osc-network-command-sdk-support