[RFE] Plugin support for API resource attribute extensions

[Boden R]

NOTE:
On 11/28/2017 I removed the old description from this bug and replaced with a simpler more consolidated version based on an example. It seems the old description's verbose nature may cause elongated understanding of the issue.

PROBLEM:
Neutron plugin API extensions that add attributes to existing API resources have no proper way to extend the python-openstackclient (OSC).

EXAMPLE:
a) We have a plugin that extends an existing API resource [1]; in this example we add a 'logging' attribute to the existing 'security_groups' API resource.
b) Now we want to add CLI support for the 'logging' attribute. But there's not clean way with OSC today:
..1) To get the new 'logging' attribute to even show up (parse from API and display on output), we need to update the sdk [2]. Traditionally this has not been something done for "non-stadium projects".
..2) The python-openstackclient (OSC) already defines the logic for 'security_groups'; we want to be able to extend it to handle our 'logging' attribute, but there's no current way [3] to extend an OSC networking command.
..3) As a result we need to duplicate the OSC code [3] for security_groups to add our custom logic [4]. Now we have to try and keep [4] in sync with [3] which is less than optimal.

Some existing plugins that have resource attribute extensions today: [1][5][6][7][8]

MITIGATION:
Today we can still use the python-neutronclient to interface with these types of extensions. This support is provided by the "looseness" of the neutronclient and it's support for arbitrary kwargs.

For example with neutron client you can do something like:

  neutron security-group-create --logging=blah

and the client will pass logging=blah to the API allowing the extension to function properly. In addition the neutronclient will display the value(s) of 'logging' in the CLI output without and code changes to the python-neutronclient.

That said the current mitigation is to use the neutronclient.

CONSIDERATIONS:
- If the longer term goal is to move away from API extensions in neutron (like nova did [9]), then maybe we just support neutronclient until that time.
- While the current support we have for these extensions in neutronclient maybe a "side effect" (e.g. loose kwarg passing and CLI input/output), I know we have existing deployments relying on this support. So removing such support would leave them with no options.
- There maybe some middle ground to get support in OSC in the shorter-term, but in general it's my understanding that the SDK/OSC has few reviewers and maybe undergoing a "restructure" [10].

[1] https://github.com/openstack/vmware-nsx/blob/master/vmware_nsx/extensions/securitygrouplogging.py
[2] https://github.com/openstack/python-openstacksdk/blob/master/openstack/network/v2/security_group.py
[3] https://github.com/openstack/python-openstackclient/blob/master/openstackclient/network/v2/security_group.py#L117
[4] https://review.openstack.org/#/c/393322/4/vmware_nsx/osc/v2/security_group.py
[5] http://git.openstack.org/cgit/openstack/networking-h3c/tree/networking_h3c/extensions/portextensions.py
[6] http://git.openstack.org/cgit/openstack/astara-neutron/tree/astara_neutron/extensions/routerstatus.py
[7] http://git.openstack.org/cgit/openstack/networking-cisco/tree/networking_cisco/plugins/cisco/extensions/routertypeawarescheduler.py
[8] http://git.openstack.org/cgit/openstack/group-based-policy/tree/gbpservice/neutron/extensions/implicit_subnetpools.py
[9] https://blueprints.launchpad.net/nova/+spec/api-no-more-extensions-pike
[10] http://lists.openstack.org/pipermail/openstack-dev/2017-August/120673.html

[Boden R (boden)]

I plan to let this age a little here, and then I'll try to get a slot on the OSC meeting to discuss; say in about 2 weeks time.

Again I'm willing to do this work, but want some agreement that the work can land in some form before investing the time hacking on some code.

There are obviously many ways to solve this problem; I have some ideas, but would prefer to get some input form those who have a history with this code.

Thanks

[Matt Riedemann (mriedem)]

Nova does not support stevedore loading of API extensions anymore:

https://blueprints.launchpad.net/nova/+spec/api-no-more-extensions-pike

So I don't think this bug impacts Nova at all.

[Boden R (boden)]

Matt: thanks for the info on nova's approach to API extensions. Admittedly I'm out of sync with nova, but it's good to understand how other projects are approaching to the topic.

[Boden R (boden)]

In regards to the "gap" herein specifically with neutron support in OSC; resource attribute extensions today have the following problems:
- They must redefine the entry points they want to override. For example [1][2]. While this works sometimes, there's no guarantees as to which entry point will be loaded first/last.
- They must duplicate the "base action" code since there's no way to extend them in base OSC. For example [3]. Ideally extensions can reuse OSCs functionality only providing the difference they need. Without such constructs the code in the client extension can get out of sync with the base OSC (since it duplicates code).
- There's still no way to add attributes to collect/display in the SDK resource. Thus even if the above things are done, the attribute(s) are not collected/shown in the OSC CLI output.

[1] https://github.com/openstack/vmware-nsx/blob/master/setup.cfg#L69
[2] https://github.com/openstack/python-openstackclient/blob/master/setup.cfg#L456
[3] https://github.com/openstack/vmware-nsx/blob/master/vmware_nsx/osc/v2/security_group.py#L80

[Boden R (boden)]

Related ML discussions:

http://lists.openstack.org/pipermail/openstack-dev/2017-August/120613.html

http://lists.openstack.org/pipermail/openstack-dev/2017-August/120618.html

[Brian Curtin (brian.curtin)]

I don't really get what you're asking here with regards to SDK. If you want to add things to SDK, just add them. Turning it into a mess of feature plugins would go the opposite direction of one of the reasons the project exists in the first place.

There does exist some minimal plugin functionality, though it's intended to be for cloud providers to expose platform specifics like authentication.

[Boden R (boden)]

Brian: please see #b under the problem description in the defect text.
Non-stadium projects are not supposed to be adding "custom" logic to OSC/SDK/etc. so therein lies one gap.

[Brian Curtin (brian.curtin)]

We are 100% not adding extensions to openstacksdk. If a service offers some API and it wants to be exposed via openstacksdk, add it to openstacksdk. The end.

[Ihar Hrachyshka (ihar-hrachyshka)]

I think what @boden tries to get is affirmation that gaps between neutronclient and OSC can be solved by adding implementation for missing Neutron APIs (including those that add new attributes to existing resources like port or network) to OSC itself.

The ML thread where @boden tried to collect feedback seemed to indicate that some project participants are not willing to open doors of OSC for new Neutron APIs (that are implemented with new extensions, since Neutron doesn't have a notion of API microversioning).

Is OSC team ready to allow new attributes added to existing Neutron resources as needed?

[Armando Migliaccio (armando-migliaccio)]

Let's brainstorm a little on this one during one of the drivers meetings.

[YAMAMOTO Takashi (yamamoto)]

just a data point:

networking-midonet has an extension [1], which currently relies on neutronclient's "extra arguments" feature. [2]
in this particular case, it doesn't need to be "extra arguments" though.
ie. there's no problem to implement it explicitly in osc. (if it's ok for osc folks)

[1] https://docs.openstack.org/networking-midonet/latest/reference/specs/mitaka/bgp-speaker-router-insertion.html#other-end-user-impact
[2] https://docs.openstack.org/python-neutronclient/latest/cli/neutron.html#cli-extra-arguments

[Boden R (boden)]

Not sure I understand comment #11.
Based on what I see referenced there, the neutron client is being used; not OSC.
The support for "extra args" in the neutron client is one of the features that allows the extensions to work with neutron client, but not OSC (OSC doesn't support extra args as far as I know).

[YAMAMOTO Takashi (yamamoto)]

Boden,

I just gave an example of existing extensions.

[Akihiro Motoki (amotoki)]

Sorry for a long post.

I think there are several points in this discussion.
- (a) community-wide API direction,
- (b) neutron API future, and
- (c) CLI support for the CURRENT neutron API (with extension support)

Note that I use "extension" as the context of API extensions by third-party plugins/drivers.
APIs defined in neutron-lib are not considered as "API extensions" and they are "core" in this context.

Regarding (a), "inter-operability" across OpenStack clouds is an important aspect. This leads to a question on whether OpenStack will accept API extensions in future as a whole. In my understanding, the Foundation, TC, and API-SIG are suggesting not to use API extensions. Neutron is alone now because we are behind other projects (it's my fault in most parts....). I am not sure we continue the current approach. Ideally all API extensions provided by vendors should be part of the neutron official APIs. I think most of the feedback from the ML discussion are based on this direction (+ openstacksdk/shade development policies).

Regrading (b), IMHO the long-term goal is same as (a), but in the short/middle term we must take into account the current situation where we allow "API extensions" by vendors. Nova took long time to drop API extensions (as Matt mentioned in comment #2). We should assume the similar period (two or three years?) even if we drop API extensions).

Regarding (c), I think this is the main point that Boden would like to discuss.
There seems two ways: (c1) to convince OSC folks to accept third-party extension, or (c2) to keep
"neutron" CLI.

Even if we go to the route of (c1), we need to honor the middle/long term direction of OpenStack API. I cannot say much though I am not a fan of vendor extensions. On this route, we cannot determine the future only in the neutron team.

If we go to the router of (c2), the next question is which will be the priority for us, OSC vs "neutron" CLI. IMHO OSC is the long-term direction. If so, new features are supported by OSC and some volunteers need to catch up with them in 'neutron' CLI.

Ideally all vendor API extensions are upstreamed defined in neutron-lib, but this is perhaps not acceptable though.

From the implementation perspective, the extra argument mechanism is completely a mistake (at least to me). This requires us a super-loose argument parsing. "neutron" CLI completely depends on the order of arguments and we cannot provide an interface which is user-friendly and understood easily. We received a number of bugs on "neutron" CLI behaves in a confusing way if only the order of some arguments are changed. As a result, we have this kind of ugly argument definitions [1]. If we have some support of vendor API extension in OSC, the argument parsing must be more strict not to repeat the technical debt in neutron CLI.

# Personally I am tired of maintaining the extra argument mechanism. If we continue to keep it, some volunteer to maintain it other than me.

[1] http://git.openstack.org/cgit/openstack/python-neutronclient/tree/neutronclient/neutron/v2_0/subnet.py#n95

[Boden R (boden)]

Akihiro thanks for the insight.

I'm aware of the longer term goal of transitioning away from API extensions, but as stated that's a long ways out given the amount of such we have today and the work required to "unify". In the meantime I'm fine with just continuing to support the neutronclient for this reason, but again that means we are stuck with neutronclient for the meantime.

In regards to the "extra args" in the neutronclient, I totally understand your perspective on their "looseness", but at the same time we have a lot of extensions/users relying on that support. If were to stop supporting such args, we'll leave a number of customers without a supported way to interface with the respective extensions from a CLI perspective. With that in mind I really don't see how we can drop the support for extra args without another solution.

[Akihiro Motoki (amotoki)]

The discussion in the drivers meeting http://eavesdrop.openstack.org/meetings/neutron_drivers/2017/neutron_drivers.2017-11-16-22.00.log.html#l-86

[Boden R (boden)]

RE comment #16 and the neutron drivers meeting.

- I don't feel the issue herein was fully understood in that discussion. What we are talking about herein is related to the CLI and the fact that we have no way to add support for resource attribute extensions to the OSC. I'm not really understand why the discussion in the drivers meeting is talking about API definitions in neutron-lib; how does that help the CLI issue herein?
- W/R/T to support the extra args; we've had this for years. We can't just up and drop support for it.
- Going back to comment #14... If the long term direction is to transition away from API extensions in neutron, then as a result I suspect the issue herein will go away; since there won't be extensions (the root of this issue IMO). That said, I think we need to understand when that will happen and if we can keep the neutronclient until that time. Otherwise we have to invest in getting OSC to work with these extensions (the issue herein) and that work will get "thrown away (per say)" when we unify the API.

That said, I will try to follow up with the drivers team to re-tee this topic; again I'm not sure the full context was understood in that meeting. Nor do I think the full impact of removing the extra args is understood.

[Akihiro Motoki (amotoki)]

(Although I could not join the drivers meeting Nov 16) I see several points:

* The neutron team considers API definitions in neutron-lib as the official APIs (regardless of they are called "extensions")
* Why can't vendor API extensions be upstreamed into neutron-lib? If it can, the situation would be simpler.

This hits me another question: Do vendor API extensions live forever? We will hit the same problem in the API side if we go to the route of disallowing API extensions in future (as most OpenStack APIs do and are going to do). I think we need to take into account this. We need to assume this will happen sooner (two cycles?) or later (more that two years?). What will vendor API extensions want to do then?

If vendor API extensions cannot be merged into neutron API, it means we cannot drop extensions forever. I think this is the same discussion of CLI extra argument.

[Miguel Lavalle (minsel)]

On November 30th we decided to continue support of neutron python client so we can support API resource attribute extension

[Akihiro Motoki (amotoki)]

The meeting log is http://eavesdrop.openstack.org/meetings/neutron_drivers/2017/neutron_drivers.2017-11-30-22.00.log.html#l-18

I am trying to compile the discussion more.