Allow ml2 MechanismDrivers to start own rpc listeners

[Sebastian Lohff]

Some MechanismDrivers need their own RPC backend that can be used by an agent to talk to this driver. When running in uwsgi/rpc mode we need to make sure these RPCs listeners are started as part of the neutron-rpc-server. Currently this is only done for service plugins, but MechanismDrivers are part of the Ml2Plugin (which is itself a service plugin).

To allow drivers to start their own RPC listener I propose that the ML2Plugin takes care of this. Ml2Plugin can call MechanismManager.start_driver_rpc_listeners(), which then calls start_rpc_listeners() on each MechanismDriver. The resulting server plugins could then also be returned as part of Ml2Plugin.start_rpc_listeners().

I have a PoC for this running in my dev environment and if we agree on a rough concept would be willing to provide a patch for this.

[Miguel Lavalle (minsel)]

Hi Sebastian,

Thanks for this proposal.

1) What backend do you use? What are the agents involved in your case? Can you give more details on the use case and why this is needed?

2) What is needed to "agree on a rough concept"? Maybe answering question 1 and then proposing the PoC code in Gerrit can get the conversation going.

[Sebastian Lohff (sebageek)]

We have a couple of custom MechanismDrivers for vendor hardware, e.g. for Cisco ACI[0], NSX-T[1] or a custom EVPN fabric with multiple backends[2]. These drivers have their own RPC backend. As we run our cloud on UWSGI + neutron-rpc-server I want to make sure the backends run only in the neutron-rpc-server part. As my drivers are no service plugins there is no proper way for me to do that. We currently misuse get_workers() and start the workers as part of neutron-rpc-server, but I'd like to get away from that.

So to properly start the RPC listeners/backends of our drivers and make sure they only run in neutron-rpc-server I would like to start them as part of Ml2Plugin.start_rpc_listeners().

I can definitely propose the changes (one to neutron-lib and one to neutron). With "agree on a rough concept" I wanted to make sure that we are roughly on the same page and that I didn't miss something obvious. Showing by code what I mean is probably the easiest way to proceed.

[0] https://github.com/sapcc/networking-aci/blob/04d7b29b1106b167c06e701b05c0ca4755001cd6/networking_aci/plugins/ml2/drivers/mech_aci/driver.py
[1] https://github.com/sapcc/networking-nsx-t/blob/a6889f58b769bf30213e7398d2c37932028da4be/networking_nsxv3/plugins/ml2/drivers/nsxv3/driver.py
[2] https://github.com/sapcc/networking-ccloud/blob/acf44a7bc347c27c622cbe2ceff670cd647d52fc/networking_ccloud/ml2/mech_driver.py

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/919589

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/919590