neutron

enable vhost-user support with neutron ovs agent

Bug #1506127 reported by sean mooney on 2015-10-14

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Wishlist	Terry Wilson	neutron mitaka-1 "m1"

Bug Description

In the kilo cycle vhost-user support was added to nova and supported out of tree
via the networking-ovs-dpdk ml2 driver and l2 agent on stack forge.

in liberty agent modification were up streamed to enable the standard
neutron openvswitch agent to manage the netdev datapath.

in mitika it is desirable to remove all dependence on the networking-ovs-dpdk repo
and enable the standard ovs ml2 driver to support vhost-user on enabled vswitches.

to enable vhost-user support the following changes are proposed to the neutron openvswitch agent and ml2 mechanism driver.

AGENT CHANGES:

To determine if a vswitch supports vhost user interface two pieces of information are required
the bridge datapath_type and the list of supported interfaces form the ovsdb.
the datapath_type feild is require to ensure the node is configured to used the dpdk enabled netdev datapath.

the supported interfaces types field in the ovsdb contains a list of all supported interface types for all supported datapath_types.
if the ovs-vswitchd process has been compiled with supported for dpdk interface but not started with dpdk enabled , dpdk interfaces will be omitted from this list.

the ovs neutron agent will be extended to query supported interfaces parameter in the ovsdb and append it to the configuration section of the agent state report. the ovs neutron agent will be extended to append the configured datapath_type to the configuration section of the agent state report. The OVS lib will be extended to retrieve the supported interfaces from the ovsdb.

ML2 DRIVER CHANGES:

the ovs ml2 agent will be extended to consult the agent configuration when selecting the vif type and vif binding details to install.

if the datapath is netdev and the supported interface types contains vhost-user it will be enabled. in all other cases it will fall back to the current behavior. this mechanism will allow easy extension of the ovs neutron agent to support other ovs interfaces type in the future if enabled in nova.

Tags:

sean mooney (sean-k-mooney) on 2015-10-14

Changed in neutron:
assignee:	nobody → sean mooney (sean-k-mooney)

Revision history for this message

Terry Wilson (otherwiseguy) wrote on 2015-10-15:

Some comments:

> To determine if a vswitch supports vhost user interface two pieces of information are required the bridge datapath_type and the list of supported interfaces form the ovsdb. the datapath_type feild is require to ensure the node is configured to used the dpdk enabled netdev datapath.

Since a config option to say "we want to use DPDK on this host" is required no matter what (in the above, it's setting datapath_type=netdev), that makes the autodetection basically just a sanity check to make sure that the host supports it. Also, datapth_type=netdev does not necessarily mean DPDK, so I'd prefer not to overload that term.

> ML2 DRIVER CHANGES

Storing the datapath and supported interfaces in the agent state and then querying it in the mech driver leads to having to store the logic in the mech driver and update it any time a vif-details changing feature is added. You could end up with complex chains of if/else statements based on what combinations of options were passed. It seems much cleaner to me to either define a separate class for each agent 'driver' that stores its own datapath type and can compute its own vif details based on the port context and to pass that via the agent state configuration. That way, there is a single config option on the agent, and the mech driver doesn't have to be modified when adding a new feature that changes the vif details.

An example of the above is here: https://review.openstack.org/#/c/234517/

It subclasses the the agent, but doesn't necessarily need to. There could be an abstract base class OVSPluginDetails that returns vif type and vif details based on a port context and stores the datapath type.

OpenStack Infra (hudson-openstack) on 2015-10-15

Changed in neutron:
assignee:	sean mooney (sean-k-mooney) → Terry Wilson (otherwiseguy)
status:	New → In Progress

Revision history for this message

sean mooney (sean-k-mooney) wrote on 2015-10-16:

Download full text (4.3 KiB)

the netdev datapath is as you said not related to dpdk support in the vswitch.
it exist with out without linking ovs against the dpdk libary.

the primary detection mechanism i am proposing is to check the supported interfaces feild in the Open_vSwitch tables in the ovsdb
the datapath comparison is a sanity check to ensure that kernel datapath is not being used.

sudo ovs-vsctl list Open_vSwitch
_uuid : b8761c30-5845-4bbd-885b-ee1d2e026568
bridges : [1003bd42-5e7f-4867-af8f-b927789610b7, 3fe690a5-b197-494c-8edf-dcba7592838a, 4fab8831-cbb7-492f-8025-13bde85abccc]
cur_cfg : 20
datapath_types : [netdev, system]
db_version : []
external_ids : {}
iface_types : [dpdk, dpdkr, dpdkvhostuser, geneve, gre, internal, ipsec_gre, lisp, patch, stt, system, tap, vxlan]
manager_options : []
next_cfg : 20
other_config : {pmd-cpu-mask="4000000004"}
ovs_version : []
ssl : []
statistics : {}
system_type : []
system_version : []

my main dislike for having an explicit configuration option to enable dpdk support is
that i have prviously proposed adding such an option during the juno cycle at both atlanta and paris summits.
it was reject by the neutron core team with the request that we develop a runtime detection mechanism instead.

based on the discussions at the paris summit i reached out to the openvswitch mailing list and proposed introducing the
iface_types field above. when they were receptive to the enhancement i worked with our vswitch and odl teams to submit patches which have now been accepted and shipped in the latest release of both projects.

the reason i bring up odl is that one of the things i would like to do is simplify the configuration of ovs with dpdk regardless of the
ml2 plugin that is manage it. in the openvswitch neutron agent case a config value could be used but that is not possible if
ovs is managed by an openflow controller such as odl. the following review https://review.openstack.org/#/c/215612/ (review.openstack.org is down so cant double the link) enables vhost-user support with the networking-odl based on the same dection mechanisium of checking the supported interface and checking datapath_type as a sanity check.

ideally i would like to use the same mechanism for the ovs neutron agent, odl and the ofagent to reduce the variation between deployment choices and reduce documentation/operator burden.

having access to the iface_type feild has other advantages also . if dpdkvhostcuse or dpdkr(ivshmem) support is added to nova in the future it would be easy to add support to the ovs ml2 driver.

if the iface_types were exposed via a rest api they could be used for network capability based scheduling.
for example in the snippit above you can see that the ovs on my system supports the following tunneling protocals(geneve,gre,stt,vxlan and lisp). in future when contrack and nsh iface types are added to ovs you will also be able
to use this field to determine if ovs supports those features.

the choice of what vif_type to uses and the relevant bindings i think is fundamentally a controller decisio...

the netdev datapath is as you said not related to dpdk support in the vswitch.
it exist with out without linking ovs against the dpdk libary.

sudo ovs-vsctl list Open_vSwitch
_uuid               : b8761c30-5845-4bbd-885b-ee1d2e026568
bridges             : [1003bd42-5e7f-4867-af8f-b927789610b7, 3fe690a5-b197-494c-8edf-dcba7592838a, 4fab8831-cbb7-492f-8025-13bde85abccc]
cur_cfg             : 20
datapath_types      : [netdev, system]
db_version          : []
external_ids        : {}
iface_types         : [dpdk, dpdkr, dpdkvhostuser, geneve, gre, internal, ipsec_gre, lisp, patch, stt, system, tap, vxlan]
manager_options     : []
next_cfg            : 20
other_config        : {pmd-cpu-mask="4000000004"}
ovs_version         : []
ssl                 : []
statistics          : {}
system_type         : []
system_version      : []

my main dislike for having an explicit configuration option to enable dpdk support is
that i have prviously proposed adding such an option during the juno cycle at both atlanta and paris summits.
it was reject by  the neutron core team with the request that we develop a runtime detection mechanism instead.

based on the discussions at the paris summit i reached out to the openvswitch mailing list and proposed introducing the 
iface_types field above. when they were receptive to the enhancement i worked with our vswitch and odl teams to submit patches which have now been accepted and shipped in the latest release of both projects.

the reason i bring up odl is that one of the things i would like to do is simplify the configuration of ovs with dpdk regardless of the 
ml2 plugin that is manage it.  in the openvswitch neutron agent case a config value could be used but that is not possible if 
ovs is managed by  an openflow controller such as odl. the following review https://review.openstack.org/#/c/215612/ (review.openstack.org is down so cant double the link) enables vhost-user support with the networking-odl based on the same dection mechanisium of checking the supported interface and checking datapath_type as a sanity check.

ideally i would like to use the same mechanism for the ovs neutron agent, odl and the ofagent to reduce the variation between deployment choices and reduce documentation/operator burden.

having access to the iface_type feild has other advantages also . if dpdkvhostcuse or dpdkr(ivshmem) support is added to nova in the future it would be easy to add support to the ovs ml2 driver.

if the iface_types were exposed via a rest api they could be used for network capability based scheduling.
for example in the snippit above you can see that the ovs on my system supports the following tunneling protocals(geneve,gre,stt,vxlan and lisp). in future when contrack and nsh iface types are added to ovs  you will also be able
to use this field to determine if ovs supports those features.

the choice of what vif_type to uses and the relevant bindings i think is fundamentally a controller decision not an agent one.
the agent is responsible for adding the appropriate flows and switch configuration to the local ovs instance to establish l2 connectivity between the ovs port created by nova and the logical neutron tenant network. the  role of the neutron pluging/ml2 dirver is to generate the appropriate binding details and vif type to satisfy the users request and inform nova how to plug the compute instances interface in the network back-end on the compute node.

the mechanism you suggest in https://review.openstack.org/#/c/234517/ would work but it is essential an inversion of control away from the network control to the agent based on the config value. it has technically been possible to use an unmodified neutron agent with ovs with dpdk since icehouse. this restricted you to kernel vhost but it worked.  as of liberty the standard openvswtich l2 agent and ml2 driver supports kernel vhost with the linux kernel datapath, the windows kernel datapath and the dpdk and non dpdk userspace netdev datapaths.  since ovs has a feature that was expcitly added to allow a controllers to detect what interface it supports i prefer to leverage it then subclasses the agent and add a config option.

Revision history for this message

sean mooney (sean-k-mooney) wrote on 2015-10-16:

that was longer then i taught.

TL;DR

following discussion at the paris summit about the preference for an auto detection mechanism
a new feature was implemented in the ovsdb to allow openstack and odl to detect the interface types
it supports.

the iface_type field in the ovsdb has now been added to both the latest release of ovs and odl and can be used to detect vhost-user support at runtime. as the current agent implementation supports kernel vhost with the windows kernel, linux kernel and netdev datapath i think this approch is cleaner then introducing a config vaule to declare dpdk support to enable vhost-user.

Revision history for this message

Terry Wilson (otherwiseguy) wrote on 2015-10-16:

tl;dr I don't understand how autodetection as described is actually saving us any configuration.

Here are my concerns with the autodetection scheme as described above:
1) It seems to conflate capability and intent. Just because OVS is running with DPDK/vhost-user enabled on a host, doesn't necessarily mean that *Neutron* should be making use of it. OVS can handle DPDK and non-DPDK bridges at the same time. Some external app on a host could use DPDK for example, but maybe we wouldn't?

2) Having the config option for datapath_type means that we aren't doing autodetection anyway--we're just using the detection as a means of verifying that our config option makes sense. Or datapath_type itself is the equivalent of "are you sure? y/n" if we really want to rely on how OVS is started as an indication of intent how neutron should behave. If we are willing to make the assumption that we should be using vhost-user on a host just because it is available, why do we need the config value. If we need the config value, what does the autodetection really buy us other than a sanity check?

3) If we start supporting the other vhost methods at some point, again knowing that they are all available at the same time, we're still going to have to have a way to distinguish which one to use...so again we need more config options anyway.

Sean: You've been at this much longer than I have and know *way* more about DPDK than I (hopefully!) ever will. If my concerns above are a result of me misunderstanding something, I'll happily bow to your greater knowledge and experience. :) Implementing the autodetection is trivial, it just seemed like trading a config option for a config option plus runtime detection.

Revision history for this message

sean mooney (sean-k-mooney) wrote on 2015-10-18:

Download full text (7.1 KiB)

hi these are all good questions and a the risk of been asked to capture this as a spec ill try to answer them inline.

so for point 1 i have two comments.
first the datapath_type option in the ml2_conf.ini will enable this scenario.
if you want neutron to use the kenel datapath simply omit/set the datapath_type config option to its default value of system.
then the external application can create a separate dpdk bridge with a datapath of netdev for its own use.
the reverse should be also possible where by neutron can use the netdev datapath and the external application on the host could use. With the auto-detection scheme described above when neuton is configured to use the kernel datapath it would use kernel vhost and when configured to use the userspace datapath it will use vhost-user if available.

my second comment on point 1. when using the dpdk netdev datapath we always recommend removing the openvswitch kernel module. while it should technical be possible to run both datapaths concurrently that configuration is not well tested or documented anywhere to my knowledge. using ovs in this way would be a very advanced usecase as there are many edge cases to deal with. as an interface can only be added to one bridge/datapath either 2 interface are required or you need to interconnect the two datapaths at some point. the standard ways to interconnect bridges is patch ports. patch ports only work between two bridges of the same datapath_type, this fact is not documented clearly in ovs. they can be created between any bridge but will not function if the datapath is different, and will not emit an error. while veth paris or internal ports+ a linux bridge can be used to interconnect ovs bridges of different datapath types the performance is very limited due to the need to copy the packets. unless you are intimately familiar with the features of ovs and there constraints we recommend using only one datapath on a host at any one time.

ill come back to your second point shortly but first ill comment on you last point.
regarding point 3 while it is possible for either the dpdkvhostuser or dpdkvhostcuse port type
to appear in the iface_type field they cannot appear togeter. dpdk provides a vhost library containing
two similar but slightly different implementations. both implementations use the same datapath but each
different control path and therefore have the same performance characteristics .
dpdkvhostcuse uses a cuse character device to sent and receive control plane messages between the vhost back-end and qemu.
dpdkvhostuser uses a unix socket to communicate with qemu instead. dpdk can only be compiled with support for one of the two
vhost library implementations. as ovs support for vhost-user or vhost-cuse is provided by the dpdk vhost library only one of the two interface will be available in the iface_types field.

Finally back to you second point.
The introduction of the datapath_type config option is entirely* unrelated to vhost-user. ill expand on the * later.
it was introduced to address 3 issues:
1.) closing a gap in neutron's knowledge of the vswitch configuration.
2.) ensuring patch ports can be used with ovs-dpdk and neutron....

hi these are all good questions and a the risk of been asked to capture this as a spec ill try to answer them inline.

so for point 1 i have two comments.
first the datapath_type option in the ml2_conf.ini will enable this scenario.
if you want neutron to use the kenel datapath simply omit/set the datapath_type config option to its default value of system.
then the external application can create a separate dpdk bridge with a datapath of netdev for its own use.
the reverse should be also possible where by neutron can use the netdev datapath and the external application on the host could use. With  the auto-detection scheme described above when neuton is configured to use the kernel datapath it would use kernel vhost and when configured to use the userspace datapath it will use vhost-user if available.

my second comment on point 1. when using the dpdk netdev datapath we always recommend removing the openvswitch kernel module. while it should technical be possible to run both datapaths concurrently that configuration is not well tested or documented anywhere to my knowledge. using ovs in this way would be a very advanced usecase as there are many edge cases to deal with.  as an interface can only be added to one bridge/datapath either 2 interface are required or you need to interconnect the two datapaths at some point. the standard ways to interconnect bridges is patch ports. patch ports only work between two bridges of the same datapath_type, this fact is not documented clearly in ovs. they can be created between any bridge but will not function if the datapath is different, and will not emit an error. while veth paris or internal ports+ a linux bridge can be used to interconnect ovs bridges of different datapath types the performance is very limited due to the need to copy the packets.  unless you are intimately familiar with the features of ovs and there constraints we recommend using only one datapath on a host at any one time.

ill come back to your second point shortly but first ill comment on you last point.
regarding point 3 while it is possible for either the dpdkvhostuser or dpdkvhostcuse port type 
to appear in the iface_type field  they cannot appear togeter.  dpdk provides a vhost library containing 
two similar but slightly different implementations. both implementations use the same datapath but each
different control path and therefore have the same performance characteristics . 
dpdkvhostcuse uses a cuse character device to sent and receive control plane messages between the vhost back-end and qemu. 
dpdkvhostuser uses a unix socket to communicate with qemu instead. dpdk can only be compiled with support for one of the two
vhost library implementations. as ovs support for vhost-user or vhost-cuse is provided by the dpdk vhost library only one of the two interface will be available in the iface_types field.

1.) prior to adding the datapath_config option neutron did not know what datapath ovs was using.
when deploying neutron with openvswitch the openstack controlled bridges can be create in 1 of 2 ways.
A.) they can be created by the deployment tool/operator before starting the ovs agent.
B.) they can be created by the ovs agent when it starts if they are not present.

Option A allow for greater flexibility as steps such as adding physical interfaces to bridges, assigning the tunnel endpoint ip adders and configuring datapath can be done before starting the  ovs neutron agent.
Option B is less work but as neutron had no knowledge that ovs had datapaths to chose from it did not set that parameter when creating the bridges and as a result ovs defaulted to the kernel datapath which may or may not exist on the platform.

2.) patch port support is of particular importance in the context of ovs with dpdk. the primary reason for using ovs with dpdk is
to increase the maximium packet forwarding rate. as i said previously patch ports require all bridges they connect to use the same datapath. to ensure that patch ports work when use_veth_interconnection=false needs to enforce that the same datapath is uses on all bridges it controls.

3.) initially you may ask why br-tun is special and why i have not called out any other bridges. obviously the br-tun bridge
server the unique role of tunnel termination but due to issue 1 neutrons lack of knowledge about datapath_types and issue 2
the desire to support patch port the br-tun became an issue. to work around other issues the ovs neutron agent prior to the datapath change on start up reset the br-tun as part of its initialisation. this resulted in any datapath that was configured by the operator or deployment tool being lost. as the other bridges were not reset if patch ports were used to connect the br-tun with the br-int they would cease to function and no error would be emited by ovs or neutron.

Finally to the * in entirely unrelated. 
internally in openvswitch's implementation netdevs(interfaces) and datapaths are very loosely coupled.
so loosely coupled in fact that each netdev type (interface type) does not directly know what datapaths it supprots
and each datapath does not diretly know what netdevs it supports. i cant recall as i type this if the add-port command will give an error(i belive no error if --no-wait is used) but this means you can add a unsupported netdev(interface type) to a bridge and it will be accepted by the ovsdb. as a result the db will have added the port but when the vswtichd goes to instantiate the port it will fail leading to an inconstant state.  as a result to enable advanced use case such as that described in you first point a sanity check to ensure an unsupported port is not added to a datapath is desirable.

one last taught on the assumption that if vhost-user is available we should use it. while kernel vhost technically works with the dpdk datapath due to the decoupled nature of datapaths and netdevs in ovs its performance is very very low. for context with a standard openstack configuration using kernel vhost with ovs 2.4 and 1 vm looping back 64b udp packets from/too a phyical port. the maximum packet forward rate i was able to achieve was ~300,000 pps with security groups enabled or ~700,000 with them disables. with vhost-user on a dpdk based ovs i was able to reach ~3,800,000mpps, with the dpdkr interface i could reach ~7,700,000 mpps and with kernel vhost and the dpdk data path i could reach  only 32,000 pps.  take these number with a grain of salt, more as a guide then a absolute as they are clock speed and system memory dependent but i cant think of a usecase for using kernel vhost if the switch and datapath supports vhost-user. the performance of kernel vhost with the userspace datapath is so low as it involves 3 times as many copies and  userspace/kernelspace context switchs vs kernel vhost+ kernel datapath or vhost-user+ userspace datapath.

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2015-10-20:

To be discussed at the drivers meeting.

Changed in neutron:
status:	In Progress → Triaged

OpenStack Infra (hudson-openstack) on 2015-10-27

Changed in neutron:
status:	Triaged → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-10-28: Change abandoned on neutron (master)

Change abandoned by Terry Wilson (<email address hidden>) on branch: master
Review: https://review.openstack.org/234517
Reason: Abandoned for now in favor of https://review.openstack.org/#/c/237264/

Revision history for this message

Terry Wilson (otherwiseguy) wrote on 2015-10-28:

Implementation in https://review.openstack.org/#/c/237264/

OpenStack Infra (hudson-openstack) on 2015-10-30

Changed in neutron:
assignee:	Terry Wilson (otherwiseguy) → sean mooney (sean-k-mooney)

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2015-10-30:

Because of the long discussion, and someone mentioned it, I think it would be fair to ask for a brief spec that captures the rationale behind the design decisions made?

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2015-10-30:

#10

I would also like to understand what the implications to networking-ovs-dpdk are.

Armando Migliaccio (armando-migliaccio) on 2015-11-10

Changed in neutron:
status:	In Progress → New

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2015-11-11:

#11

I think this a good step in the right direction.

I initially suggested a spec, but a devref would help too in order to document some of the discussion/answer questions as reported/mentioned here.

I leave the choice to the bug authors, but I see documentation as a preliminary step to allow 237264 to go through. This shouldn't be too hard and other folks interested in this work would be grateful to see black on white in a more structured way.

Revision history for this message

Armando Migliaccio (armando-migliaccio) wrote on 2015-11-11:

#12

I think that it is also worthwhile raising the visibility of this initiative in the milestone planning [1], and for this reason I'll match it with a blueprint.

This would help us make sure that this enhancement won't fall through the cracks when it comes the time of wrapping up release notes.

[1] https://launchpad.net/neutron/+milestone/mitaka-1

Changed in neutron:
status:	New → Triaged
tags:	added: rfe-approved removed: rfe

Armando Migliaccio (armando-migliaccio) on 2015-11-11

Changed in neutron:
importance:	Undecided → Wishlist

OpenStack Infra (hudson-openstack) on 2015-11-17

Changed in neutron:
assignee:	sean mooney (sean-k-mooney) → Terry Wilson (otherwiseguy)
status:	Triaged → In Progress

Armando Migliaccio (armando-migliaccio) on 2015-11-20

Changed in neutron:
milestone:	none → mitaka-1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-11-20: Fix merged to neutron (master)

#13

Reviewed: https://review.openstack.org/237264
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=34d4d46c40b5204ffaf8c8a3e2464a19f9d8b2cd
Submitter: Jenkins
Branch: master

commit 34d4d46c40b5204ffaf8c8a3e2464a19f9d8b2cd
Author: Terry Wilson <email address hidden>
Date: Thu Oct 15 18:50:40 2015 -0500

Add vhost-user support via ovs capabilities/datapath_type

    Adds the ovs 'config' property which returns the contents of the
    single row of the Open_vSwitch table. This gives access to certain
    OVS capabilities such as datapath_types and iface_types.

    Using this information in concert with the datapath_type config
    option, vif details are calculated by the OVS mech driver. If
    datapath_type == 'netdev' and OVS on the agent host is capable of
    supporting dpdkvhostuser, then it is used.

Authored-By: Terry Wilson <email address hidden>
Co-Authored-By: Sean Mooney <email address hidden>

Closes-Bug: #1506127
Change-Id: I5047f1d1276e2f52ff02a0cba136e222779d059c