[RFE] Distributed Portbinding for all port types

[Andreas Scheuring]

Summary
=======

Today only DVR ports can be bound to multiple hosts. But having a port bound to
multiple hosts does also make sense for a compute port during live migration.
For a certain period of time the port could be bound to the source and
target at the same time (Although only one is being used). The information of both bindings needs to be set and accessible from Nova via the ReST API.

Use Cases
=========

* Instance in error state when portbinding fails

In the live migration process, port binding is triggered by Nova after the migration already succeeded. If port binding fails, the instance is stuck in error state. If portbinding for the target node would be done in pre_live_migration, migration could be aborted on a binding failure and the instance would still be active on the migration source host. But we cannot just do so, as some TOR mech drivers would shut down the source port after the binding has been updated, although the instance is still active on the source. If we could bind a compute port to both hosts, such drivers could keep the source port open, and already process the target port in parallel.

* Live Migration between hosts running different l2 agents

Another use case is live migration between hosts that run different l2 agents. This requires that Nova updates the instance definition before migration is executed (in case of libvirt, update the domain.xml with target interface definition).

A specialized variant of this use case is the migration from an agent with one firewall driver to another (e.g. from ovs hybrid-fw driver to new ovs conntrackd firewall driver).

* Live migration with MacVTap agent when different physnet mappings is used

The third use case is live migration with MacVTap agent. Today it has some restrictions with live migration in some special scenarios [1]. It requires an update on the instance definition (libvirt domain.xml) before the migration started.

For updating the definition in time, a portbinding for the migration target node is required even before the migration started. Along the argumentation above, we need a compute port bound to multiple hosts.

Proposed Change
===============

* A refactoring of the database is required to make a normal port a special case of a distributed port. This was planned since a long time but was never finished. The efforts are tracked via this bug [1]. The patches still need to be rebased to get that going again.

* ReST API changes are required to externalize the bindings. To not overload the port API, a new subresource "bindings" could be created (like /ports/{port-id}/bindings) that holds the list of all bindings. CREATE/DELETE/UPDATE must be supported. Not UUID or would be required for this resource, as its identifier would be the host_id!

Nova Changes
============
* In pre_live_migration, nova would add a new binding for the migration target host to the port - this triggers portbinding in Neutron.
* Before migration starts, Nova would access the binding information for the target host. It would abort on "binding_failed" vif type. Otherwise it would modify the instance definition (e.g. domain.xml) for the migration target with this binding information.
* After live migration succeeded, Nova would remove the original port_binding. On Rollback, it would just remove the target port_binding.

Those changes are tracked via the following Nova blueprint [4]

Open Questions
==============
* This RFE is based on bug [1]. How to track those dependencies? Or should the content of this bug become part of this effort?
* Similar with the macvtap live migration bug [3]
* How does this effort correlate to the the RFE for externalizing multi-segment networks [2]?

[1] https://bugs.launchpad.net/neutron/+bug/1367391
[2] https://bugs.launchpad.net/neutron/+bug/1573197
[3] https://bugs.launchpad.net/neutron/+bug/1550400
[4] https://blueprints.launchpad.net/nova/+spec/migration-use-target-vif

[Armando Migliaccio (armando-migliaccio)]

I cannot help but think that we're somewhat abusing the term 'distributed' here, as in the use cases you presented the distributed nature of the binding is only short lived. I'll deep dive in the spec but I sense we may end up creating more trouble that we aim to address.

[Andreas Scheuring (andreas-scheuring)]

Thanks Armando, any feedback is highly welcome!

[venkata anil (anil-venkata)]

Can we refer this port as "migrated" port rather than "distributed" port?
We already have DVR and HA router ports(DEVICE_OWNER_DVR_INTERFACE, DEVICE_OWNER_HA_REPLICATED_INT, DEVICE_OWNER_ROUTER_SNAT) as distributed ports i.e these ports will always be present on multiple agents at same time.

[Andreas Scheuring (andreas-scheuring)]

What about "distributed portbinding for migrated ports"? Cause basically I want the functionality that is available for DVR today also for compute ports during migration + some API.

[Andreas Scheuring (andreas-scheuring)]

Spec: https://review.openstack.org/#/c/309416

[Armando Migliaccio (armando-migliaccio)]

I promise this is next on my stack. I have never had enough time to go over this one in peace.

[Armando Migliaccio (armando-migliaccio)]

Status update:

http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-06-30-22.00.log.html#l-320

[Armando Migliaccio (armando-migliaccio)]

I reviewed the spec, I think this needs to mature a little longer, but a reduction in scope seems in order.

[Armando Migliaccio (armando-migliaccio)]

Status update:

http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-07-07-22.02.log.html#l-158

[Armando Migliaccio (armando-migliaccio)]

This needs more scoping. Let's iterate on the spec to gather a better understanding of the beast at play.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/340410

[Armando Migliaccio (armando-migliaccio)]

Status update:

http://eavesdrop.openstack.org/meetings/neutron_drivers/2016/neutron_drivers.2016-07-21-22.00.log.html#l-156

[Armando Migliaccio (armando-migliaccio)]

Let's iterate on the spec until we can solidify a plan, this is now Ocata material, though some prep work will be evaluated for Newton on a best effort basis.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Andreas Scheuring (<email address hidden>) on branch: master
Review: https://review.openstack.org/340410

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/404293

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/414251

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309416
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=19eb264aaf4dd7d788f66ff37a1a7580063c8b41
Submitter: Jenkins
Branch: master

commit 19eb264aaf4dd7d788f66ff37a1a7580063c8b41
Author: Andreas Scheuring <email address hidden>
Date: Fri Apr 22 15:49:45 2016 +0200

    Spec for providing Nova portbinding information for live migration

    The goal is to provide nova the required portbinding information
    for live migration. This spec proposes to allow a port to have
    a second inactive binding. During migration, the host that
    runs the instance has the active binding, while the binding
    belonging to the other host is inactive.

    A new API is required to externalize this new inactive binding.

    Partial-Bug: #1580880
    Co-authored-by: Brian Stajkowski <email address hidden>
    Change-Id: I12d1310b8df30f5fbae4cb9c8e804d34c20ee7c9

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/404293
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7872508f9f25ea638f067cab65550778fbf1d0ed
Submitter: Jenkins
Branch: master

commit 7872508f9f25ea638f067cab65550778fbf1d0ed
Author: Anindita Das <email address hidden>
Date: Tue Nov 29 07:45:22 2016 -0800

    [Live Migration] Extend ml2_port_binding table

    Extended the Primary Key of ml2_port_binding table to include "hosts"
    and added a new column named "status" to indicate the status of the
    port similar to the ml2_distributed_port_binding table. Updated the
    PortBinding object with the above changes.

    Change-Id: Ie01f83b01ab673b9ae7e05b4eaed504ffc8db773
    Co-Authored-By: Brian Stajkowski <email address hidden>
    Partial-Bug: #1580880

[Andreas Scheuring (andreas-scheuring)]

Hi Ihar, I'm not sure if considering this for backports makes sense. It's a large feature rather than a bugfix - and it won't make ocata. But maybe I misunderstand the backport-potential-flag...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/460021

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/460021
Committed: https://git.openstack.org/cgit/openstack/neutron-specs/commit/?id=26f039390c64a698914165861981858ac849eda5
Submitter: Jenkins
Branch: master

commit 26f039390c64a698914165861981858ac849eda5
Author: Andreas Scheuring <email address hidden>
Date: Tue Apr 25 13:14:13 2017 +0200

    Repropose: Spec for providing Nova portbinding information for live migration

    This patch is reproposal of change proposed by Andreas at
    I12d1310b8df30f5fbae4cb9c8e804d34c20ee7c9 planned for Ocata.

    The goal is to provide nova the required portbinding information
    for live migration. This spec proposes to allow a port to have
    a second inactive binding. During migration, the host that
    runs the instance has the active binding, while the binding
    belonging to the other host is inactive.

    A new API is required to externalize this new inactive binding.

    Partial-Bug: #1580880
    Co-authored-by: Brian Stajkowski <email address hidden>

    Change-Id: I707e48f6c13f80f2696ac877c9e9948eb453e6e2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/484389

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/484389
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=6803dc6dddb7a689b78977903bf214a5fc69fd78
Submitter: Jenkins
Branch: master

commit 6803dc6dddb7a689b78977903bf214a5fc69fd78
Author: Jakub Libosvar <email address hidden>
Date: Mon Jul 17 14:35:14 2017 +0000

    Introduce API for port bindings extended

    This is a neutron-lib part that contains API resource definition of
    extended port bindings.

    Partial-Bug: #1580880

    Change-Id: I431be79975f6fc371822038ed4acdf02e7b8e99b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/512124
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=4c39634d2853c8a5fa8e7bb481c079ba384c9fab
Submitter: Zuul
Branch: master

commit 4c39634d2853c8a5fa8e7bb481c079ba384c9fab
Author: Miguel Lavalle <email address hidden>
Date: Sun Oct 15 18:21:23 2017 -0500

    Add new resource for port bindings

    As a result of implementing providing Nova port binding information for
    live migration [1], a new resource for callbacks is required. This
    resource will be used by the multiple port binding service plugin [2]
    to notify the core plugin and other subscribers when create, update and
    delete operations are performed on a port binding.

    [1] https://review.openstack.org/#/c/460021
    [2] https://review.openstack.org/#/c/414251

    Related-Bug: #1580880

    Change-Id: Ibbc7af67cf1282cbf7d9d3a681b9a04a7922c24a

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/558990

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/563803

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/564381

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/564381
Committed: https://git.openstack.org/cgit/openstack/neutron-lib/commit/?id=1d645c8ef30f6a616ab4b40cdf8d6f098674be4f
Submitter: Zuul
Branch: master

commit 1d645c8ef30f6a616ab4b40cdf8d6f098674be4f
Author: Miguel Lavalle <email address hidden>
Date: Wed Apr 25 21:38:46 2018 -0500

    Add port bindings to RPC topics resources

    This change adds port bindings to the RPC topics resources. It also
    adds the 'activate' and 'deactivate' messages for this resource. This
    is done in support of the implementation of multiple port bindings.

    Change-Id: I5ed542b7a137338a0878f683decc5dcca7f814de
    Partial-Bug: #1580880

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/567461

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/567462

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/414251
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f7b62a7f29e6b88fdbcc3bff779b39b541d1de47
Submitter: Zuul
Branch: master

commit f7b62a7f29e6b88fdbcc3bff779b39b541d1de47
Author: Jakub Libosvar <email address hidden>
Date: Thu Jul 27 14:15:11 2017 +0000

    Multiple port binding for ML2

    Functionality is added to the ML2 plugin to handle multiple port
    bindings

    Co-Authored-By: Anindita Das <email address hidden>
    Co-Authored-By: Miguel Lavalle <email address hidden>

    Partial-Bug: #1580880

    Change-Id: Ie31d4e27e3f55edfe334c4029ca9ed685e684c39

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/558990
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=390b6a531f7bb58ebc655306feee74cb0b6cd30c
Submitter: Zuul
Branch: master

commit 390b6a531f7bb58ebc655306feee74cb0b6cd30c
Author: Miguel Lavalle <email address hidden>
Date: Wed Apr 4 16:39:02 2018 -0500

    Update get_port_binding_host for multiple bindings

    With the adoption of multiple port bindings, get_port_binding_host is
    updated to select only the active binding

    Change-Id: I57ee16aa970527de2c2d787ebf0df71241f01e74
    Partial-Bug: #1580880

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/563803
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f374697760c65d3db53c50e77080a7a76819c24c
Submitter: Zuul
Branch: master

commit f374697760c65d3db53c50e77080a7a76819c24c
Author: Miguel Lavalle <email address hidden>
Date: Mon Apr 23 21:53:43 2018 -0500

    Add binding_deactivate method to Linux bridge agent

    This commit adds a binding_deactivate method to the Linux bridge agent
    to receive messages from the ML2 plugin when a binding is de-activated
    for a port. After receiving that message, the agent un-plugs the
    corresponding tap interface from the port's network bridge.

    To support this, a binding_deactivate method is also added to the agents
    notifier. Finally, the activate method in the ML2 plugin is updated to
    use the binding_deactivate method in the agents notifier.

    Change-Id: I3f4e34766791c472a2c81842190094f697baa05c
    Partial-Bug: #1580880

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567461
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=be7ad302e1f8b9a6b57c7b0c9f6c17f43f63f391
Submitter: Zuul
Branch: master

commit be7ad302e1f8b9a6b57c7b0c9f6c17f43f63f391
Author: Miguel Lavalle <email address hidden>
Date: Mon May 7 16:39:56 2018 -0500

    Support binding_deactivate in CacheBackedPluginApi

    CacheBackedPluginApi enables the neutron server to push resources
    updates to L2 agents. The agents retrieve the resources updates locally
    from the cache implemented by it. CacheBackedPluginApi also emulates
    server notifications to the agents, such as port_delete or port_update,
    based on the updated data received by the cache. This commit adds code
    to CacheBackedPluginApi to implement a binding_deactivate notification
    from the server to the agents

    Change-Id: I023ccbd405bc41379007d87a9c1051970aa8d603
    Partial-Bug: #1580880

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/567462
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0694bebd6de3934528492aadb78e3048dd5a07ea
Submitter: Zuul
Branch: master

commit 0694bebd6de3934528492aadb78e3048dd5a07ea
Author: Miguel Lavalle <email address hidden>
Date: Wed May 9 10:39:05 2018 -0500

    Add binding de-activation to OVS agent

    To support multiple port bindings, the L2 agents have to have the
    capability to handle binding deactivation notifications from the
    Neutron server. This patch adds the necessary code to the OVS agent.

    After receiving the notification, the agent un-plugs the corresponding
    VIF from the integration bridge.

    Change-Id: I78178de2039ccabc649558de4f6549a38de90418
    Partial-Bug: #1580880

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/574058
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=5c3bf124966a310cbc6c8ffad5ab30b144d9d7aa
Submitter: Zuul
Branch: master

commit 5c3bf124966a310cbc6c8ffad5ab30b144d9d7aa
Author: Miguel Lavalle <email address hidden>
Date: Sat Jun 9 20:03:39 2018 -0500

    Add binding activation to OVS agent

    As part of the implementation of multiple port bindings [1], add binding
    activation support to the OVS agent. This will enable the execution in
    OVS agents of the complete sequence of steps outlined in [1] during an
    instance migration:

    1) Create inactive port bindings for destination host
    2) Migrate the instance to the destination host and plug its VIFs
    3) Activate the port bindings in the destination host
    4) Delete the port bindings for the source host

    [1] https://review.openstack.org/#/c/309416/

    Change-Id: Iabca39364ec95633b2a8891fc295b3ada5f4f5e0
    Partial-Bug: #1580880

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/577637
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f7064f2b6c6ba1d0ab5f9872b2d5ad7969a64e7b
Submitter: Zuul
Branch: master

commit f7064f2b6c6ba1d0ab5f9872b2d5ad7969a64e7b
Author: Miguel Lavalle <email address hidden>
Date: Sat Jun 23 22:58:42 2018 -0500

    Add binding activation to the Linuxbridge agent

    As part of the implementation of multiple port bindings [1], add binding
    activation support to the linux bridge agent. This will enable the
    execution with linux bridge agents of the complete sequence of steps
    outlined in [1] during an instance migration:

    1) Create inactive port bindings for destination host
    2) Migrate the instance to the destination host and plug its VIFs
    3) Activate the port bindings in the destination host
    4) Delete the port bindings for the source host

    [1] https://review.openstack.org/#/c/309416/

    Change-Id: I2c937cc0a551e5ce0e8534c4dd4384ec2ca92da1
    Partial-Bug: #1580880

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/632045

[OpenStack Infra (hudson-openstack)]

Change abandoned by Paweł Suder (<email address hidden>) on branch: master
Review: https://review.openstack.org/632045
Reason: Thank you Miguel for clarification :))