revert resize: vif-plugged external event sent too soon if Neutron is using OVS hybrid plug

Bug #1832028 reported by Artom Lifshitz on 2019-06-07
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Artom Lifshitz
Rocky
Medium
Artom Lifshitz
Stein
Medium
Artom Lifshitz

Bug Description

Description
===========

This is all only when Neutron is using OVS with hybrid plugging.

When reverting a resized instance back to its original source host, Nova will timeout waiting for the vif-plugged external event, and never finish the revert. This happens because the event is sent by Neutron as soon as Nova updates the port binding to point back to the original source. This happens before the virt driver gets ready to listen for external events, so the event arrives, just too soon, and Nova times out.

Steps to reproduce
==================

1. Resize an instance
2. When it's in VERIFY_RESIZE, revert it

Expected result
===============

Instance reverts correctly.

Actual result
=============

Instance goes to ERROR.

Environment
===========

OVS with hybrid plug. Reported in OSP14/Rocky [1], reproduced on master [2] [3].

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1678681
[2] https://review.opendev.org/#/c/660782/
[3] https://review.opendev.org/#/c/653498/

Changed in nova:
assignee: nobody → Artom Lifshitz (notartom)
status: New → In Progress
Changed in nova:
assignee: Artom Lifshitz (notartom) → sean mooney (sean-k-mooney)
sean mooney (sean-k-mooney) wrote :

triaging as medium as this can technically as this race conditon can technically be worked around by
either using the default ovs contrack security group driver instead of the legacy ip tables firewall or
by disableling waiting for vif plugged event.

otherwise i would set this as high.

Changed in nova:
assignee: sean mooney (sean-k-mooney) → Artom Lifshitz (notartom)
tags: added: network neutron resize
Changed in nova:
assignee: Artom Lifshitz (notartom) → sean mooney (sean-k-mooney)
importance: Undecided → Medium
assignee: sean mooney (sean-k-mooney) → Artom Lifshitz (notartom)
Changed in nova:
assignee: Artom Lifshitz (notartom) → sean mooney (sean-k-mooney)
Matt Riedemann (mriedem) wrote :

Because of https://review.opendev.org/#/c/605041/ I think we can take the fix back to stable/rocky.

Changed in nova:
assignee: sean mooney (sean-k-mooney) → Artom Lifshitz (notartom)
tags: added: libvirt
Patrick Oberdorf (obi12341) wrote :

We are facing this issue too. Is it fixed? https://review.opendev.org/#/c/605041/ is 7 months old, maybe I am missing something.

Download full text (3.6 KiB)

Reviewed: https://review.opendev.org/644881
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=19f9b37721d9bc13bc1ed35a4f368b1d21b10a5b
Submitter: Zuul
Branch: master

commit 19f9b37721d9bc13bc1ed35a4f368b1d21b10a5b
Author: Artom Lifshitz <email address hidden>
Date: Wed Mar 20 10:38:12 2019 -0400

    Revert resize: wait for events according to hybrid plug

    Since 4817165fc5938a553fafa1a69c6086f9ebe311af, when reverting a
    resized instance back to the source host, the libvirt driver waits for
    vif-plugged events when spawning the instance. When called from
    finish_revert_resize() in the source compute manager, libvirt's
    finish_revert_migration() does not pass vifs_already_plugged to
    _create_domain_and_network(), making the latter use the default False
    value.

    When the source compute manager calls
    network_api.migrate_instance_finish() in finish_revert_resize(), this
    updates the port binding back to the source host. If Neutron is
    configured to use OVS hybrid plug, it will send the vif-plugged event
    immediately after completing this request. This happens before the
    virt driver's finish_revert_migration() method is called. This causes
    the wait in the libvirt driver to time out because the event is
    received before Nova starts waiting for it.

    The neutron ovs l2 agent sends vif-plugged events when two conditions
    are met. First the port must be bound to the host managed by the
    l2 agent and second, the agent must have completed configuring the
    port on ovs. This involves assigning the port a local VLAN for tenant
    isolation, applying security group rules if required and applying
    QoS policies or other agent extensions like service function chaining.

    During the boot process, we bind the port first to the host
    then plug the interface into ovs which triggers the l2 agent to
    configure it resulting in the emission of the vif-plugged event.
    In the revert case, as noted above, since the vif is already plugged
    on the source node when hybrid-plug is used, binding the port to the
    source node fulfils the second condition to send the vif-plugged event.

    Events sent immediately after port binding update are hereafter known
    as "bind-time" events. For ports that do not use OVS hybrid plug,
    Neutron will continue to send vif-plugged events only when Nova
    actually plugs the VIF. These types of events are hereafter known as
    "plug-time" events. OVS hybrid plug is a per agent setting, so for
    a particular host, bind-time events are an all-or-nothing thing for the
    ovs backend: either all VIF_TYPE=ovs ports have them, or no ovs ports
    have them. In general, a host will only have one network backend.
    The only exception to this is SR-IOV. SR-IOV is commonly deployed on
    the same host as other network backends such as OVS or linuxbridge.
    SR-IOV ports with VNIC_TYPE=direct-physical will always have only
    bind-time events. If an instance mixes OVS ports with hybrid-plug=False
    with direct physical ports, it will have both kinds of events.

    This patch adds functions to the NetworkIn...

Read more...

Changed in nova:
status: In Progress → Fix Released
Patrick Oberdorf (obi12341) wrote :

Cool thanks :) will this be backported to stable/rocky?

Matt Riedemann (mriedem) wrote :

I'm going to mark this as not fixed yet because it introduced a regression for same host resize + revert and the fix is being reverted on master since it broke neutron's CI jobs:

https://review.opendev.org/#/c/667035/

Changed in nova:
status: Fix Released → Triaged

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/666951

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/666952

Fix proposed to branch: master
Review: https://review.opendev.org/667177

Changed in nova:
assignee: Artom Lifshitz (notartom) → sean mooney (sean-k-mooney)
status: Triaged → In Progress
Changed in nova:
assignee: sean mooney (sean-k-mooney) → Artom Lifshitz (notartom)
Download full text (3.8 KiB)

Reviewed: https://review.opendev.org/667177
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7a7a223602ca5aa0aca8f65a6ab143f1d8f8ec1b
Submitter: Zuul
Branch: master

commit 7a7a223602ca5aa0aca8f65a6ab143f1d8f8ec1b
Author: Artom Lifshitz <email address hidden>
Date: Wed Mar 20 10:38:12 2019 -0400

    Revert resize: wait for events according to hybrid plug

    Since 4817165fc5938a553fafa1a69c6086f9ebe311af, when reverting a
    resized instance back to the source host, the libvirt driver waits for
    vif-plugged events when spawning the instance. When called from
    finish_revert_resize() in the source compute manager, libvirt's
    finish_revert_migration() does not pass vifs_already_plugged to
    _create_domain_and_network(), making the latter use the default False
    value.

    When the source compute manager calls
    network_api.migrate_instance_finish() in finish_revert_resize(), this
    updates the port binding back to the source host. If Neutron is
    configured to use OVS hybrid plug, it will send the vif-plugged event
    immediately after completing this request. This happens before the
    virt driver's finish_revert_migration() method is called. This causes
    the wait in the libvirt driver to time out because the event is
    received before Nova starts waiting for it.

    The neutron ovs l2 agent sends vif-plugged events when two conditions
    are met. First the port must be bound to the host managed by the
    l2 agent and second, the agent must have completed configuring the
    port on ovs. This involves assigning the port a local VLAN for tenant
    isolation, applying security group rules if required and applying
    QoS policies or other agent extensions like service function chaining.

    During the boot process, we bind the port first to the host
    then plug the interface into ovs which triggers the l2 agent to
    configure it resulting in the emission of the vif-plugged event.
    In the revert case, as noted above, since the vif is already plugged
    on the source node when hybrid-plug is used, binding the port to the
    source node fulfils the second condition to send the vif-plugged event.

    Events sent immediately after port binding update are hereafter known
    as "bind-time" events. For ports that do not use OVS hybrid plug,
    Neutron will continue to send vif-plugged events only when Nova
    actually plugs the VIF. These types of events are hereafter known as
    "plug-time" events. OVS hybrid plug is a per agent setting, so for
    a particular host, bind-time events are an all-or-nothing thing for the
    ovs backend: either all VIF_TYPE=ovs ports have them, or no ovs ports
    have them. In general, a host will only have one network backend.
    The only exception to this is SR-IOV. SR-IOV is commonly deployed on
    the same host as other network backends such as OVS or linuxbridge.
    SR-IOV ports with VNIC_TYPE=direct-physical will always have only
    bind-time events. If an instance mixes OVS ports with hybrid-plug=False
    with direct physical ports, it will have both kinds of events.

    For same host resize reverts we do not upd...

Read more...

Changed in nova:
status: In Progress → Fix Released
Download full text (4.1 KiB)

Reviewed: https://review.opendev.org/670645
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7a3a8f325ef6eaa97de1cf74efbaa0079f61a9e6
Submitter: Zuul
Branch: stable/stein

commit 7a3a8f325ef6eaa97de1cf74efbaa0079f61a9e6
Author: Artom Lifshitz <email address hidden>
Date: Wed Mar 20 10:38:12 2019 -0400

    Revert resize: wait for events according to hybrid plug

    Since 4817165fc5938a553fafa1a69c6086f9ebe311af, when reverting a
    resized instance back to the source host, the libvirt driver waits for
    vif-plugged events when spawning the instance. When called from
    finish_revert_resize() in the source compute manager, libvirt's
    finish_revert_migration() does not pass vifs_already_plugged to
    _create_domain_and_network(), making the latter use the default False
    value.

    When the source compute manager calls
    network_api.migrate_instance_finish() in finish_revert_resize(), this
    updates the port binding back to the source host. If Neutron is
    configured to use OVS hybrid plug, it will send the vif-plugged event
    immediately after completing this request. This happens before the
    virt driver's finish_revert_migration() method is called. This causes
    the wait in the libvirt driver to time out because the event is
    received before Nova starts waiting for it.

    The neutron ovs l2 agent sends vif-plugged events when two conditions
    are met. First the port must be bound to the host managed by the
    l2 agent and second, the agent must have completed configuring the
    port on ovs. This involves assigning the port a local VLAN for tenant
    isolation, applying security group rules if required and applying
    QoS policies or other agent extensions like service function chaining.

    During the boot process, we bind the port first to the host
    then plug the interface into ovs which triggers the l2 agent to
    configure it resulting in the emission of the vif-plugged event.
    In the revert case, as noted above, since the vif is already plugged
    on the source node when hybrid-plug is used, binding the port to the
    source node fulfils the second condition to send the vif-plugged event.

    Events sent immediately after port binding update are hereafter known
    as "bind-time" events. For ports that do not use OVS hybrid plug,
    Neutron will continue to send vif-plugged events only when Nova
    actually plugs the VIF. These types of events are hereafter known as
    "plug-time" events. OVS hybrid plug is a per agent setting, so for
    a particular host, bind-time events are an all-or-nothing thing for the
    ovs backend: either all VIF_TYPE=ovs ports have them, or no ovs ports
    have them. In general, a host will only have one network backend.
    The only exception to this is SR-IOV. SR-IOV is commonly deployed on
    the same host as other network backends such as OVS or linuxbridge.
    SR-IOV ports with VNIC_TYPE=direct-physical will always have only
    bind-time events. If an instance mixes OVS ports with hybrid-plug=False
    with direct physical ports, it will have both kinds of events.

    For same host resize reverts we do n...

Read more...

Download full text (4.3 KiB)

Reviewed: https://review.opendev.org/670648
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d9892abd2f096c05a2885a4902128941a24f5670
Submitter: Zuul
Branch: stable/rocky

commit d9892abd2f096c05a2885a4902128941a24f5670
Author: Artom Lifshitz <email address hidden>
Date: Wed Mar 20 10:38:12 2019 -0400

    Revert resize: wait for events according to hybrid plug

    Since 4817165fc5938a553fafa1a69c6086f9ebe311af, when reverting a
    resized instance back to the source host, the libvirt driver waits for
    vif-plugged events when spawning the instance. When called from
    finish_revert_resize() in the source compute manager, libvirt's
    finish_revert_migration() does not pass vifs_already_plugged to
    _create_domain_and_network(), making the latter use the default False
    value.

    When the source compute manager calls
    network_api.migrate_instance_finish() in finish_revert_resize(), this
    updates the port binding back to the source host. If Neutron is
    configured to use OVS hybrid plug, it will send the vif-plugged event
    immediately after completing this request. This happens before the
    virt driver's finish_revert_migration() method is called. This causes
    the wait in the libvirt driver to time out because the event is
    received before Nova starts waiting for it.

    The neutron ovs l2 agent sends vif-plugged events when two conditions
    are met. First the port must be bound to the host managed by the
    l2 agent and second, the agent must have completed configuring the
    port on ovs. This involves assigning the port a local VLAN for tenant
    isolation, applying security group rules if required and applying
    QoS policies or other agent extensions like service function chaining.

    During the boot process, we bind the port first to the host
    then plug the interface into ovs which triggers the l2 agent to
    configure it resulting in the emission of the vif-plugged event.
    In the revert case, as noted above, since the vif is already plugged
    on the source node when hybrid-plug is used, binding the port to the
    source node fulfils the second condition to send the vif-plugged event.

    Events sent immediately after port binding update are hereafter known
    as "bind-time" events. For ports that do not use OVS hybrid plug,
    Neutron will continue to send vif-plugged events only when Nova
    actually plugs the VIF. These types of events are hereafter known as
    "plug-time" events. OVS hybrid plug is a per agent setting, so for
    a particular host, bind-time events are an all-or-nothing thing for the
    ovs backend: either all VIF_TYPE=ovs ports have them, or no ovs ports
    have them. In general, a host will only have one network backend.
    The only exception to this is SR-IOV. SR-IOV is commonly deployed on
    the same host as other network backends such as OVS or linuxbridge.
    SR-IOV ports with VNIC_TYPE=direct-physical will always have only
    bind-time events. If an instance mixes OVS ports with hybrid-plug=False
    with direct physical ports, it will have both kinds of events.

    For same host resize reverts we do n...

Read more...

This issue was fixed in the openstack/nova 19.0.2 release.

This issue was fixed in the openstack/nova 18.2.2 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers