Wrong OVS flows created for new networks

Bug #1507776 reported by Claudiu Belu
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
High
YAMAMOTO Takashi

Bug Description

neutron-openvswitch-agent seems to create wrong OVS flows for newly created networks. This causes package losses, including lost DHCP requests, resulting in instances that did not receive an IP. This can cause tempest tests to fail.

Restarting the neutron-openvswitch-agent will result in properly created OVS flows, and the traffic flowing properly.

The issue has been observed using VLAN as a network_type and no other special configurations (like DVR being enabled), while running tempest tests: tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.
For example, testr run tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops randomly fails due to the fact that the instance did not get any reply for its DHCP request. NOTE: while the OVS flows are wrong for each run of the test, the test still has a small chance of success.

Hypervisor used: Hyper-V. The port on Hyper-V are properly bound and the VLAN tag properly set, the ports are marked as "alive". The network traffic arrives with the proper VLAN tag at the Networking Controller. The bug is unrelated to neutron-hyperv-agent and Hyper-V.

This issue has been observed in the Liberty release.

Details: http://paste.openstack.org/show/476764/

IRC discussion: http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/%23openstack-neutron.2015-10-19.log.html#t2015-10-19T20:27:31

Claudiu Belu (cbelu)
description: updated
Revision history for this message
Kevin Benton (kevinbenton) wrote :

For a brief summary. The local VLAN mapping between a network VLAN and the br-int tag seems to be getting mangled. Somehow two tags become associated with one real external network. This leads to incorrect rewrite rules for the devices on the other tag.

Claudiu Belu (cbelu)
description: updated
description: updated
Claudiu Belu (cbelu)
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/237878

Changed in neutron:
assignee: nobody → YAMAMOTO Takashi (yamamoto)
status: New → In Progress
Changed in neutron:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/237878
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c69407e676c483820f5c505d8d6d7923b0461819
Submitter: Jenkins
Branch: master

commit c69407e676c483820f5c505d8d6d7923b0461819
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Oct 21 13:54:06 2015 +0900

    Fix _restore_local_vlan_map race

    Local vlan mappings provisioned by _restore_local_vlan_map
    might not be cleaned up properly if the corresponding ports are
    removed before we enter the main loop.

    This commit fixes the problem by making _restore_local_vlan_map
    just record local vlan IDs and leave the rest of provisioning logic
    to the main loop.

    Closes-Bug: #1507776
    Change-Id: I84ff6684ae3fda7d839c14b1a7382871f45ce610

Changed in neutron:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/238986

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/238986
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=24d0b05f83861826e61d941611591703ca321b1e
Submitter: Jenkins
Branch: stable/liberty

commit 24d0b05f83861826e61d941611591703ca321b1e
Author: YAMAMOTO Takashi <email address hidden>
Date: Wed Oct 21 13:54:06 2015 +0900

    Fix _restore_local_vlan_map race

    Local vlan mappings provisioned by _restore_local_vlan_map
    might not be cleaned up properly if the corresponding ports are
    removed before we enter the main loop.

    This commit fixes the problem by making _restore_local_vlan_map
    just record local vlan IDs and leave the rest of provisioning logic
    to the main loop.

    Closes-Bug: #1507776
    Change-Id: I84ff6684ae3fda7d839c14b1a7382871f45ce610
    (cherry picked from commit c69407e676c483820f5c505d8d6d7923b0461819)

tags: added: in-stable-liberty
tags: removed: liberty-backport-potential
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/neutron 8.0.0.0b1

This issue was fixed in the openstack/neutron 8.0.0.0b1 development milestone.

Changed in neutron:
status: Fix Committed → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/neutron 7.0.1

This issue was fixed in the openstack/neutron 7.0.1 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers