[Neutron] OVS agent is marking ports as dead before they are deleted

Bug #1731208 reported by Alexander Rubtsov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Won't Fix
Medium
Unassigned
9.x
Fix Released
Medium
Denis Meltsaykin

Bug Description

--- Release ---
MOS 9.2

--- Description ---
(from the original upstream bug - https://bugs.launchpad.net/neutron/+bug/1493414)

When trying to clear the gateway port and tenant network interface delete in router, the OVS agent is marking the port as dead instead of treat them as removed: security group removed and port_unbound

This is causing to left stale OVS flows in br-int, and it may affect the port_unbound() logic in ovs_neutron_agent.py.

The ovs_neutron_agent is in one iteration of rpc_loop processing the deleted port via process_deleted_ports() method, marking the qg- port as dead (ovs flow rule to drop the traffic) and in another iteration, the ovs_neutron_agent is processing the removed port by treat_devices_removed() method.

In first iteration, the port deleting is triggered by port_delete() method:
2015-09-04 14:16:20.337 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e43234b1-633b-404d-92d0-0f844dadb586 admin 0f6c0469ea6e4d95a27782c46021243a] port_delete message processed for port 1c749258-74fb-498b-9a08-1fec6725a1cf from (pid=136030) port_delete /opt/openstack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py:410

and in second iteration, the device removed is triggered by ovsdb:
2015-09-04 14:16:20.848 DEBUG neutron.agent.linux.ovsdb_monitor [-] Output received from ovsdb monitor: {"data":[["bab86f35-d004-4df6-95c2-0f7432338edb","delete","qg-1c749258-74",49,["map",[["attached-mac","fa:16:3e:99:37:68"],["iface-id","1c749258-74fb-498b-9a08-1fec6725a1cf"],["iface-status","active"]]]]],"headings":["row","action","name","ofport","external_ids"]}
 from (pid=136030) _read_stdout /opt/openstack/neutron/neutron/agent/linux/ovsdb_monitor.py:50

Log from ovs neutron agent:
http://paste.openstack.org/show/445479/

Steps to reproduce:
1. Create router
2. Add tenant network interface to the router
3. Launch a VM
4. Add external network gateway to created router
5. Check the br-int for current port numbers
6. Remove external network gateway
7. Check the br-int for dead port flows (removed port qg-)
8. Remove the network interface from tenant network
9. Check the br-int for dead port flows.

Repeat the steps 4-9 few times to see if dead port flows will appear in br-int.

This is affecting the legacy, dvr and HA router.

Revision history for this message
Alexander Rubtsov (arubtsov) wrote :

sla2 for 9.0-updates

Changed in mos:
importance: Undecided → Medium
tags: added: customer-found sla2
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/neutron (9.0/mitaka)

Fix proposed to branch: 9.0/mitaka
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/37132

Changed in mos:
status: New → Won't Fix
Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/neutron (9.0/mitaka)

Reviewed: https://review.fuel-infra.org/37132
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: b4c645a69013a76ca18a37a3100955d7c207cdc4
Author: Denis V. Meltsaykin <email address hidden>
Date: Thu Nov 9 12:10:20 2017

Remove stale ofport drop-rule upon port-delete

When a port is deleted, that port is set to a dead-vlan, and
an ofport drop-flow is added in port_dead().

The ofport drop-flow gets removed only in some cases
in _bind_devices() - depending on the timing of the
concurrent port-deletion. In other cases, the drop-flow
never gets removed, and such garbage drop-flow rules
accumulate forever until the ovs-agent restarts.

The fix is to use the function update_stale_ofport_rules which
solves this problem of tracking stale ofport flows
in deleted ports, but currently only applies only to
prevent_arp_spoofing.

(cherry-picked from 5289d9494984b7c95407ad2f9b761b2e647953b2)

Change-Id: I0d1dbe3918cc7d7b3d0cdc49d7b6ff85f9b02a17
Closes-Bug: #1731208

Revision history for this message
Vladimir Jigulin (vjigulin) wrote :

Verified on 9.2-mu-4 using steps in description

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/neutron (mcp/1.0/mitaka)

Fix proposed to branch: mcp/1.0/mitaka
Change author: Denis V. Meltsaykin <email address hidden>
Review: https://review.fuel-infra.org/38064

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.