KeyError in ovs_neutron_agent._bind_devices

[Matt Riedemann]

Seeing this all over the gate lately:

http://logs.openstack.org/95/176395/1/gate/gate-tempest-dsvm-neutron-full/37e1139/logs/screen-q-agt.txt.gz?level=TRACE#_2015-05-07_19_27_38_157

2015-05-07 19:27:38.157 ERROR neutron.plugins.openvswitch.agent.ovs_neutron_agent [req-0266e813-4a74-47c6-a1ed-6acc81317cb9 None None] Error while processing VIF ports
2015-05-07 19:27:38.157 21868 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2015-05-07 19:27:38.157 21868 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1637, in rpc_loop
2015-05-07 19:27:38.157 21868 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent ovs_restarted)
2015-05-07 19:27:38.157 21868 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1422, in process_network_ports
2015-05-07 19:27:38.157 21868 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent self._bind_devices(need_binding_devices)
2015-05-07 19:27:38.157 21868 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 736, in _bind_devices
2015-05-07 19:27:38.157 21868 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent lvm = self.local_vlan_map[port_detail['network_id']]
2015-05-07 19:27:38.157 21868 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent KeyError: u'323b3bcb-1530-4c76-83b1-49a35f255179'
2015-05-07 19:27:38.157 21868 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiRXJyb3Igd2hpbGUgcHJvY2Vzc2luZyBWSUYgcG9ydHNcIiBBTkQgbWVzc2FnZTpcIktleUVycm9yXCIgQU5EIHRhZ3M6XCJzY3JlZW4tcS1hZ3QudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzEwMzIyNzE5MTR9

2321 hits in 7 days. It's not directly related to failures, the jobs are still 93% successful, but it's spiking in the last few days.

[Matt Riedemann (mriedem)]

Looks like this is in some polling loop so maybe expected and we're just in a teardown of a port or something, not sure.

[Sean M. Collins (scollins)]

Also seeing issues with ovs-vsctl where a port is missing

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwibm8gcm93XCIgQU5EIG1lc3NhZ2U6XCJpbiB0YWJsZSBQb3J0XCIgQU5EIHRhZ3M6XCJzY3JlZW4tcS1hZ3QudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0MzEwMzI5MTYyMDl9

[Matt Riedemann (mriedem)]

Given the timing of the spike, this is probably the culprit:

https://github.com/openstack/neutron/commit/bd5373b670cdd7f21f8a1ece98fde6be9fda71ab

[Sean M. Collins (scollins)]

It's possibly being caused by https://review.openstack.org/#/c/118274/ which added code that modifies the local_vlan_map data structure in the agent

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/181185

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/181185
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=3a1175b88a436eecf00b8f04e5cc9f5cbce3ee06
Submitter: Jenkins
Branch: master

commit 3a1175b88a436eecf00b8f04e5cc9f5cbce3ee06
Author: Kevin Benton <email address hidden>
Date: Sat May 2 23:10:52 2015 -0700

    Check for missing network in _bind_devices

    _bind_devices was making the assumption that the ports it
    was operating had local VLAN map entries for their network.
    This wasn't the case when a network was deleted right before
    _bind_ports was called because the VLAN was reclaimed.

    This patch just checks to see if the the network ID has an entry
    in the map. If not, it skips the port. The port will be handled
    on the next scan_ports iteration when the agent will discover that
    the port is no longer defined on the plugin and it will be placed
    in the DEAD vlan.

    Change-Id: Ica51d727aceb41848fec0f4edbd16916365941ee
    Closes-Bug: #1452903

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: neutron-pecan
Review: https://review.openstack.org/185072