KeyError: 'gateway_mac' in ovs dvr agent

[Armando Migliaccio]

This trace has been observed recently.

The logstash query that captures the trace is below:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiS2V5RXJyb3I6ICdnYXRld2F5X21hYydcIiBBTkQgYnVpbGRfbmFtZTpcImNoZWNrLXRlbXBlc3QtZHN2bS1uZXV0cm9uLWR2clwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDE4Nzk2MDQwNTM3LCJtb2RlIjoiIiwiYW5hbHl6ZV9maWVsZCI6IiJ9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/142697

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kyle Mestery (<email address hidden>) on branch: master
Review: https://review.openstack.org/142697
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Assaf Muller (amuller)]

Here's a reliable but somewhat silly way to reproduce this (I don't know what Tempest test would do this, this requires more investigation, or perhaps this isn't the only reproducer):

Create network, subnet, distributed router
Attach router to subnet but give it a different IP address than the subnet's gateway_ip, this can be accomplished by:
1) Horizon has an optional field that lets you specify the router's IP on the subnet
2) Via the CLI, create a port with a custom fixed IP, then attach it to the router
3) Boot a VM on the network, so that the node will have a port bound to it
4) Set the router as a gateway. This will start an error loop.

in _bind_distributed_router_interface_port, you get KeyError: 'ip_version'
in _bind_centralized_snat_port_on_dvr_subnet you get KeyError: 'gateway_mac'

Observing the code, this happens because when you set a DVR router as a gateway, an internal SNAT port is allocated, and the OVS agent executes _bind_centralized_snat_port_on_dvr_subnet, which calls:

subnet_info = self.plugin_rpc.get_subnet_for_dvr(self.context, subnet_uuid)

subnet_info will return as None, and the following couple of lines populates the local subnets cache with None. This means that any other code that retrieves this subnet from the cache (In the same method, or other methods that plug VM/DHCP ports, as well as regular distributed ports) will retrieve None and fail on KeyErrors.

[Assaf Muller (amuller)]

Sorry the above comment makes more sense if you remove '3)' and '4)' but keep their respective lines.

[Assaf Muller (amuller)]

Following my investigation and conversation with Swami, closing this bug as a duplicate of:
https://bugs.launchpad.net/neutron/+bug/1404823