OVS DVR: KeyError: 'gateway_mac'

[YAMAMOTO Takashi]

get_subnet_for_dvr RPC returns {} on error.
OVS agent, namely _bind_centralized_snat_port_on_dvr_subnet, doesn't handle the case gracefully.

eyJzZWFyY2giOiJtZXNzYWdlOiBcIktleUVycm9yOiAnZ2F0ZXdheV9tYWMnXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6Ijg2NDAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQzMTU3MzExODcxNH0

[req-4c481831-bcb8-47db-9487-e29eb396e871 None None] Error while processing VIF ports
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1641, in rpc_loop
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent ovs_restarted)
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1411, in process_network_ports
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent devices_added_updated, ovs_restarted))
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1318, in treat_devices_added_or_updated
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 1220, in treat_vif_port
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent fixed_ips, device_owner, ovs_restarted)
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/openvswitch/agent/ovs_neutron_agent.py", line 723, in port_bound
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent device_owner)
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/openvswitch/agent/ovs_dvr_neutron_agent.py", line 671, in bind_port_to_dvr
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent File "/opt/stack/new/neutron/neutron/plugins/openvswitch/agent/ovs_dvr_neutron_agent.py", line 641, in _bind_centralized_snat_port_on_dvr_subnet
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent (subnet_info['gateway_mac'],
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent KeyError: 'gateway_mac'
2015-05-13 15:46:44.026 5536 TRACE neutron.plugins.openvswitch.agent.ovs_neutron_agent

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/272025

[Oleg Bondarev (obondarev)]

What was the error in get_subnet_for_dvr() that made it return {}? Was it some race?

[Hong Hui Xiao (xiaohhui)]

Yes, I think it is some race. From [1], there are 2 possible case:

1) The agent is about to bind dvr port, but the subnet is deleted concurrently.
2) The agent is about to bind dvr port, but the gateway port is deleted concurrently.

[1] https://github.com/openstack/neutron/blob/d56b5a6718a678b7e7fe9c4c42cb8036b7228937/neutron/db/dvr_mac_db.py#L160-L188

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/272025
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7b5ba5b35a2c793a21a1f2d336cc339e37eeee2f
Submitter: Jenkins
Branch: master

commit 7b5ba5b35a2c793a21a1f2d336cc339e37eeee2f
Author: Hong Hui Xiao <email address hidden>
Date: Mon Jan 25 06:20:49 2016 -0500

    Log warning message if get_subnet_for_dvr fails

    When subnet or gateway port has been deleted concurrently, rpc call
    to get_subnet_for_dvr will return an empty dict without any value.
    ovs_dvr_neutron_agent should be more gracefull and at least log
    warning message in that situation.

    Change the existing error log to warning, because it is not a server
    error, but a fact that should be noticed.

    Change-Id: Icb3a57553a8b0eb635c0d85e2c60e7ce519893f6
    Closes-bug: #1454921

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/302259

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/302259
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=77e0d2f73ed65afd8b24ab1b8658afe7650c3d2d
Submitter: Jenkins
Branch: stable/liberty

commit 77e0d2f73ed65afd8b24ab1b8658afe7650c3d2d
Author: Hong Hui Xiao <email address hidden>
Date: Mon Jan 25 06:20:49 2016 -0500

    Log warning message if get_subnet_for_dvr fails

    When subnet or gateway port has been deleted concurrently, rpc call
    to get_subnet_for_dvr will return an empty dict without any value.
    ovs_dvr_neutron_agent should be more gracefull and at least log
    warning message in that situation.

    Change the existing error log to warning, because it is not a server
    error, but a fact that should be noticed.

    Change-Id: Icb3a57553a8b0eb635c0d85e2c60e7ce519893f6
    Closes-bug: #1454921
    (cherry picked from commit 7b5ba5b35a2c793a21a1f2d336cc339e37eeee2f)

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 7.1.2 release.