Key error in l3_dvr_db

[Eugene Nikanorov]

The following stack trace observed in the gate:

ERROR oslo.messaging.rpc.dispatcher [req-110db567-3322-4922-95c8-a54d166c8ead ] Exception during message handling: u'b132e9da-3ee2-47ac-9ed5-f04ca73c01d1'
TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
TRACE oslo.messaging.rpc.dispatcher incoming.message))
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/db/l3_rpc_base.py", line 57, in sync_routers
TRACE oslo.messaging.rpc.dispatcher context, host, router_ids)
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/db/l3_agentschedulers_db.py", line 191, in list_active_sync_routers_on_active_l3_agent
TRACE oslo.messaging.rpc.dispatcher active=True)
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 306, in get_sync_data
TRACE oslo.messaging.rpc.dispatcher DEVICE_OWNER_DVR_INTERFACE])
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 1016, in _get_router_info_list
TRACE oslo.messaging.rpc.dispatcher active=active)
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/db/l3_db.py", line 919, in _get_sync_routers
TRACE oslo.messaging.rpc.dispatcher return self._build_routers_list(context, router_dicts, gw_ports)
TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/new/neutron/neutron/db/l3_dvr_db.py", line 241, in _build_routers_list
TRACE oslo.messaging.rpc.dispatcher rtr['gw_port'] = gw_ports[gw_port_id]
TRACE oslo.messaging.rpc.dispatcher KeyError: u'b132e9da-3ee2-47ac-9ed5-f04ca73c01d1'a

Link: http://logs.openstack.org/48/112948/3/check/check-tempest-dsvm-neutron-pg/503d619/logs/screen-q-svc.txt.gz?#_2014-08-11_07_19_33_893

[Armando Migliaccio (armando-migliaccio)]

This is the L3-agent side of the trace:

http://logs.openstack.org/48/112948/3/check/check-tempest-dsvm-neutron-pg/503d619/logs/screen-q-vpn.txt.gz?level=TRACE#_2014-08-11_07_19_32_134

Incidentally I wonder if this is another manifestation of:

https://review.openstack.org/#/c/113144/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113400

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/113441

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113580

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113400
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fe49457605f96184d1fb334208d1ba57f513152a
Submitter: Jenkins
Branch: master

commit fe49457605f96184d1fb334208d1ba57f513152a
Author: armando-migliaccio <email address hidden>
Date: Mon Aug 11 16:15:28 2014 -0700

    Make plugin and l3plugin available as mixin's properties

    This makes the code simpler and slightly more efficient.
    This is a minor refactoring related to a race condition
    observed when the agent is processing a router while its
    interfaces are being removed.

    Related-bug: #1355409

    Change-Id: Ic0ce25aa6a977e28b1ab76081c74da4c46c2ca6b

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113441
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2affc39d3ad2d30c54e3373249446f61ee59f881
Submitter: Jenkins
Branch: master

commit 2affc39d3ad2d30c54e3373249446f61ee59f881
Author: armando-migliaccio <email address hidden>
Date: Mon Aug 11 22:43:31 2014 -0700

    Fix PortNotFound exception during sync_routers

    This trace is observed when an L3 agent invokes
    sync_routers right about the same time a port
    interface is removed from a router.

    Related-bug: #1355409

    Change-Id: I825b25080cbf054462318fc01248692b9e0e4ecb

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113580
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=aee5344db7972e2e12ed056c2e6467f7952aec3f
Submitter: Jenkins
Branch: master

commit aee5344db7972e2e12ed056c2e6467f7952aec3f
Author: armando-migliaccio <email address hidden>
Date: Tue Aug 12 09:11:50 2014 -0700

    Fix KeyError during sync_routers

    Method sync_routers is used by the L3 agent to query
    routers it knows about. Routers and GW ports lists
    are populated in two different times, which means that
    they can be interleaved by a delete request which
    results in gateway ports being missing in one of the
    two data structures.

    This patch takes care of the race condition.

    Closes-bug: #1355409

    Change-Id: Id3a6fe145058f690e107bfe7023980ede61cff90