There are many DBKeyNotFound exceptions in dragonflow's log when running fullstack

[duankebo]

 DEBUG dragonflow.db.api_nb [-] Event update: Action:set, Table:lswitch, Key:44619f66-deed-4f38-8e3d-27166415e980 Value:None Topic:D _read_db_changes_from_queue /opt/stack/new/dragonflow/dragonflow/db/api_nb.py:163
 DEBUG etcd.client [-] Issuing read for key /lswitch/44619f66-deed-4f38-8e3d-27166415e980 with args {} read /usr/local/lib/python2.7/dist-packages/etcd/client.py:521
 ERROR dragonflow.db.api_nb [-] DB Key not found, key=44619f66-deed-4f38-8e3d-27166415e980
 ERROR dragonflow.db.api_nb Traceback (most recent call last):
 ERROR dragonflow.db.api_nb File "/opt/stack/new/dragonflow/dragonflow/db/api_nb.py", line 170, in _read_db_changes_from_queue
 ERROR dragonflow.db.api_nb self.next_update.key)
 ERROR dragonflow.db.api_nb File "/opt/stack/new/dragonflow/dragonflow/db/drivers/etcd_db_driver.py", line 109, in get_key
 ERROR dragonflow.db.api_nb raise df_exceptions.DBKeyNotFound(key=key)
 ERROR dragonflow.db.api_nb DBKeyNotFound: DB Key not found, key=44619f66-deed-4f38-8e3d-27166415e980
 ERROR dragonflow.db.api_nb

[duankebo (duankebo)]

It seems that df-local-controller is quit slow in processing nb events. The port was created at 16:18:07, but local controller didn't process it untill 16:18:33. At that time,the port had already be removed and so the DBKeyNotFound occured!

2016-05-20 16:18:07.301 23254 DEBUG dragonflow.db.pubsub_drivers.zmq_pubsub_driver [req-5b031983-464b-4c4b-895b-1f7dfec94e99 admin -] sending Action:create, Table:lport, Key:521d3300-fe20-479e-9198-eb2b6b1f995a Value:None Topic:958ab8e7012e43979d03ce4f04ca3903 send_event /opt/stack/new/dragonflow/dragonflow/db/pubsub_drivers/zmq_pubsub_driver.py:92

2016-05-20 16:18:33.350 21847 DEBUG dragonflow.db.api_nb [-] Event update: Action:create, Table:lport, Key:521d3300-fe20-479e-9198-eb2b6b1f995a Value:None Topic:958ab8e7012e43979d03ce4f04ca3903 _read_db_changes_from_queue /opt/stack/new/dragonflow/dragonflow/db/api_nb.py:163

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/319765
Committed: https://git.openstack.org/cgit/openstack/dragonflow/commit/?id=6062a431707229426e0c3454f0df146b2046c622
Submitter: Jenkins
Branch: master

commit 6062a431707229426e0c3454f0df146b2046c622
Author: duankebo <email address hidden>
Date: Mon May 23 08:18:00 2016 +0000

    Limit the frequency of db sync

    If exception occurs when processing a northbound event, one second delay
    will be introduced by db sync, it leads to more exceptions because of
    the delay. Avalanche will occur then. This patch limits the frequency of
    db sync by one time every three minutes at maximum.

    Closes-Bug: #1583992

    Change-Id: I2f2635467b8fc5ab816ea4ac355e3e1cc488d5a2