The database should only be changed by Neutron's API process. If it is changed in any other way, and the DB table is one that is synced to etcd, the value is not propagated because the mechanism driver does not get any warning of the change. That said, this happens on occasion:
1. the initial 'default' security group is added (at least by devstack) directly and not via an API call, which means that it is never
2. any manual change by the operator
3. when devstack is unstacked and stacked, etcd is not cleaned at any point. This can leave old information in etcd, which may or may not be overwritten (and if it's an old port then in all likelihood the data will remain indefinitely and be acted upon by the agent when restacked).
The security group pending patch uses a workaround for the above security group issue in that it will force population of a security group if the group is not in etcd when the port is bound, though this checks etcd from the REST thread and can potentially be problematic (two cases: if etcd is not reachable - in which case the API call will fail - and because the pending change is in the journal but not yet output - in which case a redundant update will be sent in some cases but not others, meaning behaviour is unpredictable and hard to test). Removing this functionality has been seen to show the problem; without the workaround the mechanism driver is nonfunctional.
It's unclear what the best solution would be. One option would be to prepopulate the journal table when the DB is initially populated on the basis of any rows already in the Neutron API tables, though this resolves only case 1. Another would be to provide a utility that would lock the journal table, ensure that it is emptied out, and then manually run a check to resync etcd with the database.
My current test shows that the DHCP port for the private network that devstack creates is not consistently forwarded to etcd which is a problem when using a devstack test setup.