HAHT - Fix datasource add/delete for out-of-sync situations

[Eric K]

When Congress is deployed in multi-node config, in-memory state can differ from DB state. Several errors can occur because the code does not properly account for it.

Examples 1:
- Nodes 1 and 2 both have DS0
- Delete DS0 on node1 (change made to both node1 and DB)
- Add DS0 on node2

Change is rejected because DS0 is still in memory in node2. Proper behavior is to add DS0 back to DB.

Example 2:
- Add ds named 'dsname' on node1
- 'dsname' automatically added on node2
- Remove ds name 'dsname' on node1, immediately add a *different* ds also named 'dsname' on node1.

The next time sync is triggered on node2, it keeps the old ds named 'dsname'. Proper behavior should be to update to the new ds on node2. What we need to do is to base sync on UUID (maybe plus other info) rather than name in ds sync.

Please see spec and blueprint for more information.
https://github.com/openstack/congress-specs/blob/master/specs/newton/high-availability-design.rst
https://blueprints.launchpad.net/congress/+spec/high-availability-design

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/356157

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/356157
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=9a10a90c15c2e85381fa3468fffc306b2a4b9f1e
Submitter: Jenkins
Branch: master

commit 9a10a90c15c2e85381fa3468fffc306b2a4b9f1e
Author: Eric K <email address hidden>
Date: Tue Aug 16 16:09:06 2016 -0700

    HAHT - datasource synchronizer use UUID not name

    Small but important change to used UUID instead of name in
    datasource sync. Avoids issues coming from name reuse.

    Also incidentally uncovered an issue with non-persistent
    datasources used for testing being deleted by synchronizer

    Temporarily fixed by assigning different DataService type
    to avoid being subject to sync.

    Closes-Bug: 1612007
    Partially implements blueprint: high-availability-design

    Change-Id: Ib67ed542b98cc21282cc510910114fc64970d417

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/congress 4.0.0.0b3 development milestone.