pre-populate and hash ring boostrapping is racey

[Adam Gandelman]

On startup, astara-orchestrator pre-populates the notifications queue with messages for all neutron resources it finds. The notification queue is also used by the coordination subprocess to push cluster membership to all the workers, so they can seed their copies of the cluster hash ring. If messages for resources that are found during pre-populate reach the workers before the initial cluster bootstrapping message is received and processed, the message may be ignored.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/289609

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/289609
Committed: https://git.openstack.org/cgit/openstack/astara/commit/?id=f9c5941b972da0aae8957fd07c55e47ef844bb2b
Submitter: Jenkins
Branch: master

commit f9c5941b972da0aae8957fd07c55e47ef844bb2b
Author: Adam Gandelman <email address hidden>
Date: Mon Mar 7 14:54:05 2016 -0800

    Handle a race between pre-populate and hash ring bootstrapping

    On startup, if the worker receives messages for pre-populated resources
    prior to processing the initial cluster rebalance event, the messages
    will be dropped. This fixes the race by tracking when the hash ring
    has been initialized. Any events it receives prior to finishing init
    will be batched up and processed as part of the initial bootstrapping
    procedure.

    Change-Id: I3caf95f57380076ab48e4270e1cd575906fba386
    Closes-bug: #1554248